ccontrol - Control Cluster Resources

ccontrol is a command-line tool for managing and controlling CraneSched cluster resources.

ccontrol provides SQL-style syntax for managing nodes, partitions, jobs, and reservations in a CraneSched cluster. The tool supports operations such as viewing status, updating configurations, holding/releasing jobs, and creating/deleting reservations.

Job Query Visibility

Job and step summaries (state, resources, times, and nodes) remain public. Command lines, environments, working directories, extra attributes, and container metadata are returned only to authenticated owners or global Operator/Admin/Root roles. Sharing an account or being a coordinator does not grant private-detail access. Step details require access to both the parent job and the step. Container image passwords are never returned, including to owners and administrators; the original execution and persisted values remain unchanged.

Restricted text fields display <restricted> or are omitted. JSON retains field types, using empty values or omitted messages and a query_visibility value of PUBLIC or DETAILS. User/Job ID filters and output formats do not grant access. The same rules apply to cacct, cqueue, ccon, and their Slurm wrappers.

Non-TLS connections cannot authenticate a caller UID, so they receive public summaries even for an owner or root. ccon logs/inspect/inspectp and updates that need the original extra attributes fail explicitly when details are restricted. Use matching frontend and backend releases; this feature needs no database migration.

show config still displays the local public configuration without an administrator check. Configuration permissions and display of TLS file paths are unchanged.

Command Structure

ccontrol <ACTION> <ENTITY> [OPTIONS]

Actions

• show - Display information about entities
• update - Modify attributes of entities
• hold - Hold entities (prevent jobs from starting)
• release - Release previously held entities
• create - Create new entities
• delete - Delete existing entities

Entities

• node - Compute nodes
• partition - Node partitions
• job - Jobs/tasks
• step - Job steps (sub-execution units within jobs)
• reservation - Resource reservations
• lic - License
• hostnames - Expanded hostnames from a hostlist expression

Global Options

• -h/--help: Display help message
• -v/--version: Show version information
• -J/--json: Format output as JSON
• -C/--config: Specify alternative configuration file path (default: "/etc/crane/config.yaml")

Command Reference

Show Commands

Show Nodes

Display information about compute nodes.

# Show all nodes
ccontrol show node

# Show specific node
ccontrol show node <nodename>

Example:

ccontrol show node
ccontrol show node crane01

[root@CraneProject CraneSched-FrontEnd]# ccontrol show node
NodeName=CraneProject State=idle[power_idle] CPU=2.00 AllocCPU=0.00 FreeCPU=2.00
    RealMemory=2048M AllocMem=0 FreeMem=2048M
    Gres=None AllocGres=None FreeGres=None
    Partition=CPU RunningJob=0 Version=unknown
    Os=Linux 5.14.0-570.23.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 26 19:29:53 UTC 2025
    BootTime=2025-07-04 16:09:10 CranedStartTime=2025-07-04 16:57:15
    LastBusyTime=unknown

Show Partitions

Display information about partitions.

# Show all partitions
ccontrol show partition

# Show specific partition
ccontrol show partition <partition_name>

Example:

ccontrol show partition
ccontrol show partition CPU

[root@CraneProject CraneSched-FrontEnd]# ccontrol show partition
PartitionName=CPU State=UP
    AllowedAccounts=ALL DeniedAccounts=None
    TotalNodes=1 AliveNodes=1
    TotalCPU=2.00 AvailCPU=2.00 AllocCPU=0.00
    TotalMem=2048M AvailMem=2048M AllocMem=0
    TotalGres=None AvailGres=None AllocGres=None
    DefaultMemPerCPU=1024M MaxMemPerCPU=0
    HostList=CraneProject

Show Jobs

Display information about jobs.

# Show all jobs
ccontrol show job

# Show specific job
ccontrol show job <job_id>

Example:

ccontrol show job
ccontrol show job 12345

[root@CraneProject CraneSched-FrontEnd]# ccontrol show job
JobId=12345 JobName=Test_Job
        User=root(0) GroupId=root(0) Account=ROOT
        JobState=Running RunTime=00:00:05 TimeLimit=00:03:01 SubmitTime=2026-03-18 20:16:12
        StartTime=2026-03-18 20:16:14 EndTime=2026-03-18 20:19:15 Partition=CPU NodeList=crane01 ExecutionHost=crane01
        CmdLine="cbatch cbatch_test.sh" Workdir=/root
        Priority=0 Qos=UNLIMITED CpusPerTask=1 MemPerNode=20M
        ReqRes:node=1 cpu=1.00 mem=20M gres=None
        AllocRes:node=1 cpu=1.00 mem=20M gres=None
        ReqNodeList=None ExcludeNodeList=None
        Exclusive=false Comment= Wckey=
        SubmitNode=crane01

Show Steps

Display detailed information about job steps.

# Show specific step
ccontrol show step <job_id>.<step_id>

# Show all steps (requires --all flag)
ccontrol show step --all

Syntax: - Step ID format: jobid.stepid (e.g., 123.1, 456.2)

Output Fields: - StepId: Step identifier in format jobid.stepid - Name: Step name - Partition: Partition name (inherited from parent job) - User: Username - Account: Account name (inherited from parent job) - QoS: Quality of Service (inherited from parent job) - State: Current step state (PENDING, RUNNING, COMPLETED, FAILED, CANCELLED) - TimeLimit: Time limit for the step - ElapsedTime: Time elapsed since step started - NodeList: Nodes allocated to the step - CPUs: Number of CPUs allocated to the step - Memory: Memory allocated to the step - GRES: Generic resources (GPU, etc.) allocated to the step - StartTime: Time when the step started

Examples:

# Show details of step 1 from job 123
ccontrol show step 123.1

# Show details of step 2 from job 456
ccontrol show step 456.2

Example Output:

StepId=123.1 StepName=task1
Partition=compute Account=research_group QoS=normal
State=RUNNING TimeLimit=01:00:00 ElapsedTime=00:15:30
NodeList=node[01-02] CPUs=8 Memory=4096MB
GRES=gpu:2
StartTime=2025-01-15T10:30:00

Usage Notes: - Steps inherit partition, account, and QoS from their parent job - Use cqueue --step for a list view of all steps - Use ccontrol show step for detailed information about a specific step - Step states include: PENDING, RUNNING, COMPLETED, FAILED, CANCELLED

Show Reservations

Display information about reservations.

# Show all reservations
ccontrol show reservation

# Show specific reservation
ccontrol show reservation <reservation_name>

Example:

ccontrol show reservation
ccontrol show reservation my_reservation

[root@CraneProject CraneSched-FrontEnd]# ccontrol show reservation
No reservation is available.

Show License

Display information about the license.

# Show all licenses
ccontrol show lic

#  Show specific license
ccontrol show lic <license_name>

Example:

ccontrol show lic
ccontrol show lic my_lic

LicenseName=ansys
        Total=100 Used=0  Free=100
LicenseName=fluent
        Total=30 Used=0  Free=30

Expand Hostnames

Expand a hostlist expression and print one hostname per line. The singular entity name hostname is accepted as an alias. Expansion is performed locally and does not query the controller.

ccontrol show hostnames [<hostlist>]

Examples:

$ ccontrol show hostnames 'node[01-03]'
node01
node02
node03

$ ccontrol show hostname 'rack[0-1]_blade[01-02]'
rack0_blade01
rack0_blade02
rack1_blade01
rack1_blade02

When <hostlist> is omitted, ccontrol uses the first non-empty variable in this order: SLURM_JOB_NODELIST, SLURM_NODELIST, CRANE_JOB_NODELIST, then CRANE_NODELIST. This makes the command convenient inside an allocated job:

ccontrol show hostnames > hostfile

The Slurm-compatible form is also available through cwrapper:

scontrol show hostnames 'node[01-03]'
scontrol show hostnames > hostfile

Without an explicit hostlist, scontrol checks SLURM_JOB_NODELIST, SLURM_NODELIST, CRANE_JOB_NODELIST, and CRANE_NODELIST, in that order.

Update Commands

Update Node

Modify node attributes, typically to drain or resume nodes.

ccontrol update nodeName=<nodename> state=<state> [reason=<reason>]

Parameters:

• nodeName (required): Name of the node to update
• state (required): New state for the node
• drain - Prevent new jobs from being scheduled on the node
• resume - Make node available for scheduling again
• reason (required when draining): Reason for draining the node

Examples:

# Drain a node
ccontrol update nodename=crane01 state=drain reason="Maintenance"

# Resume a node
ccontrol update nodename=crane01 state=resume

[root@CraneProject CraneSched-FrontEnd]# ccontrol update nodename=test_node state=drain reason="test"
Failed to modify node: test_node. Reason: Invalid node name specified..
ERRO[0000] change node state failed:                     
ERRO[0000] command execution failed

Update Job

Modify job attributes including priority, time limit, comment, and email settings.

ccontrol update jobid=<job_id> [priority=<priority>] [timelimit=<timelimit>] [comment=<comment>] [mailuser=<email>] [mailtype=<type>] [deadline=<deadline>]

Parameters:

• jobid (required): ID of the job to update
• priority (optional): New priority value (floating-point number)
• timelimit (optional): New time limit (format: HH:MM:SS or D-HH:MM:SS)
• comment (optional): New comment string for the job
• mailuser (optional): Email address for notifications
• mailtype (optional): Type of email notifications (NONE, BEGIN, END, FAIL, TIMELIMIT, ALL)
• deadline (optional)：Deadline time(various time formats)

Examples:

# Update job priority
ccontrol update jobid=11 priority=100

# Update time limit
ccontrol update jobid=11 timelimit=01:00:00

# Update multiple attributes
ccontrol update jobid=11 priority=1 timelimit=01:00:00 comment="High priority job"

# Update email notifications
ccontrol update jobid=11 mailuser=user@example.com mailtype=END

[root@CraneProject CraneSched-FrontEnd]# ccontrol update jobid=11 priority=1 timelimit=01:00:00
Failed to modify job: 11. Reason: Task #11 was not found in pending queue..
ERRO[0000] change task priority failed:
Failed to modify job: 11. Reason: Task #11 was not found in running or pending queue..
ERRO[0000] change task time limit failed:
ERRO[0000] command execution failed

Update Partition

Modify partition access control lists.

ccontrol update partitionName=<partition> [accounts=<accounts>] [deniedaccounts=<accounts>]

Parameters:

• partitionName (required): Name of the partition to update
• accounts (optional): Comma-separated list of accounts allowed to use the partition
• deniedaccounts (optional): Comma-separated list of accounts denied from using the partition

Note: When AllowedAccounts is set, DeniedAccounts will not take effect.

Examples:

# Allow specific accounts
ccontrol update partition=GPU accounts=research,staff

# Deny specific accounts
ccontrol update partition=CPU deniedaccounts=guest

[root@CraneProject CraneSched-FrontEnd]# ccontrol update partitionName=test accounts=test_user
ERRO[0000] Modify partition test failed: The entered account does not exist.
ERRO[0000] command execution failed

Hold and Release Commands

Hold Job

Hold specified job(s) to prevent them from starting.

ccontrol hold <job_id> [timelimit=<duration>]

Parameters:

• job_id (required): ID of the job to hold
• timelimit (optional): Duration to hold the job (format: HH:MM:SS)

Examples:

# Hold a job indefinitely
ccontrol hold 1

# Hold a job for 1 hour
ccontrol hold 1 timelimit=01:00:00

[root@CraneProject CraneSched-FrontEnd]# ccontrol hold 1 timelimit=01:00:00
Failed to modify job: 1. Reason: Task #1 was not found in pending queue..
ERRO[0000] hold jobs failed:
ERRO[0000] command execution failed

Release Job

Release a previously held job.

ccontrol release <job_id>

Example:

ccontrol release 1

[root@CraneProject CraneSched-FrontEnd]# ccontrol release 1
Failed to modify job: 1. Reason: Task #1 was not found in pending queue..
ERRO[0000] release jobs failed:
ERRO[0000] command execution failed

Create and Delete Commands

Create Reservation

Create a new resource reservation.

ccontrol create reservation <name> startTime=<time> duration=<duration> account=<account> [partition=<partition>] [nodes=<nodelist>] [nodeCnt=<count>] [user=<username>]

Parameters:

• name (required): Name of the reservation
• startTime (required): When the reservation starts (format: YYYY-MM-DDTHH:MM:SS)
• duration (required): Length of reservation (format: HH:MM:SS or D-HH:MM:SS)
• account (required): Account to associate with the reservation
• partition (optional): Partition to reserve resources from
• nodes (optional): Specific nodes to reserve (comma-separated list)
• nodeCnt (optional): Number of nodes to reserve (used when nodes is not specified)
• user (optional): User to associate with the reservation

Examples:

# Create a reservation with specific nodes
ccontrol create reservation my_reservation startTime=2024-12-01T10:00:00 duration=01:00:00 partition=CPU nodes=crane01,crane02 account=research

# Create a reservation with node count
ccontrol create reservation test_reservation startTime=2024-12-01T14:00:00 duration=02:00:00 partition=GPU nodeCnt=2 account=staff user=alice

[root@CraneProject CraneSched-FrontEnd]# ccontrol create reservation test_reservation duration=01:00:00 partition=test_partition nodes=test_node account=test_account
ERRO[0000] missing required fields: starttime          
ERRO[0000] command execution failed

Delete Reservation

Delete an existing reservation.

ccontrol delete reservation <name>

Example:

ccontrol delete reservation my_reservation

[root@CraneProject CraneSched-FrontEnd]# ccontrol delete reservation test_reservation
ERRO[0000] Failed to delete reservation: Reservation test_reservation not found.
ERRO[0000]
ERRO[0000] command execution failed

JSON Output

All commands support JSON output format using the --json flag:

ccontrol --json show node
ccontrol --json show partition CPU
ccontrol --json show job 12345

This is useful for scripting and automation purposes.

Permission Requirements

Different operations require different permission levels:

Operation	Permission Required
Show commands	User can view their own jobs; admin can view all
Update node	Administrator only
Update partition	Administrator only
Update job	Job owner or administrator
Hold/Release job	Job owner or administrator
Create reservation	Administrator or authorized user
Delete reservation	Reservation creator or administrator

Use Cases

Node Maintenance

# Drain node for maintenance
ccontrol update nodename=crane01 state=drain reason="Hardware upgrade"

# After maintenance, resume node
ccontrol update nodename=crane01 state=resume

Job Priority Management

# Increase priority for urgent job
ccontrol update jobid=1234 priority=1000

# Extend time limit for long-running job
ccontrol update jobid=1234 timelimit=10:00:00

Resource Reservation

# Reserve nodes for scheduled maintenance window
ccontrol create reservation maint_window startTime=2024-12-15T02:00:00 duration=04:00:00 partition=ALL nodes=crane[01-10] account=admin

# Delete reservation when no longer needed
ccontrol delete reservation maint_window

Related Commands

• cinfo - View cluster resource status
• cqueue - View job queue
• cacctmgr - Manage accounts and users
• cbatch - Submit batch jobs
• ccancel - Cancel jobs