torc 0.21.0

Workflow management system
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

<div class="cheatsheet">

# Torc CLI Cheat Sheet

## Quick Start

| Command                                        | Description                                    |
| ---------------------------------------------- | ---------------------------------------------- |
| `torc workflows create <spec>`                 | Create from spec file                          |
| `torc run <spec.yaml>`                         | Create workflow from spec and run locally      |
| `torc submit <spec.yaml>`                      | Create and submit to scheduler (needs actions) |
| `torc submit-slurm --account ACCT <spec.yaml>` | Auto-generate Slurm schedulers and submit      |
| `torc reports summary <id>`                    | Workflow completion summary                    |
| `torc watch <id>`                              | Monitor workflow until completion              |
| `torc watch <id> --recover`                    | Monitor and auto-recover from failures         |
| `torc-dash`                                    | Launch web dashboard                           |
| `torc tui`                                     | Launch interactive terminal UI                 |

## Managing Workflows

| Command                      | Description                    |
| ---------------------------- | ------------------------------ |
| `torc workflows list`        | List your workflows            |
| `torc workflows status <id>` | Get job counts by status       |
| `torc workflows get <id>`    | Get workflow details           |
| `torc workflows cancel <id>` | Cancel workflow and Slurm jobs |
| `torc workflows delete <id>` | Delete workflow                |

## Job Management

| Command                           | Description         |
| --------------------------------- | ------------------- |
| `torc jobs list <id>`             | List all jobs       |
| `torc jobs list -s ready <id>`    | List jobs by status |
| `torc jobs get <job_id>`          | Get job details     |
| `torc results list <id>`          | List job results    |
| `torc results list --failed <id>` | List failed jobs    |

## Recovery & Diagnostics

| Command                                        | Description                                    |
| ---------------------------------------------- | ---------------------------------------------- |
| `torc reports summary <id>`                    | Workflow completion summary                    |
| `torc reports check-resource-utilization <id>` | Check memory/CPU/time usage                    |
| `torc reports results <id>`                    | JSON report of job results with log paths      |
| `torc recover <id>`                            | One-shot recovery (diagnose + fix + resubmit)  |
| `torc watch <id> --recover --auto-schedule`    | Full production recovery mode                  |
| `torc workflows sync-status <id>`              | Fix orphaned jobs (stuck in "running")         |
| `torc workflows correct-resources <id>`        | Upscale violated + downsize over-allocated RRs |
| `torc slurm sacct <id>`                        | Get Slurm accounting data                      |
| `torc slurm stats <id>`                        | Per-job sacct stats stored in the database     |
| `torc slurm usage <id>`                        | Total compute node and CPU time consumed       |

## Remote Workers

| Command                                  | Description                              |
| ---------------------------------------- | ---------------------------------------- |
| `torc remote add-workers <id> <host>...` | Add remote workers to a workflow         |
| `torc remote list-workers <id>`          | List remote workers for a workflow       |
| `torc remote run <id>`                   | Start workers on remote machines via SSH |
| `torc remote status <id>`                | Check status of remote workers           |
| `torc remote stop <id>`                  | Stop workers on remote machines          |
| `torc remote collect-logs <id>`          | Collect logs from remote workers         |

## Events & Logs

| Command                    | Description                 |
| -------------------------- | --------------------------- |
| `torc events monitor <id>` | Monitor events in real-time |
| `torc logs analyze <id>`   | Analyze logs for errors     |

## Global Options

| Option              | Description                        |
| ------------------- | ---------------------------------- |
| `--url <URL>`       | Server URL (or set `TORC_API_URL`) |
| `-f json`           | Output as JSON instead of table    |
| `--log-level debug` | Enable debug logging               |

</div>