torc 0.22.2

Workflow management system
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

# How to Parameterize Jobs with Files

Process multiple input files by combining parameterization with file path templating.

## Basic Pattern

Use a parameter to generate jobs for each file:

```yaml
name: process_files

jobs:
  - name: process_{dataset}
    command: python process.py --input data/{dataset}.csv --output results/{dataset}.json
    parameters:
      dataset: "[train, test, validation]"
```

This creates 3 jobs:

- `process_train` → processes `data/train.csv`
- `process_test` → processes `data/test.csv`
- `process_validation` → processes `data/validation.csv`

## With File Dependencies

Combine parameterization with explicit file definitions for dependency tracking:

```yaml
name: file_pipeline

files:
  - name: raw_{dataset}
    path: data/{dataset}.csv
  - name: processed_{dataset}
    path: results/{dataset}.json

jobs:
  - name: process_{dataset}
    command: python process.py -i ${files.input.raw_{dataset}} -o ${files.output.processed_{dataset}}
    parameters:
      dataset: "[train, test, validation]"

  - name: aggregate
    command: python aggregate.py --input results/ --output summary.json
    depends_on:
      - process_{dataset}
    parameters:
      dataset: "[train, test, validation]"
```

The `aggregate` job automatically waits for all `process_*` jobs to complete.

## Processing Numbered Files

Use range syntax for numbered file sequences:

```yaml
jobs:
  - name: convert_{i:03d}
    command: ffmpeg -i video_{i:03d}.mp4 -o audio_{i:03d}.mp3
    parameters:
      i: "1:100"
```

Creates jobs for `video_001.mp4` through `video_100.mp4`.

## Multi-Dimensional Sweeps

Combine multiple parameters for Cartesian product expansion:

```yaml
jobs:
  - name: analyze_{region}_{year}
    command: python analyze.py --region {region} --year {year} --output results/{region}_{year}.json
    parameters:
      region: "[north, south, east, west]"
      year: "2020:2024"
```

Creates 20 jobs (4 regions × 5 years).

## See Also

- [Simple Parameterization](../tutorials/simple-params.md) — Basic parameter tutorial
- [Advanced Parameterization](../tutorials/advanced-params.md) — Multi-dimensional sweeps
- [Job Parameterization Reference](../reference/parameterization.md) — Complete syntax