prodigy 0.4.4

Turn ad-hoc Claude sessions into reproducible development pipelines with parallel AI agents
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
## Workflow-Level vs Command-Level Retry

Prodigy has two distinct retry systems that serve different purposes. Understanding when to use each is critical for effective error handling.

### Overview

| Feature | Command-Level (retry_v2) | Workflow-Level (error_policy) |
|---------|--------------------------|--------------------------------|
| **Scope** | Individual command execution | Work item failure in MapReduce |
| **Location** | `retry_config` on commands | `error_policy` + `retry_config` in workflow |
| **Implementation** | `src/cook/retry_v2.rs` | `src/cook/workflow/error_policy.rs` |
| **Use Case** | Retry transient command failures | Retry failed MapReduce work items |
| **Features** | Backoff, jitter, error matchers, circuit breakers | DLQ integration, failure thresholds, batch collection |

### Command-Level Retry (retry_v2)

The **enhanced retry system** (`retry_v2`) provides sophisticated retry capabilities for individual command execution.

**Source**: `src/cook/retry_v2.rs`

#### Key Features

1. **Multiple Backoff Strategies**:
   - Exponential (default)
   - Linear
   - Fibonacci
   - Fixed
   - Custom

2. **Error Matchers**:
   - Selective retry based on error type
   - Built-in matchers: Network, Timeout, ServerError, RateLimit
   - Custom regex patterns

3. **Jitter Support**:
   - Prevents thundering herd in distributed systems
   - Configurable jitter factor (default: 0.3)

4. **Retry Budget**:
   - Time-based caps on total retry time
   - Prevents infinite retry loops

5. **Failure Actions**:
   - Stop (default) - halt workflow
   - Continue - proceed despite failure
   - Fallback - execute alternative command

6. **Circuit Breakers**:
   - Fail-fast when downstream is down
   - Automatic recovery testing (HalfOpen state)

#### Configuration

```yaml
commands:
  - shell: "curl https://api.example.com/data"
    retry_config:
      attempts: 5
      backoff: exponential
      initial_delay: "1s"
      max_delay: "30s"
      jitter: true
      jitter_factor: 0.3
      retry_on:
        - network
        - timeout
      retry_budget: "5m"
      on_failure: stop
```

**RetryConfig Fields** (src/cook/retry_v2.rs:14-52):
- `attempts: u32` - Maximum retry attempts
- `backoff: BackoffStrategy` - Delay calculation strategy
- `initial_delay: Duration` - Starting delay
- `max_delay: Duration` - Delay cap
- `jitter: bool` - Enable jitter
- `jitter_factor: f64` - Jitter randomization (0.0-1.0)
- `retry_on: Vec<ErrorMatcher>` - Selective retry matchers
- `retry_budget: Option<Duration>` - Total retry time limit
- `on_failure: FailureAction` - Final failure handling

#### When to Use Command-Level Retry

Use `retry_config` on individual commands for:
- **External API calls** with transient failures
- **Network operations** that might timeout
- **Database operations** with lock conflicts
- **Resource initialization** that needs retry
- Any **single command** that benefits from retry

**Example**:
```yaml
commands:
  - shell: "make build"
    retry_config:
      attempts: 3
      retry_on:
        - network  # Only retry network errors during dependency fetch
```

### Workflow-Level Retry (error_policy)

The **workflow-level retry system** handles failures in MapReduce work items, integrating with the Dead Letter Queue (DLQ).

**Source**: `src/cook/workflow/error_policy.rs`

#### Key Features

1. **Work Item Failure Handling**:
   - DLQ (Dead Letter Queue) - Save failed items for retry
   - Retry - Immediate retry with backoff
   - Skip - Skip failed item, continue
   - Stop - Stop entire workflow
   - Custom - Custom failure handler

2. **Failure Thresholds**:
   - `max_failures` - Stop after N failures
   - `failure_threshold` - Stop at failure rate (0.0-1.0)

3. **Error Collection Strategies**:
   - Aggregate - Collect all errors before reporting
   - Immediate - Report errors as they occur
   - Batched - Report in batches of N errors

4. **Circuit Breaker Integration**:
   - Workflow-level circuit breaker
   - Failure/success thresholds
   - Half-open request limits

5. **DLQ Retry**:
   - Failed items stored in DLQ
   - Retry with `prodigy dlq retry <job_id>`
   - Preserves correlation IDs

#### Configuration

```yaml
name: mapreduce-workflow
mode: mapreduce

error_policy:
  on_item_failure: dlq          # Send failures to DLQ
  continue_on_failure: true     # Keep processing other items
  max_failures: 10              # Stop after 10 failures
  failure_threshold: 0.25       # Or stop at 25% failure rate
  error_collection: aggregate   # Collect errors before reporting
  circuit_breaker:
    failure_threshold: 5        # Open after 5 failures
    success_threshold: 3        # Close after 3 successes
    timeout: "30s"              # Recovery timeout
    half_open_requests: 3       # Test requests in half-open
  retry_config:                 # Workflow-level retry (simpler)
    max_attempts: 3
    backoff: exponential

map:
  input: "items.json"
  json_path: "$.items[*]"
  agent_template:
    - shell: "process ${item.id}"
```

**WorkflowErrorPolicy Fields** (src/cook/workflow/error_policy.rs:131-179):
- `on_item_failure: ItemFailureAction` - What to do when item fails
- `continue_on_failure: bool` - Continue processing other items
- `max_failures: Option<usize>` - Maximum failures before stopping
- `failure_threshold: Option<f64>` - Failure rate threshold (0.0-1.0)
- `error_collection: ErrorCollectionStrategy` - Error reporting strategy
- `circuit_breaker: Option<CircuitBreakerConfig>` - Circuit breaker config
- `retry_config: Option<RetryConfig>` - Simplified retry config

**Workflow-Level RetryConfig** (src/cook/workflow/error_policy.rs:90-129):
- `max_attempts: u32` - Maximum retry attempts
- `backoff: BackoffStrategy` - Simpler backoff variants

#### When to Use Workflow-Level Retry

Use `error_policy` in MapReduce workflows for:
- **Work item failure handling** - DLQ, thresholds, collection
- **MapReduce workflows** - Parallel work item processing
- **Failure rate monitoring** - Stop at threshold percentage
- **Batch error handling** - Collect and report errors in batches

**Example**:
```yaml
error_policy:
  on_item_failure: dlq        # Failed items go to DLQ
  continue_on_failure: true   # Process other items
  max_failures: 5             # But stop if more than 5 fail
```

### Using Both Systems Together

You can combine both retry systems for comprehensive error handling:

```yaml
name: robust-mapreduce
mode: mapreduce

# Workflow-level error policy
error_policy:
  on_item_failure: dlq        # Failed items to DLQ
  continue_on_failure: true   # Keep processing
  max_failures: 10            # Stop after 10 failures
  circuit_breaker:
    failure_threshold: 5
    timeout: "30s"

map:
  input: "items.json"
  json_path: "$.items[*]"
  max_parallel: 10

  agent_template:
    # Command-level retry for transient failures
    - shell: "process ${item.id}"
      retry_config:
        attempts: 3              # Try 3 times per agent
        backoff: exponential
        initial_delay: "1s"
        max_delay: "30s"
        jitter: true             # Prevent thundering herd
        retry_on:
          - network
          - timeout
        on_failure: stop         # Let workflow error_policy handle final failure
```

**How they work together**:

1. **Agent attempts to process work item**
2. **Command fails**`retry_config` retries with exponential backoff (up to 3 attempts)
3. **All command retries fail**`on_failure: stop` ends agent execution
4. **Agent reports failure to workflow**`error_policy` sends item to DLQ
5. **Workflow continues processing other items**`continue_on_failure: true`
6. **After map phase** → Retry DLQ items with `prodigy dlq retry <job_id>`

**Benefits**:
- **Fast recovery** from transient errors (command-level retry)
- **Isolation** of persistent failures (DLQ)
- **Continued processing** of other items (workflow-level policy)
- **Manual review** of failed items before retry

### Key Differences

#### Scope of Retry

**Command-Level**: Single command execution
```yaml
commands:
  - shell: "api-call.sh"
    retry_config:
      attempts: 5
```
→ Retries the `api-call.sh` command 5 times

**Workflow-Level**: Entire work item (may contain multiple commands)
```yaml
map:
  agent_template:
    - shell: "step1.sh ${item.id}"
    - shell: "step2.sh ${item.id}"
    - shell: "step3.sh ${item.id}"
```
→ If agent fails, entire work item (all 3 steps) sent to DLQ for retry

#### Retry Granularity

**Command-Level**: Immediate retry after command failure
- Retry happens in same agent execution
- Backoff delays between retries
- Same environment/context

**Workflow-Level**: Work item retry after agent failure
- Retry happens in new agent execution (via DLQ)
- Fresh environment/context
- Manual triggering with `prodigy dlq retry`

#### Feature Comparison

| Feature | Command-Level | Workflow-Level |
|---------|---------------|----------------|
| Backoff strategies | 5 strategies | 4 strategies |
| Jitter support | ✅ Yes | ❌ No |
| Error matchers | ✅ Yes | ❌ No |
| Retry budget | ✅ Yes | ❌ No |
| Failure actions | ✅ Yes (Stop/Continue/Fallback) | ❌ No (uses error_policy) |
| Circuit breakers | ✅ Yes (per RetryExecutor) | ✅ Yes (per workflow) |
| DLQ integration | ❌ No | ✅ Yes |
| Failure thresholds | ❌ No | ✅ Yes |
| Error collection | ❌ No | ✅ Yes (Aggregate/Immediate/Batched) |

### Decision Tree

**Use command-level retry** when:
- You need fine-grained control over which errors to retry
- You want jitter to prevent thundering herd
- You need retry budget to cap total time
- You want fallback commands on failure
- You're configuring a single command, not a work item

**Use workflow-level retry** when:
- You're running a MapReduce workflow
- You want DLQ integration for failed work items
- You need failure rate thresholds (stop at 25% failure)
- You want batch error collection
- You want to retry entire work items later

**Use both** when:
- You want fast recovery from transient errors (command-level)
- AND you want to isolate persistent failures (DLQ)
- AND you want to continue processing other items

### Examples

#### Example 1: Command-Level Only (Standard Workflow)

```yaml
name: standard-workflow
mode: standard

commands:
  - shell: "fetch-data.sh"
    retry_config:
      attempts: 5
      backoff: exponential
      retry_on:
        - network
        - timeout

  - shell: "process-data.sh"
    retry_config:
      attempts: 3
      on_failure: continue  # Non-critical, can fail

  - shell: "upload-results.sh"
    retry_config:
      attempts: 5
      backoff: exponential
      retry_on:
        - network
        - server_error
      on_failure:
        fallback:
          command: "save-to-local.sh"
```

#### Example 2: Workflow-Level Only (Simple MapReduce)

```yaml
name: simple-mapreduce
mode: mapreduce

error_policy:
  on_item_failure: dlq
  continue_on_failure: true
  max_failures: 10

map:
  input: "items.json"
  json_path: "$.items[*]"
  agent_template:
    - shell: "process ${item.id}"
    # No retry_config - relies on DLQ for failed items
```

#### Example 3: Both Systems (Robust MapReduce)

```yaml
name: robust-mapreduce
mode: mapreduce

error_policy:
  on_item_failure: dlq
  continue_on_failure: true
  max_failures: 10
  circuit_breaker:
    failure_threshold: 5
    timeout: "30s"

map:
  input: "items.json"
  json_path: "$.items[*]"
  max_parallel: 10

  agent_template:
    - shell: "process ${item.id}"
      retry_config:
        attempts: 3           # Try 3 times quickly
        backoff: exponential
        jitter: true          # Prevent simultaneous retries
        retry_on:
          - network
          - timeout
        on_failure: stop      # Let DLQ handle final failure

reduce:
  - shell: "aggregate ${map.results}"
```

**Retry flow**:
1. Command fails with network error
2. Command retries (attempt 2, 3) with exponential backoff
3. All command retries fail → Agent fails
4. Work item sent to DLQ
5. Other work items continue processing
6. After workflow completes → `prodigy dlq retry <job_id>` to retry failed items

### See Also

- [Basic Retry Configuration](./basic-retry-configuration.md) - Command-level retry config
- [Backoff Strategies](./backoff-strategies.md) - Backoff strategy details
- [Best Practices](./best-practices.md) - When to use each system
- [Complete Examples](./complete-examples.md) - Full workflow examples