swarm-engine-eval 0.1.3

# swarm-engine-eval

Scenario-based evaluation framework for SwarmEngine agent swarms.

## Usage

### Running from CLI (Recommended)

```bash
# From project root
cargo run --package swarm-engine-ui -- eval <SCENARIO_PATH>

# Example: Troubleshooting scenario
cargo run --package swarm-engine-ui -- eval crates/swarm-engine-eval/scenarios/troubleshooting.toml

# With options
cargo run --package swarm-engine-ui -- eval crates/swarm-engine-eval/scenarios/troubleshooting.toml \
    -n 5 \          # Number of runs (default: 1)
    -v \            # Verbose output (show tick snapshots)
    --learning      # Enable learning data collection
```

### CLI Options

| Option | Description |
|--------|-------------|
| `-n, --runs <N>` | Number of evaluation runs (default: 1) |
| `-s, --seed <SEED>` | Random seed (default: 42) |
| `-o, --output <FILE>` | JSON report output file |
| `-v, --verbose` | Verbose output with tick snapshots |
| `--learning` | Enable learning data collection |
| `--variant <NAME>` | Select scenario variant |
| `--list-variants` | List available variants |

## Scenarios

### Built-in Scenarios

Located in `scenarios/` directory:

| Scenario | Description |
|----------|-------------|
| `troubleshooting.toml` | Service diagnosis and recovery |
| `code_exploration.toml` | Codebase exploration |
| `search.toml` | Search tasks |
| `internal_diagnosis.toml` | Internal system diagnosis |

### Scenario Format

```toml
[meta]
name = "Service Troubleshooting"
id = "user:troubleshooting:v2"
version = "2.0.0"
description = "Diagnose and fix a service outage"
tags = ["troubleshooting", "diagnosis", "ops"]

[task]
goal = "Diagnose the failing service and restart it"
expected = "Worker successfully restarts the problematic service"

[task.context]
target_service = "user-service"
worker_count = 1

[llm]
provider = "llama-server"
model = "LFM2.5-1.2B"
endpoint = "http://localhost:8080"
temperature = 0.1
timeout_ms = 30000
max_tokens = 512

[manager]
process_every_tick = false
process_interval_ticks = 5
immediate_on_escalation = true
confidence_threshold = 0.3

[[actions.actions]]
name = "CheckStatus"
description = "Check the status of services"

[[actions.actions.params]]
name = "service"
description = "Optional: specific service name to check"
required = false

[[actions.actions]]
name = "ReadLogs"
description = "Read logs for a specific service"

[[actions.actions]]
name = "Diagnose"
description = "Diagnose the root cause of issues"

[[actions.actions]]
name = "Restart"
description = "Restart a service"
category = "node_state_change"

[app_config]
tick_duration_ms = 10
max_ticks = 150
```

### Scenario Variants

Scenarios can define variants for different configurations:

```toml
# List variants
cargo run --package swarm-engine-ui -- eval troubleshooting.toml --list-variants

# Run with variant
cargo run --package swarm-engine-ui -- eval troubleshooting.toml --variant complex
```

## Environment Types

| Type | Description |
|------|-------------|
| `troubleshooting` | Service troubleshooting simulation |
| `codebase` | File operation environment (Read/Write/Grep/Glob) |
| `none` | Empty environment (for testing) |

## Learning Integration

The eval system integrates with the offline learning system:

```bash
# 1. Collect learning data
cargo run --package swarm-engine-ui -- eval troubleshooting.toml -n 30 --learning

# 2. Run offline learning
cargo run --package swarm-engine-ui -- learn once troubleshooting

# 3. Next eval will use learned parameters
cargo run --package swarm-engine-ui -- eval troubleshooting.toml -n 5 -v
```

## Assertions

Scenarios can define assertions for pass/fail criteria:

```toml
[[assertions]]
name = "minimum_success_rate"
metric = "success_rate"
op = "gte"
expected = 0.5

[[assertions]]
name = "max_ticks_limit"
metric = "total_ticks"
op = "lte"
expected = 100
```

## Output

Eval produces:
- Console output with progress and results
- JSON report (with `-o` option)
- Learning data (with `--learning` option)
- Tick snapshots in verbose mode (with `-v` option)