veerdiff 0.1.0

Semantic diff engine for LLM and agent execution traces — find exactly where two runs veer apart.
Documentation
# veer

**A semantic diff engine for LLM & agent execution traces — find exactly where two runs veer apart.**

[![CI](https://github.com/zaydmulani09/veer/actions/workflows/ci.yml/badge.svg)](https://github.com/zaydmulani09/veer/actions/workflows/ci.yml)
[![crates.io](https://img.shields.io/crates/v/veerdiff.svg)](https://crates.io/crates/veerdiff)
[![docs.rs](https://img.shields.io/docsrs/veerdiff)](https://docs.rs/veerdiff)
[![license](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg)](#license)

---

You change a prompt, upgrade a model, or edit a tool, and your agent starts behaving differently. Which step actually changed? `diff`, `jq`, and snapshot tests are almost useless here: two runs of the same agent are never byte-identical. Request ids and timestamps churn, JSON keys come back in a different order, whitespace drifts, and "the same" tool call is rarely character-for-character equal.

`veer` diffs runs the way you actually reason about them — as an **ordered sequence of steps** (messages, tool calls, tool results) — and tells you where they *meaningfully* diverge, ignoring the noise.

```console
$ veer diff before.json after.json
veer  (- left / + right)
similarity 0.782  ·  2 equal, 3 changed, 2 added, 0 removed
first divergence at op 2

   = message system
   = message user
   ~ tool_call assistant · get_weather  (similarity 0.75)
        args.units: [-celsius-] {+fahrenheit+}
   ~ tool_result tool  (similarity 0.87)
        content: [-18C,-] {+64F,+} partly cloudy
   + tool_call assistant · get_air_quality  {"city":"Paris"}
   + tool_result tool  "AQI 42, good"
   ~ message assistant  (similarity 0.77)
        content: It's currently [-18°C-] {+64°F+} and partly cloudy in [-Paris.-] {+Paris,+} {+with+} {+good+} {+air+} {+quality.+}
```

Notice what it *didn't* flag: the `get_weather` arguments were `{"city","units"}` in one run and `{"units","city"}` in the other — reordered keys, not a real change — so only the `units` value shows up.

## Why it's different

`veer` is not a JSON differ with agent branding. Its core is an **alignment** algorithm: it pairs each step in run A with its closest counterpart in run B using an edit-distance dynamic program whose substitution cost is a *semantic similarity score*, not equality. That is what lets it say "these two tool calls are the same call with one changed argument" instead of "everything after step 3 is different."

- **Structural, not textual.** Steps are aligned; inserted/removed/changed/equal steps are reported as an edit script, with the first real divergence called out.
- **Noise-aware.** A configurable normalization pass masks volatile substrings (UUIDs, timestamps, emails, hashes), canonicalizes JSON key order, rounds floats, and drops metadata keys — *before* comparison.
- **Format-agnostic.** Imports OpenAI chat-completions, Anthropic Messages (including content blocks), and a native schema, all onto one normalized model, so you can even diff a run recorded by two different SDKs.
- **CI-ready.** `--check` turns "did behaviour drift?" into an exit code. `--format json` gives tooling a structured diff.
- **A library first.** The CLI is a thin shell over a typed Rust API (`veerdiff::diff`); the library has three small dependencies and no unsafe code.

## Install

```bash
# from crates.io (installs the `veer` command)
cargo install veerdiff

# or build from source
git clone https://github.com/zaydmulani09/veer
cd veer
cargo build --release   # binary at target/release/veer
```

As a library (the crate is `veerdiff`; it imports as `veerdiff`):

```toml
[dependencies]
veerdiff = "0.1"
# library-only, no CLI deps:
# veerdiff = { version = "0.1", default-features = false }
```

## CLI usage

```bash
veer diff <A> <B> [options]     # diff two traces
veer show <FILE> [options]      # import + normalize a trace, print the native schema
```

Common options for `diff`:

| Flag | Meaning |
| --- | --- |
| `-f, --format <human\|json\|summary>` | Output format (default `human`). |
| `-i, --input <auto\|veer\|openai\|anthropic>` | Input format (default `auto`). |
| `-c, --config <FILE>` | Load normalization/weights from a `.toml` or `.json` file. |
| `--mask <uuid\|timestamp\|email\|hex\|number>` | Apply a built-in masker (repeatable). |
| `--drop-meta <KEY>` | Drop a metadata key before comparison (repeatable). |
| `-v, --verbose` | Show unchanged steps too. |
| `--context <N>` | Unchanged steps of context around each change (default 2). |
| `--check` | Exit non-zero if the traces differ (for CI). |
| `--threshold <0..1>` | With `--check`, tolerate differences while similarity ≥ this. |
| `--color <auto\|always\|never>` | Colorize human output. |

### In CI

Fail a job when a run drifts too far from a golden trace:

```bash
veer diff golden.json candidate.json --check --threshold 0.95 --mask uuid --mask timestamp
```

## Library usage

```rust
use veerdiff::{Config, diff};
use veerdiff::model::{Step, StepKind, Trace};
use serde_json::json;

let before = Trace::from_steps(vec![
    Step::new(StepKind::ToolCall)
        .with_name("issue_refund")
        .with_args(json!({ "order_id": "12345", "amount": 49.99 })),
]);
let after = Trace::from_steps(vec![
    Step::new(StepKind::ToolCall)
        .with_name("issue_refund")
        .with_args(json!({ "order_id": "12345", "amount": 59.99 })),
]);

let d = diff(&before, &after, &Config::default());
assert_eq!(d.stats.changed, 1);
assert!(!d.is_identical());
// d.ops is a structured edit script; d.first_divergence points at op 0.
```

Importing provider traces:

```rust
use veerdiff::import::{import_str, Format};

let a = import_str(&std::fs::read_to_string("run_a.json")?, Format::Auto)?;
let b = import_str(&std::fs::read_to_string("run_b.json")?, Format::Auto)?;
```

See [`examples/programmatic.rs`](examples/programmatic.rs) for a runnable end-to-end example (`cargo run --example programmatic`).

## How it works

```
input (openai / anthropic / native)
        │  import.rs        ── normalize onto a common model
        ▼
   model::Trace  =  [ Step { kind, role, name, content, args, meta }, … ]
        │  normalize.rs     ── mask ids/timestamps, canonicalize JSON, drop noise
        ▼
   align.rs                 ── edit-distance DP with a semantic cost model
        │                      → equal / change / insert / delete edit script
        ▼
   report.rs                ── human (colored) · summary · json
```

The heart is `align.rs`. For traces of length *n* and *m* it fills a Wagner–Fischer cost matrix where:

- deleting or inserting a step costs `1`,
- pairing steps `a` and `b` costs `1 − similarity(a, b)`, and is only allowed when `similarity ≥ align_floor` (otherwise the optimal script prefers a delete + insert over a nonsensical pairing).

`similarity` is a weighted blend over the fields present on either step: exact match on `role`, and — via a Sørensen–Dice coefficient over character bigrams for text, and a recursive structural score for JSON — fuzzy match on `name`, `content`, and `args`. Steps of different *kind* are incomparable. Backtracking the matrix yields the edit script; each *changed* pairing then gets a nested diff (an inline word-level diff for text, a JSON-path diff for arguments).

## Configuration

Pass `--config veer.toml`. Every field is optional; defaults are conservative.

```toml
# veer.toml
equal_threshold = 0.999   # ≥ this similarity (and equal metadata) ⇒ "equal"
align_floor     = 0.4     # below this, steps are not paired

[normalizer]
canonicalize_json  = true     # sort JSON keys in args
trim_whitespace    = true
collapse_whitespace = false
lowercase          = false
round_numbers      = 2        # round floats in args to N decimals (omit to disable)
drop_meta          = ["request_id", "trace_id"]
mask               = ["uuid", "timestamp"]   # uuid|timestamp|email|hex|number
mask_meta          = false

[weights]
role = 1.0
name = 2.0
content = 3.0
args = 3.0
```

## Supported input formats

| Format | Detected shape | Notes |
| --- | --- | --- |
| `openai` | array of messages, or `{ "messages": [...] }` | `tool_calls` → tool-call steps; `role:"tool"` → tool-result steps; string `arguments` are parsed. |
| `anthropic` | messages with `content` block arrays | `text`, `tool_use`, `tool_result`, `thinking` blocks. |
| `veer` | `{ "steps": [...] }`, a bare `[Step]` array, or newline-delimited steps | The native schema (`veer show` prints it). |

## Performance

Alignment is `O(n·m)` in the number of steps. On a laptop (`cargo bench`), a full diff takes roughly **4 ms at 50 steps, 78 ms at 200, and 0.5 s at 500** — comfortably interactive for the traces agents actually produce. Beyond ~1,000 steps the quadratic term dominates; see the roadmap for planned linear-space alignment and bigram-profile precomputation.

## Limitations

- Alignment is `O(n·m)` in time and memory; it is built for traces up to ~1,000 steps, not millions. Hirschberg linear-space alignment and per-step similarity-profile caching are on the roadmap.
- Similarity is heuristic. The default weights and thresholds are sensible but not magic; tune them via config for your domain.
- Semantic *equivalence* of free text (paraphrases) is out of scope — `veer` compares surface form after normalization, not meaning via embeddings. (An optional embedding-backed matcher is a candidate for a future release.)

## Roadmap

- Linear-space (Hirschberg) alignment and cached per-step similarity profiles for very long traces.
- Custom regex maskers (behind a `regex` feature) and JSON-pointer masking.
- A `veer assert` mode with golden-file management.
- Optional embedding-based text similarity.
- More importers (LangSmithLangGraph, OpenTelemetry GenAI spans).

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). In short: `cargo fmt`, `cargo clippy --all-targets`, and `cargo test` should all be clean.

## License

Licensed under either of [MIT](LICENSE-MIT) or [Apache-2.0](LICENSE-APACHE) at your option.