veerdiff 0.1.0

Semantic diff engine for LLM and agent execution traces — find exactly where two runs veer apart.
Documentation

veer

A semantic diff engine for LLM & agent execution traces — find exactly where two runs veer apart.

CI crates.io docs.rs license


You change a prompt, upgrade a model, or edit a tool, and your agent starts behaving differently. Which step actually changed? diff, jq, and snapshot tests are almost useless here: two runs of the same agent are never byte-identical. Request ids and timestamps churn, JSON keys come back in a different order, whitespace drifts, and "the same" tool call is rarely character-for-character equal.

veer diffs runs the way you actually reason about them — as an ordered sequence of steps (messages, tool calls, tool results) — and tells you where they meaningfully diverge, ignoring the noise.

$ veer diff before.json after.json
veer  (- left / + right)
similarity 0.782  ·  2 equal, 3 changed, 2 added, 0 removed
first divergence at op 2

   = message system
   = message user
   ~ tool_call assistant · get_weather  (similarity 0.75)
        args.units: [-celsius-] {+fahrenheit+}
   ~ tool_result tool  (similarity 0.87)
        content: [-18C,-] {+64F,+} partly cloudy
   + tool_call assistant · get_air_quality  {"city":"Paris"}
   + tool_result tool  "AQI 42, good"
   ~ message assistant  (similarity 0.77)
        content: It's currently [-18°C-] {+64°F+} and partly cloudy in [-Paris.-] {+Paris,+} {+with+} {+good+} {+air+} {+quality.+}

Notice what it didn't flag: the get_weather arguments were {"city","units"} in one run and {"units","city"} in the other — reordered keys, not a real change — so only the units value shows up.

Why it's different

veer is not a JSON differ with agent branding. Its core is an alignment algorithm: it pairs each step in run A with its closest counterpart in run B using an edit-distance dynamic program whose substitution cost is a semantic similarity score, not equality. That is what lets it say "these two tool calls are the same call with one changed argument" instead of "everything after step 3 is different."

  • Structural, not textual. Steps are aligned; inserted/removed/changed/equal steps are reported as an edit script, with the first real divergence called out.
  • Noise-aware. A configurable normalization pass masks volatile substrings (UUIDs, timestamps, emails, hashes), canonicalizes JSON key order, rounds floats, and drops metadata keys — before comparison.
  • Format-agnostic. Imports OpenAI chat-completions, Anthropic Messages (including content blocks), and a native schema, all onto one normalized model, so you can even diff a run recorded by two different SDKs.
  • CI-ready. --check turns "did behaviour drift?" into an exit code. --format json gives tooling a structured diff.
  • A library first. The CLI is a thin shell over a typed Rust API (veerdiff::diff); the library has three small dependencies and no unsafe code.

Install

# from crates.io (installs the `veer` command)
cargo install veerdiff

# or build from source
git clone https://github.com/zaydmulani09/veer
cd veer
cargo build --release   # binary at target/release/veer

As a library (the crate is veerdiff; it imports as veerdiff):

[dependencies]
veerdiff = "0.1"
# library-only, no CLI deps:
# veerdiff = { version = "0.1", default-features = false }

CLI usage

veer diff <A> <B> [options]     # diff two traces
veer show <FILE> [options]      # import + normalize a trace, print the native schema

Common options for diff:

Flag Meaning
-f, --format <human|json|summary> Output format (default human).
-i, --input <auto|veer|openai|anthropic> Input format (default auto).
-c, --config <FILE> Load normalization/weights from a .toml or .json file.
--mask <uuid|timestamp|email|hex|number> Apply a built-in masker (repeatable).
--drop-meta <KEY> Drop a metadata key before comparison (repeatable).
-v, --verbose Show unchanged steps too.
--context <N> Unchanged steps of context around each change (default 2).
--check Exit non-zero if the traces differ (for CI).
--threshold <0..1> With --check, tolerate differences while similarity ≥ this.
--color <auto|always|never> Colorize human output.

In CI

Fail a job when a run drifts too far from a golden trace:

veer diff golden.json candidate.json --check --threshold 0.95 --mask uuid --mask timestamp

Library usage

use veerdiff::{Config, diff};
use veerdiff::model::{Step, StepKind, Trace};
use serde_json::json;

let before = Trace::from_steps(vec![
    Step::new(StepKind::ToolCall)
        .with_name("issue_refund")
        .with_args(json!({ "order_id": "12345", "amount": 49.99 })),
]);
let after = Trace::from_steps(vec![
    Step::new(StepKind::ToolCall)
        .with_name("issue_refund")
        .with_args(json!({ "order_id": "12345", "amount": 59.99 })),
]);

let d = diff(&before, &after, &Config::default());
assert_eq!(d.stats.changed, 1);
assert!(!d.is_identical());
// d.ops is a structured edit script; d.first_divergence points at op 0.

Importing provider traces:

use veerdiff::import::{import_str, Format};

let a = import_str(&std::fs::read_to_string("run_a.json")?, Format::Auto)?;
let b = import_str(&std::fs::read_to_string("run_b.json")?, Format::Auto)?;

See examples/programmatic.rs for a runnable end-to-end example (cargo run --example programmatic).

How it works

input (openai / anthropic / native)
        │  import.rs        ── normalize onto a common model
        ▼
   model::Trace  =  [ Step { kind, role, name, content, args, meta }, … ]
        │  normalize.rs     ── mask ids/timestamps, canonicalize JSON, drop noise
        ▼
   align.rs                 ── edit-distance DP with a semantic cost model
        │                      → equal / change / insert / delete edit script
        ▼
   report.rs                ── human (colored) · summary · json

The heart is align.rs. For traces of length n and m it fills a Wagner–Fischer cost matrix where:

  • deleting or inserting a step costs 1,
  • pairing steps a and b costs 1 − similarity(a, b), and is only allowed when similarity ≥ align_floor (otherwise the optimal script prefers a delete + insert over a nonsensical pairing).

similarity is a weighted blend over the fields present on either step: exact match on role, and — via a Sørensen–Dice coefficient over character bigrams for text, and a recursive structural score for JSON — fuzzy match on name, content, and args. Steps of different kind are incomparable. Backtracking the matrix yields the edit script; each changed pairing then gets a nested diff (an inline word-level diff for text, a JSON-path diff for arguments).

Configuration

Pass --config veer.toml. Every field is optional; defaults are conservative.

# veer.toml
equal_threshold = 0.999   # ≥ this similarity (and equal metadata) ⇒ "equal"
align_floor     = 0.4     # below this, steps are not paired

[normalizer]
canonicalize_json  = true     # sort JSON keys in args
trim_whitespace    = true
collapse_whitespace = false
lowercase          = false
round_numbers      = 2        # round floats in args to N decimals (omit to disable)
drop_meta          = ["request_id", "trace_id"]
mask               = ["uuid", "timestamp"]   # uuid|timestamp|email|hex|number
mask_meta          = false

[weights]
role = 1.0
name = 2.0
content = 3.0
args = 3.0

Supported input formats

Format Detected shape Notes
openai array of messages, or { "messages": [...] } tool_calls → tool-call steps; role:"tool" → tool-result steps; string arguments are parsed.
anthropic messages with content block arrays text, tool_use, tool_result, thinking blocks.
veer { "steps": [...] }, a bare [Step] array, or newline-delimited steps The native schema (veer show prints it).

Performance

Alignment is O(n·m) in the number of steps. On a laptop (cargo bench), a full diff takes roughly 4 ms at 50 steps, 78 ms at 200, and 0.5 s at 500 — comfortably interactive for the traces agents actually produce. Beyond ~1,000 steps the quadratic term dominates; see the roadmap for planned linear-space alignment and bigram-profile precomputation.

Limitations

  • Alignment is O(n·m) in time and memory; it is built for traces up to ~1,000 steps, not millions. Hirschberg linear-space alignment and per-step similarity-profile caching are on the roadmap.
  • Similarity is heuristic. The default weights and thresholds are sensible but not magic; tune them via config for your domain.
  • Semantic equivalence of free text (paraphrases) is out of scope — veer compares surface form after normalization, not meaning via embeddings. (An optional embedding-backed matcher is a candidate for a future release.)

Roadmap

  • Linear-space (Hirschberg) alignment and cached per-step similarity profiles for very long traces.
  • Custom regex maskers (behind a regex feature) and JSON-pointer masking.
  • A veer assert mode with golden-file management.
  • Optional embedding-based text similarity.
  • More importers (LangSmithLangGraph, OpenTelemetry GenAI spans).

Contributing

Contributions are welcome — see CONTRIBUTING.md. In short: cargo fmt, cargo clippy --all-targets, and cargo test should all be clean.

License

Licensed under either of MIT or Apache-2.0 at your option.