veer
A semantic diff engine for LLM & agent execution traces — find exactly where two runs veer apart.
You change a prompt, upgrade a model, or edit a tool, and your agent starts behaving differently. Which step actually changed? diff, jq, and snapshot tests are almost useless here: two runs of the same agent are never byte-identical. Request ids and timestamps churn, JSON keys come back in a different order, whitespace drifts, and "the same" tool call is rarely character-for-character equal.
veer diffs runs the way you actually reason about them — as an ordered sequence of steps (messages, tool calls, tool results) — and tells you where they meaningfully diverge, ignoring the noise.
$ veer diff before.json after.json
veer (- left / + right)
similarity 0.782 · 2 equal, 3 changed, 2 added, 0 removed
first divergence at op 2
= message system
= message user
~ tool_call assistant · get_weather (similarity 0.75)
args.units: [-celsius-] {+fahrenheit+}
~ tool_result tool (similarity 0.87)
content: [-18C,-] {+64F,+} partly cloudy
+ tool_call assistant · get_air_quality {"city":"Paris"}
+ tool_result tool "AQI 42, good"
~ message assistant (similarity 0.77)
content: It's currently [-18°C-] {+64°F+} and partly cloudy in [-Paris.-] {+Paris,+} {+with+} {+good+} {+air+} {+quality.+}
Notice what it didn't flag: the get_weather arguments were {"city","units"} in one run and {"units","city"} in the other — reordered keys, not a real change — so only the units value shows up.
Why it's different
veer is not a JSON differ with agent branding. Its core is an alignment algorithm: it pairs each step in run A with its closest counterpart in run B using an edit-distance dynamic program whose substitution cost is a semantic similarity score, not equality. That is what lets it say "these two tool calls are the same call with one changed argument" instead of "everything after step 3 is different."
- Structural, not textual. Steps are aligned; inserted/removed/changed/equal steps are reported as an edit script, with the first real divergence called out.
- Noise-aware. A configurable normalization pass masks volatile substrings (UUIDs, timestamps, emails, hashes), canonicalizes JSON key order, rounds floats, and drops metadata keys — before comparison.
- Format-agnostic. Imports OpenAI chat-completions, Anthropic Messages (including content blocks), and a native schema, all onto one normalized model, so you can even diff a run recorded by two different SDKs.
- CI-ready.
--checkturns "did behaviour drift?" into an exit code.--format jsongives tooling a structured diff. - A library first. The CLI is a thin shell over a typed Rust API (
veerdiff::diff); the library has three small dependencies and no unsafe code.
Install
# from crates.io (installs the `veer` command)
# or build from source
As a library (the crate is veerdiff; it imports as veerdiff):
[]
= "0.1"
# library-only, no CLI deps:
# veerdiff = { version = "0.1", default-features = false }
CLI usage
Common options for diff:
| Flag | Meaning |
|---|---|
-f, --format <human|json|summary> |
Output format (default human). |
-i, --input <auto|veer|openai|anthropic> |
Input format (default auto). |
-c, --config <FILE> |
Load normalization/weights from a .toml or .json file. |
--mask <uuid|timestamp|email|hex|number> |
Apply a built-in masker (repeatable). |
--drop-meta <KEY> |
Drop a metadata key before comparison (repeatable). |
-v, --verbose |
Show unchanged steps too. |
--context <N> |
Unchanged steps of context around each change (default 2). |
--check |
Exit non-zero if the traces differ (for CI). |
--threshold <0..1> |
With --check, tolerate differences while similarity ≥ this. |
--color <auto|always|never> |
Colorize human output. |
In CI
Fail a job when a run drifts too far from a golden trace:
Library usage
use ;
use ;
use json;
let before = from_steps;
let after = from_steps;
let d = diff;
assert_eq!;
assert!;
// d.ops is a structured edit script; d.first_divergence points at op 0.
Importing provider traces:
use ;
let a = import_str?;
let b = import_str?;
See examples/programmatic.rs for a runnable end-to-end example (cargo run --example programmatic).
How it works
input (openai / anthropic / native)
│ import.rs ── normalize onto a common model
▼
model::Trace = [ Step { kind, role, name, content, args, meta }, … ]
│ normalize.rs ── mask ids/timestamps, canonicalize JSON, drop noise
▼
align.rs ── edit-distance DP with a semantic cost model
│ → equal / change / insert / delete edit script
▼
report.rs ── human (colored) · summary · json
The heart is align.rs. For traces of length n and m it fills a Wagner–Fischer cost matrix where:
- deleting or inserting a step costs
1, - pairing steps
aandbcosts1 − similarity(a, b), and is only allowed whensimilarity ≥ align_floor(otherwise the optimal script prefers a delete + insert over a nonsensical pairing).
similarity is a weighted blend over the fields present on either step: exact match on role, and — via a Sørensen–Dice coefficient over character bigrams for text, and a recursive structural score for JSON — fuzzy match on name, content, and args. Steps of different kind are incomparable. Backtracking the matrix yields the edit script; each changed pairing then gets a nested diff (an inline word-level diff for text, a JSON-path diff for arguments).
Configuration
Pass --config veer.toml. Every field is optional; defaults are conservative.
# veer.toml
= 0.999 # ≥ this similarity (and equal metadata) ⇒ "equal"
= 0.4 # below this, steps are not paired
[]
= true # sort JSON keys in args
= true
= false
= false
= 2 # round floats in args to N decimals (omit to disable)
= ["request_id", "trace_id"]
= ["uuid", "timestamp"] # uuid|timestamp|email|hex|number
= false
[]
= 1.0
= 2.0
= 3.0
= 3.0
Supported input formats
| Format | Detected shape | Notes |
|---|---|---|
openai |
array of messages, or { "messages": [...] } |
tool_calls → tool-call steps; role:"tool" → tool-result steps; string arguments are parsed. |
anthropic |
messages with content block arrays |
text, tool_use, tool_result, thinking blocks. |
veer |
{ "steps": [...] }, a bare [Step] array, or newline-delimited steps |
The native schema (veer show prints it). |
Performance
Alignment is O(n·m) in the number of steps. On a laptop (cargo bench), a full diff takes roughly 4 ms at 50 steps, 78 ms at 200, and 0.5 s at 500 — comfortably interactive for the traces agents actually produce. Beyond ~1,000 steps the quadratic term dominates; see the roadmap for planned linear-space alignment and bigram-profile precomputation.
Limitations
- Alignment is
O(n·m)in time and memory; it is built for traces up to ~1,000 steps, not millions. Hirschberg linear-space alignment and per-step similarity-profile caching are on the roadmap. - Similarity is heuristic. The default weights and thresholds are sensible but not magic; tune them via config for your domain.
- Semantic equivalence of free text (paraphrases) is out of scope —
veercompares surface form after normalization, not meaning via embeddings. (An optional embedding-backed matcher is a candidate for a future release.)
Roadmap
- Linear-space (Hirschberg) alignment and cached per-step similarity profiles for very long traces.
- Custom regex maskers (behind a
regexfeature) and JSON-pointer masking. - A
veer assertmode with golden-file management. - Optional embedding-based text similarity.
- More importers (LangSmithLangGraph, OpenTelemetry GenAI spans).
Contributing
Contributions are welcome — see CONTRIBUTING.md. In short: cargo fmt, cargo clippy --all-targets, and cargo test should all be clean.
License
Licensed under either of MIT or Apache-2.0 at your option.