agentic-eval 0.14.4

# Changelog — agentic-eval

All notable changes to the `agentic-eval` crate. Follows
[Keep a Changelog](https://keepachangelog.com/) and [SemVer](https://semver.org/).

## [0.14.4] - 2026-07-29

### Added
- **`sigil_audit` example** — measures every AetherShell agentic-mode construct
  against the real cl100k/o200k BPE, tests candidate alternative encodings
  against the incumbent, and records an explicit **adopted/rejected**
  disposition with a reason for each cheaper candidate, so an encoding rejected
  as ambiguous is not re-proposed on the next run. It produced the v3 bare-dot
  lambda (`|w.size>1k`), a measured **23.7% token reduction on predicate
  pipelines**, and reports honest non-results too (`F.r("p")`, `DK.p()` show no
  win over their verbose forms).

  Run with:
  `cargo run -p agentic-eval --example sigil_audit --features real-tokens`

### Changed
- **README now separates measured from curated.** The crate's two headline
  benchmarks (VM/sandbox systems, web stacks) are hand-assigned expert
  judgments, not measurements, and Nervosys's own entries rank first in both —
  yet they sat in the same document as genuinely measured BPE token counts. A
  new "Epistemic status" section names which examples produce reproducible
  measurements and which encode judgment, both benchmarks are retitled
  "(curated)", and the conflict of interest is stated plainly rather than left
  for the reader to infer.

## [0.14.2] - 2026-06-08

### Added
- **SPINE profile now also cites the neural codec benchmark.** A seventh
  evidence string on `WebStack::Spine` references SPINE's neural
  encoder-decoder benchmark (`spine-protocol/benches/neural_codec_bench.rs`,
  2026-06-08): the real `TitansLatentCodec` projects text into a Titans latent
  and frames it as a self-describing `EncodedFrame` that is **66–71 % smaller
  than its JSON form** (dim 256: 1241 B vs 3942 B; dim 1024: 4314 B vs
  14803 B). The honest cost is recorded too — it is a genuine Titans forward
  pass (superlinear encode: ~94 µs at dim 128 to ~26 ms at dim 1024), a
  one-time sender-side price separate from the wire-size win. **Scores
  unchanged** — this substantiates the existing `encoding_efficiency` 0.95 for
  the latent data plane; it does not move it. Directional tests unchanged.

## [0.14.1] - 2026-06-08

### Added
- **SPINE profile now includes the measured transport benchmarks.** A sixth
  evidence string on `WebStack::Spine` cites SPINE's own agentic web-stack
  benchmarks (`src/spine-transport/benches/{spine_vs_http2,agentic_ai_workload,
  llm_tok_per_sec}`, re-run 2026-06-08 against the real `h2` HTTP/2 crate):
  single-stream latency 1.6–2.4× / throughput 1.8–2.3×, ~32× on N=64 pipelined
  multiplexing (≈1.3M req/s), embedding batches ~6–25× over HTTP/2+JSON, and
  token streaming at hundreds of M tok/s (9–15× over HTTP/2+binary at large
  batches) where JSON-SSE caps near ~10M. Honest framing kept: TCP-loopback
  medians whose absolutes are machine-dependent; direction/order-of-magnitude
  reproduce. **Scores unchanged** — the benchmarks substantiate the existing
  `encoding_efficiency` 0.95 and `streaming` 0.98, they do not move them.
  Directional tests unchanged (all still hold).

## [0.14.0] - 2026-06-08

### Changed
- **SPINE re-scored after SPINE v1.9.0; composite reaches 0.90 (1st of 7).**
  v1.9.0 made the gRPC bridge production-grade with a real, pluggable model
  backend and server reflection:
  - `streaming` 0.97 → **0.98**: gRPC `StreamChat` is now backed by a real
    pluggable model (`OpenAiChatModel` streams any OpenAI-compatible endpoint),
    mapped *lazily* so cancelling the stream actually stops upstream
    generation — verified by test. A genuine streaming-completeness gain over
    the prior demo generator.
  - `interop` 0.60 → **0.67**: the spine-grpc bridge matured from demo to
    deployable — gRPC server reflection (grpcurl introspects with zero stubs), a
    runnable `serve` example, and a real model backend that can point at SPINE's
    own gateway (the bridges compose). Still adapter-into-ecosystem, not native
    adoption, so interop remains SPINE's lowest axis.
  Composite **0.88 → 0.90**. Evidence strings, README, and benchmark note
  updated; directional tests unchanged (all still hold).

## [0.13.0] - 2026-06-08

### Changed
- **SPINE `interop` 0.50 → 0.60** after SPINE v1.8.0 added a third ecosystem
  bridge: the `spine-grpc` crate, a tonic `AgentService` exposing the agentic
  surface (ListCapabilities / CallTool / streaming StreamChat) over gRPC,
  verified end-to-end over real HTTP/2. SPINE is now reachable from the three
  dominant agent ecosystems — MCP (runnable stdio server), OpenAI (gateway), and
  gRPC/protobuf — via deployable, standards-compliant servers a client calls
  with 100% standard stubs. SPINE composite **0.86 → 0.88** (1st of 7). Honest
  caveat unchanged: the bridges map the agentic surface (not SPINE's native
  binary latent frames) and SPINE's own protocol has ~zero native install base,
  so interop remains its lowest axis despite the breadth. Evidence string,
  README, and benchmark note updated.

## [0.12.1] - 2026-06-08

### Changed
- **SPINE `encoding-efficiency` evidence corrected (score unchanged at 0.95).**
  After SPINE v1.7.0 made `wire::encode` plain CBOR by default (a benchmark
  caught the prior zstd-per-frame design costing ~250 µs/frame; the fast path
  now encodes a 1 KiB embedding in ~590 ns, ~10× faster than JSON), the evidence
  string now reports the honest *default* numbers — a 1 KiB embedding frame
  3975→1263 B (68% smaller) on the fast path, 3975→446 B (89%) via the opt-in
  `wire::encode_compressed` — rather than presenting the compressed figure as
  the default. The 0.95 score (protobuf-class density) stands and is in fact
  better supported now: fast *and* small, not small-but-slow.

## [0.12.0] - 2026-06-08

### Changed
- **SPINE `interop` 0.45 → 0.50** after SPINE v1.6.0 made the MCP bridge a
  *runnable* server (`mcp::serve_stdio` + the `mcp_stdio_server` example speak
  the MCP stdio JSON-RPC transport). The bridge is no longer just mapping code:
  a Claude Desktop / Claude Code `mcpServers` config entry spawns it and drives
  a SPINE agent today with zero SPINE-specific code. SPINE composite **0.85 →
  0.86** (still 1st of 7). The honest caveat is unchanged — these are deployable
  *adapters* into MCP/OpenAI, not native ecosystem adoption, so interop remains
  SPINE's lowest axis. Evidence string updated.

## [0.11.0] - 2026-06-08

### Changed
- **SPINE re-scored across all five axes after SPINE v1.5.0**, which shipped
  real capabilities on every axis the web benchmark measures (not score
  tweaks). SPINE now **leads the composite at 0.85 (1st of 7), edging gRPC
  (0.83)** — earned by working, tested code:
  - `streaming` 0.95 → **0.97**: `Message::StreamCancel` (multiplex-aware
    per-stream cancel) + `StreamToken.usage` (mid-stream cumulative usage).
  - `tool-discoverability` 0.90 → **0.95**: the `spine_protocol::mcp` bridge
    re-exposes capabilities over MCP `tools/list` / `tools/call` — matching the
    introspection gold standard while keeping native semantic capability search.
  - `encoding-efficiency` 0.92 → **0.95** (parity with gRPC/protobuf):
    `serde_bytes` byte-string tensor payloads bring `EncodedFrame` to
    protobuf-class density; the 1 KiB embedding frame is now 446 B (89% smaller
    than JSON), and `EncodedFrame` still moves tensors zero-token.
  - `interop` 0.15 → **0.45**: the MCP bridge means any MCP host (Claude
    Desktop/Code, MCP-capable IDEs) drives a SPINE agent with no SPINE-specific
    code, alongside the OpenAI-compatible gateway. Scored honestly as *adapters
    into the dominant contracts*, not native adoption — it still trails gRPC's
    install base and OpenAI's universality, which is why it remains the lowest
    SPINE axis.
  - `security-primitives` 0.90 → **0.95**: `spine_agentic::signed_frame` adds
    per-message Ed25519 signatures (integrity + authenticity + non-repudiation,
    verified before decode) — message-level guarantees beyond channel mTLS.
  Evidence strings rewritten and `axis_judgments_hold_directionally` extended
  (SPINE now leads composite; still trails interop; leads security over gRPC).

## [0.10.0] - 2026-06-08

### Changed
- **SPINE re-scored on `encoding-efficiency`: 0.65 → 0.92**, from measured
  data after SPINE v1.4.0 shipped a binary wire format. The body is no longer
  serde_json behind a header — it is a self-describing CBOR codec (8-byte
  `SpineWireHeader` + CBOR/RFC 8949, with opportunistic zstd past 128 B).
  Measured against the old JSON body (`spine-protocol`
  `examples/wire_sizes.rs`, header included): a 1 KiB embedding frame
  3975→546 B (86% smaller), a 2-capability advertisement 806→322 B (60%), a
  tool call 323→255 B (21%); every frame beats JSON. SPINE now sits in the
  binary-efficient tier next to gRPC/protobuf (0.95) — gRPC stays marginally
  ahead because protobuf elides field names while CBOR keeps self-describing
  keys, but SPINE's `EncodedFrame` moves raw tensor bytes zero-token, which
  protobuf has no native equivalent for. Evidence string and the
  `axis_judgments_hold_directionally` directional test updated accordingly;
  the honest `interop` 0.15 is unchanged (interop is a later concern).

## [0.9.0] - 2026-06-04

### Added
- **`web`** — curated agentic profiles of 7 **web stacks / wire protocols**
  (SPINE, OpenAI API, Anthropic API, MCP, gRPC, HTTP+JSON, GraphQL) for the
  *agent-to-service* traffic an agent actually has to speak. Scored on five
  agent-native axes distinct from the VM axes (since a wire protocol is not
  a sandbox): **streaming** (LLM-shaped output as a first-class frame family
  vs. a bolt-on on top of a document protocol), **tool-discoverability**
  (introspect the surface from the protocol itself vs. read prose),
  **encoding-efficiency** (binary framing + content-typed payloads vs.
  JSON-over-HTTP/1.1 baseline), **interop** (does the agent ecosystem already
  speak it?), and **security-primitives** (auth, W3C distributed tracing,
  content integrity inline vs. someone-else's-problem). Each profile carries
  ≥3 evidence strings; `profile` / `profiles` / `rank_web_stacks` /
  `compare_web_stacks`, `WebStack::from_name` aliases (`openai`, `claude-api`,
  `mcp` / `model-context-protocol`, `rest`, `gql`, `nervosys-spine`). Wired
  into the self-describing ontology: `describe("web")`, `describe("spine")`,
  `describe("grpc")`, and the `manifest()` index now lists web stacks.
- `examples/web_benchmark.rs` — ranked table + SPINE-vs-OpenAI head-to-head +
  SPINE evidence dump + reading summary. Run with
  `cargo run -p agentic-eval --example web_benchmark`.

### Notes
- **SPINE is evaluated honestly**: it leads on the protocol-semantics axes it
  was designed for (streaming, tool-discoverability, security-primitives) and
  carries an explicit 0.15 on `interop` for being brand new — the gateway's
  OpenAI-compatible `/v1/chat/completions`, `/v1/embeddings`, and
  `/v1/agentic/{capabilities,codecs}` routes are documented as the migration
  bridge. gRPC tops the unweighted composite because its broad strengths
  (protobuf efficiency + mTLS + reflection + bidi streaming + huge install
  base) outweigh SPINE's narrower edge on LLM-native frames; that ranking is
  the point of the benchmark.

## [0.8.0] - 2026-06-03

### Added
- **`vms`** — curated agentic profiles of 7 VM/sandbox systems (AetherVM,
  Firecracker, Cloud Hypervisor, gVisor, Kata Containers, QEMU/KVM, Docker) for
  the *ephemeral agent-sandbox* workload an agent runtime drives. Scored on five
  **agent-native axes** (distinct from the program axes, since a VM isn't text):
  **start-latency** (cold-start per tool call), **density** (sandboxes per host),
  **isolation** (boundary strength for untrusted agent-generated code),
  **snapshotting** (CoW fork / warm-pool branching), and **agent-control**
  (tool/MCP-native control plane vs. bring-your-own glue). Each profile carries
  evidence strings; `profile`/`profiles`/`rank_vms`/`compare_vms`,
  `Vm::from_name` aliases (`fc`, `chv`, `runsc`, `kvm`, `runc`, `hypermachine`).
  Wired into the self-describing ontology: `describe("vms")`,
  `describe("firecracker")`, and the `manifest()` index now lists VM systems.

### Notes
- The VM axes are workload-specific by design: a strong long-lived datacenter VM
  can rank low for the spawn-and-tear-down sandbox loop, and a shared-kernel
  container ranks high on speed/density but low on isolation for untrusted code.
  Scores are honest curated judgments with rationale — including AetherVM's
  (strong on snapshotting/agent-control, with an explicit "younger, less
  battle-tested at scale" caveat on isolation).

## [0.7.0] - 2026-06-03

### Added
Two new evaluation **subjects** — beyond programs, the crate now profiles what
agents *build with*:
- **`languages`** — curated agentic profiles of 10 programming languages
  (Python, Rust, JS, TS, Go, Bash, C, C++, Java, MechGen) on the four axes:
  token efficiency, determinism, reliability (does the toolchain catch agent
  mistakes with actionable diagnostics?), and safety (default blast radius).
  Each profile carries evidence strings; `profile`/`profiles`/`rank_languages`/
  `compare_languages`, `Language::from_name` aliases (`js`, `c++`, `golang`, …).
- **`frameworks`** — curated agentic profiles of 9 AI frameworks (PyTorch,
  TensorFlow, JAX, HF Transformers, ONNX Runtime, scikit-learn, Candle, Burn,
  RecursiveMachineIntelligence (RMI)) on the four axes **plus discoverability** (can an agent learn
  the surface from the framework itself — schemas/ontology/introspection — vs
  scraping prose?). Notes artifact-safety facts (pickle ≈ arbitrary code,
  `trust_remote_code`, safetensors). `profile`/`rank_frameworks`/
  `compare_frameworks`, `Framework::from_name` aliases (`torch`, `tf`, `hf`, `rmi`).

Both are static curated judgments (deterministic, serializable, with rationale),
not measurements — use the program-level axes to measure your own code. Wired
into the self-describing ontology: `manifest()` lists both groups;
`describe("languages")`/`describe("rust")`/`describe("pytorch")` expand them
(ranked tables / full profiles + evidence). All types re-exported at the root.

## [0.6.0] - 2026-06-03

### Added
Five new metrics across the cost/reliability/safety axes (each: typed report,
`Display`, `serde`, ontology entry, tests):
- **Token cost — output scaling** (`assess_scaling`, `ScalingReport`): least-squares
  fit of output tokens vs result size → marginal `per_item` cost + `fixed_overhead`;
  flags O(1) output. The curve that matters at agent scale, not a single sample.
- **Token cost — prompt-cache efficiency** (`assess_cache`, `CacheReport`,
  `cacheable_prefix_tokens`): models API prompt-caching — a stable prefix paid once
  at write price (×1.25) then read price (×0.1) — reporting `cacheable_ratio` and the
  session savings ratio.
- **Reliability — graded error actionability** (`assess_error_quality`,
  `ErrorQuality`/`ErrorQualityReport`): refines the binary actionable flag into a
  0–1 score over code/message/location/fix.
- **Safety — reversibility** (`assess_reversibility`, `ReversibilityReport`): fraction
  of *dangerous* effects backed by undo/rollback — the recoverable-blast-radius
  complement to gating.
- **Safety — exfiltration risk** (`assess_exfiltration`, `ExfiltrationReport`):
  source∧sink exposure — reads local state *and* has a network/exec egress path.

All re-exported at the crate root and listed in the self-describing `ontology`.

## [0.5.0] - 2026-06-02

### Added
- **`ontology` module — a complete, self-describing ontology over the crate.**
  Agentic-first: a consumer discovers the whole surface from a compact, deterministic
  `manifest()` (axes, effect taxonomy, modes, models, command count — a few hundred
  tokens) and expands any entry with `describe("<axis|effect|model|section>")`, the
  same progressive-disclosure pattern the crate measures. `ontology()` returns the
  full structured catalog (`Ontology`, `serde`-serializable): the four axes with
  entry points/output types, every `Effect` with its per-`Mode` policy `Decision` and
  example commands, the tokenizer `Model`s with exactness, and the classifier size.
- **Taxonomy enumerators** for building ontologies over the types: `Effect::all`,
  `Effect::summary`, `Effect::decision`, `Mode::all`, `Mode::name`, `Decision::name`,
  and `commands::commands_for` / `commands::known_command_count`.

### Changed
- Crate-level docs now state the agentic-first design contract (deterministic,
  execution-agnostic, structured, self-describing).

## [0.4.0] - 2026-06-02

### Added
- **`commands` module — heuristic CLI effect classification.** A curated table of
  ~200 common POSIX/Unix/dev tools mapped to their [`safety::Effect`] class, so the
  safety axis works on a **wide variety of real CLI programs** without a hand-written
  classifier: `classify_command` (name → effect), `classify_invocation` (one command
  line — strips `VAR=val` and path prefixes; unknown program → `Exec`; `sudo`/`doas`
  → `Privileged`), `classify_script` (split a script on `\n ; | & && ||` and classify
  each), and `assess_safety_script` (one call from a script to a `SafetyReport`).
  Fail-safe by design: an unrecognized program is treated as arbitrary execution, and
  multi-mode tools map to their most security-salient common effect (`git` → network,
  `docker`/`npm`/`make` → exec, `apt`/`mount` → privileged). All re-exported at the
  crate root.

## [0.3.0] - 2026-06-02

### Added
- **Pluggable tokenizer**: `tokens::evaluate_with` and `rank_with` accept any
  `Fn(&str) -> usize`, so a host can flow its own exact tokenizer through the cost
  model instead of the built-in heuristic/BPEs.
- `AgentCost::total_standing_per_turn` — the no-prompt-caching cost model (standing
  context re-sent every turn), complementing the caching-amortized `total_over`.
- `safety::assess_safety_named` — score safety from operation names plus a classifier
  closure (unknowns skipped).
- Release hardening: `#![forbid(unsafe_code)]`, `#![deny(missing_docs)]` (every public
  item is documented), full `Cargo.toml` metadata (`readme`, `documentation`,
  `rust-version`, docs.rs all-features), and a crate CHANGELOG.

## [0.2.0] - 2026-06-02

### Added
- Crate-root re-exports of the most-used types (`Model`, `Program`, `AgentCost`,
  `Effect`, `Mode`, `assess_*`, …).
- `Display` for every report type (`AgentCost`, `Comparison`, `DeterminismReport`,
  `ReliabilityReport`, `SafetyReport`, `Evaluation`).
- Optional `serde` feature deriving `Serialize` on all report/config types.
- `Model::from_name` (CLI/config parsing), `tokens::rank` (N-way comparison),
  `Evaluation` `with_*` builders, `safety::Effect::from_name`.

### Changed
- The heuristic tokenizer splits `snake_case` subwords (`file_read` ≈ 2 tokens).
- Corrected the `AnthropicClaude` model docs to state it is a heuristic approximation
  (no public offline Claude tokenizer), not a "calibrated" estimate.

## [0.1.0] - 2026-06-01

### Added
- Initial release. Four-axis evaluation of programs for agentic AI use:
  **token efficiency** (`tokens`), **determinism** (`determinism`), **reliability**
  (`reliability`), and **safety** (`safety`), plus a combined `Evaluation`.
- Token counting under OpenAI GPT-4 (`cl100k`) and GPT-4o (`o200k`) — exact with
  `--features real-tokens`, a documented Anthropic-Claude approximation, and a
  labeled heuristic otherwise. Execution-agnostic; zero heavy deps by default.