# llmp — design
How llmp works, and why.
## 1. Overview
```
client ──HTTP──► llmp proxy ──HTTP──► model server
│
▼ Record (one per exchange)
Sink ──► JSONL file / channel / callback
Records ──► normalize ──► Exchange ──► table ──► Row[] ──► findings ──► Report
(pure, no I/O)
```
- **Capture** (`proxy.rs`) forwards traffic unchanged and emits one
`Record` per request/response exchange.
- **Analysis** (`analyze`) is a pure function from records to a
`Report`: no I/O, and the same input always gives the same output.
The only link between the two halves is the recording format, so a host
can use either one alone.
| `lib.rs` | public re-exports, `analyze()`, `Config`, format versions |
| `record.rs` | the recording types, JSONL read/write, digest mode |
| `measured.rs`, `hash.rs` | `Measured<T>` and `Source`; FNV-1a 64 and canonical JSON |
| `normalize/` | `Record` → `Exchange`, server fields → `Counts`, stream merging |
| `table.rs` | `Exchange[]` → rows: sessions, side-calls, prefix, reuse, loss |
| `findings.rs`, `report.rs` | findings; report assembly and the text summary |
| `proxy.rs`, `tokenize.rs` | feature `net`: the proxy and sinks; token enrichment |
| `main.rs` | feature `cli`: the binary |
Features: none gives analysis and record I/O (only `serde`,
`serde_json`); `net` adds the proxy and tokenize (tokio, hyper 1.x);
`cli`, the default, adds the binary. `serde_json` has `float_roundtrip`
on, so a recording or report read back gives exactly the numbers
written. Edition 2024, no CLI framework, as in hwprobe and modelfit.
## 2. Principles
- **"Not measured" is not zero.** An unknown value is `None` (`null`).
Every number that could come from more than one place is a
`Measured { value, source }`, with source `server`, `tokenizer`,
`proxy` or `estimate`, strongest first. A total takes the weakest
source of its parts.
- **One table.** Every number and finding reads from the rows, so two
outputs can never disagree about one request.
- **A loss is never larger than what the server re-evaluated.**
- **Nothing is dropped silently:** skipped records, bad lines, bad
stream chunks, untimed losses and tokenize failures are all counted.
- **The proxy only observes.** It never changes a request or a
response.
- **Tested against real captures,** not only synthetic ones: each server
mapping is marked verified only once a real recording of it is checked
in (see 10).
## 3. Message identity
A message's hash is FNV-1a 64 over the canonical JSON of `[role,
content, tool_calls, name, tool_call_id]`. An empty `tool_calls` array
counts as missing, and Ollama's `tool_name` as `name`. A message's
**text** is its content (text parts joined) followed by the canonical
JSON of its tool calls; `chars`, `head` and `tail` come from it.
The digest functions in `record.rs` (`request_digest`,
`response_digest`, `message_text`) are also what normalization uses in
full mode, so a full and a digest recording of the same traffic give the
same hashes. `head` and `tail` hash the first and last 256 characters:
they tell that two versions of a message share their start or end
without keeping the text (see 6.5). Tool calls are kept as `{id, name}`
so calls can be deduplicated by id.
In digest mode only chat requests, and 2xx responses to them, are
digested; every other body is dropped, with `body_bytes` still set. A
response digest's `usage` keeps every top-level field except `choices`
and `message`, so the server's counters survive whatever their names.
## 4. Normalization
**API detection** uses the path only: `/v1/chat/completions` is OpenAI
chat, `/api/chat` is Ollama chat, anything else is counted in coverage
and not analysed.
**Skip reasons,** first match wins: `not_chat` (path), `error` (an
`error` or no response), `non_2xx`, `unparsed` (no digest and no JSON
body with `messages`). A parsed request whose response didn't parse
(e.g. truncated) is still analysed, with its counts `null`.
**Stream merging** (done by the proxy, stored merged): SSE concatenates
`delta.content` and `delta.reasoning_content` (or `delta.reasoning`),
merges `delta.tool_calls` by index, keeps the last non-null value of
every other top-level field (`usage`, `timings`), and gives the
non-streamed shape. NDJSON concatenates `message.content` and
`message.thinking`, collects tool calls, and keeps the final object's
counters. Unparseable events count as `bad_chunks`; a stream with none
readable is kept as `body_text`.
### 4.1 Server fields → `Counts`
The mapping is chosen by API, not by server: each field is taken when
present.
| `prompt_tokens` | `usage.prompt_tokens`, else rule 0 | `usage.prompt_tokens` | `prompt_eval_count` | `usage.prompt_tokens` (absent in streams) |
| `completion_tokens` | `usage.completion_tokens`, else `timings.predicted_n` | `usage.completion_tokens` | `eval_count` | `usage.completion_tokens` (absent in streams) |
| `reused_tokens` | `timings.cache_n`, else `usage.prompt_tokens_details.cached_tokens` | `usage.prompt_tokens_details.cached_tokens` | — | — |
| `evaluated_tokens` | `timings.prompt_n` | rule 1 | — | — |
| `prefill_ms` | `timings.prompt_ms` | proxy TTFT (rule 2) | `prompt_eval_duration` / 1e6 | proxy TTFT (rule 2) |
| `decode_ms` | `timings.predicted_ms` | — | `eval_duration` / 1e6 | — |
| `load_ms` | — | — | `load_duration` / 1e6 | — |
The llama.cpp (b10015) and Ollama (0.32.14) columns are verified by the
recordings in `tests/fixtures`. ⚠ vLLM and LM Studio are unverified.
What the real servers showed:
- **llama.cpp streams carry no `usage`** unless the client asks for
`stream_options.include_usage`, which most agents don't. `timings` is
still there, hence rule 0 and the `predicted_n` fallback. It reports
both `cache_n` and `cached_tokens`; they agree, and `timings` wins
because it comes with `prompt_n` and `prompt_ms`.
- **Ollama's `prompt_eval_count` is the whole prompt, cached or not:**
two identical requests both reported 459 while the prefill time fell
from 125 ms to 22 ms. So it is `prompt_tokens`, and Ollama reports
neither reused nor evaluated tokens: no loss can be measured on
Ollama, only prefill time. Older versions may count differently.
- **Ollama's OpenAI-compatible API** has no `prompt_tokens_details`,
and no `usage` in streams.
Derivation rules, in this order:
0. Prompt unknown, reused and evaluated known: `prompt = reused +
evaluated`, with the weaker source.
1. Prompt known and exactly one of reused / evaluated known: the other
is `prompt − known` (never below 0), with the known one's source.
2. **Proxy TTFT** (`t_first_byte_ms − t_start_ms`) is the prefill time
only when the response was streamed and the server gave none. Source
`proxy`; it includes queueing, so it is an upper bound.
3. A non-streamed response's time to first byte covers the whole
generation and is never used as prefill time.
## 5. Sessions, conversations, clients, side-calls
Rows are in `t_start_ms` order, ties by `id`.
**Weights.** A message weighs its tokenizer count, else `ceil(chars /
4)`; the tool block weighs the sum of its tools. A request's weights are
scaled to its `prompt_tokens` when known, so every token figure in a row
is in server units. The composition's source is `tokenizer` when every
weight was tokenized, else `estimate`, never `server`: scaling to the
server's total does not make the split a server number.
**Sessions.** A domain is the request's `model`. A session is the
consecutive requests of one domain, split where the gap exceeds
`session_gap_ms` (60 s). The gap runs from the latest *end* of the
session's requests so far to the next start, so a long prefill never
splits a conversation.
**Continuation.** B continues A when A's messages are a prefix of B's.
System messages are compared by role and position only (a date in the
system prompt must not break the chain). A's last message may differ
only when it is an assistant message, because clients append to it.
That limit matters: without it any `[system, user]` request would
continue any other request starting with a system and a user message,
and unrelated one-shot questions would look like a conversation. Every
pair in a domain is checked: O(n²), about 0.3 s for 5 000 requests of
100+ messages.
**Side-calls** run beside the conversation. First matching shape:
| `summarise` | no tools, ≤ 3 messages, user text > 90% of all text and > 20 000 chars |
| `structured` | no tools, ≤ 3 messages, and `grammar`, `response_format` or `0 < max_tokens ≤ 32` |
| `oneshot` | no tools; continues nothing and is continued by nothing in its domain; its first system message (or "none") differs from those of the domain's continuing requests; the domain has at least one continuing pair |
A **stateless domain** has ≥ 3 non-side-call requests, none continuing
the one before, with message counts that never grow. Its rows are
`stateless` and get no prefix analysis: every request is new by design.
A single request with its own system prompt that nothing continues (a
subagent that ran one turn, say) is indistinguishable from a `oneshot`
side-call, and is treated as one.
**Conversations.** Agents run subagents and parallel tasks, and several
harnesses can share one server, so the turns of different conversations
interleave. A turn is therefore not compared with the turn before it,
but with the latest turn (the *tip*) of the conversation it belongs to.
Candidates are the tips on the same model, with the same user-agent, in
the same session. First match:
1. a tip the turn continues (5, continuation): the longest such tip,
then the most recent. This is the same conversation, possibly with a
changed system prompt or tool list.
2. a tip whose messages match exactly beyond the leading system
messages: the longest match, then the most recent. The same
conversation with its history rewritten or compacted.
3. a tip that matches only in the system messages: a **new
conversation** from the same client. It is linked to that tip for its
prefix (the server could reuse the system prompt and tools), but it
has no break and nothing is charged to the client.
4. none: a new conversation with no link.
The price of rule 3: a client that rewrites its *first* user message, or
compacts down to the system prompt alone, looks like a new conversation.
Nothing is charged to the client either way, and it keeps interleaved
conversations with the same system prompt apart, which matters more.
Two user-agents are never linked. A pause longer than the session gap
still ends every conversation.
Before this rule, turns were linked to the previous turn of the session.
With two agents on one model, that compared agent A's turn with agent
B's, and produced false `system_changed` and `history_rewritten` breaks
on nearly every turn.
**Clients.** A client is labelled by its user-agent, which the proxy
records. Many harnesses send a generic SDK user-agent (`OpenAI/Python`,
`node`), so the tool set is the second signal: a harness sends the same
tool list on every request. Within one user-agent, the conversations and
the requests outside conversations are grouped by their tool names; two
groups join when one's tools are a subset of the other's, which keeps a
subagent (usually offered a subset of its parent's tools) with its
parent. When a user-agent ends up with one group, the client is the
plain user-agent; with several, `<user-agent> #1`, `#2`…, numbered by
first request. Requests without tools under a user-agent with several
groups keep the plain user-agent: nothing says which harness sent them.
llmp never names the product behind a client; the host maps
user-agents to products.
## 6. Prefix, reuse and loss
### 6.1 Links
A link joins a turn to the tip it was matched with (5, conversations);
`turn` numbers the turns of a conversation from 1. What ran on the
server in between is recorded by kind: same-model side-calls in
`side_calls_between`, same-model turns of other conversations in
`other_conversations_between`, other models' requests in
`other_domains_between`.
### 6.2 Prefix
- A different `tools_hash`: prefix 0, `tools_changed`.
- Otherwise walk the messages by index while hashes match; the prefix is
their weights (cur's), plus the tool block **only if at least the
first message matched.** Chat templates render the tools inside or
after the leading system message, so when that changed the tools are
not reusable either. Without this, a changed system prompt charged
the tool block to the server as `server_empty`.
- The break: stopped inside the overlap → `system_changed` if that
message is a system message, else `history_rewritten`; cur shorter
and matched to its end → `history_rewritten`.
- **Compaction:** `history_rewritten` where cur weighs < 70% of prev.
Normal agent behaviour, so nothing is charged to the client.
### 6.3 Already sent
For a break that is not a compaction: the tool block if unchanged, plus
every message equal to prev's at the same index, plus, for a message
with the same role at the same index that was *edited*, its weight × its
unchanged share. A message counts as edited when neither its hash nor
its counterpart's appears elsewhere in the other request; otherwise it
moved, and a moved message is not counted.
The unchanged share is (common prefix chars + common suffix chars,
non-overlapping) / cur's chars, from the text in full mode. In digest
mode it is 256 for a matching `head` plus 256 for a matching `tail`,
capped at the message's length: rougher, which is why full and digest
recordings can differ in the size of client losses, and only there.
### 6.4 Cause
Only when the server reported reuse. `predicted = prefix / prompt`,
`observed = reused / prompt`, `gap = predicted − observed` in points,
`cold = observed ≤ 5%`. First match:
1. `gap < −20` → none (the server reused more than predicted)
2. `|gap| ≤ 20` → `structural` if there is a break, else none
3. not cold → `structural` if there is a break, else `undetermined`
4. same-model requests in between → the kind of the **latest** one:
`side_call` or `other_conversation`
5. other domains in between → `other_model`
6. otherwise → `server_reset`
Rule 4 blames the last same-model request before the turn. On a
one-slot server that is the request that overwrote the cache; with
several slots it is still the best evidence. A fixed order (side-calls
first) was tried and was wrong: in the `parallel-nocache` fixture one
title side-call was blamed for evictions that four parallel
conversations caused, because most gaps contained both.
### 6.5 Loss
Only for a linked turn with prompt, reused, evaluated and prefill time
all known; those rows have `loss_measured: true`. `false` means "not
measured", not "no loss" — without the flag the two look the same.
```
reusable = prefix no break, or compaction
= max(prefix, already_sent) otherwise
lost = clamp(reusable − reused, 0, evaluated)
client = min(lost, max(0, reusable − max(prefix, reused))) break and not compaction, else 0
external = lost − client cause ∈ {side_call, other_model, server_reset, undetermined}, else 0
```
`client` is charged to the break, `external` to the cause. For
`server_reset` and `undetermined` it is refined by where reuse stopped
(`head` = tool block + leading system messages): `server_empty` when
`reused < 128`, `server_head_only` when `reused ≤ head × 1.15`,
`server_rollback` otherwise. A loss is only set when it rounds to ≥ 1
token.
**Time:** `lost_ms = lost × min(own rate, bulk rate)`, where the own rate
is this request's `prefill_ms / evaluated` and the bulk rate the domain's
over requests with ≥ 50 evaluated tokens and a server-sourced prefill.
The minimum keeps a small re-evaluation from being charged the server's
fixed overhead. `lost_ms` takes the prefill time's source.
Float sums start from `+0.0`: Rust sums an empty `f64` iterator to
`-0.0`, which leaked into the JSON.
### 6.6 Composition and tools
`system` counts every system message, `tools` the tool block, `history`
the other messages inside the prefix, `new_content` the rest. Rows
without a link have no prefix, so everything non-system is new.
Tool usage is per client and tool name, so two harnesses with a
`read_file` each are counted apart: `offered` counts requests,
`schema_tokens` is the largest weight seen, `called` counts distinct
calls from responses and re-sent history, deduplicated by id. A call
without an id (Ollama) is counted from the response only, since its
copy in history can't be matched to it. Calls to tools the client did
not offer are ignored.
## 7. Findings and report
Findings are typed data for the host to rank, word and act on; llmp
gives no advice.
| `side_call_on_conversation_model` | a domain has turns and side-calls | loss of turns with cause `side_call` |
| `prefix_break` (per client and break) | ≥ 25% of the client's links in sessions of ≥ 3 rows have that break (compactions and new conversations excluded) | client loss |
| `server_cache_loss` (per kind) | ≥ 2 rows of that kind | external loss |
| `other_conversation_eviction` | ≥ 2 rows with cause `other_conversation` | external loss |
| `other_model_eviction` | ≥ 2 rows with cause `other_model` | external loss |
| `unused_tool` (per client) | offered, never called | schema tokens per request |
| `large_tool_schema` (per client) | a tool > 800 tokens | tokens over |
| `large_system_prompt` (per client) | the client's largest system prompt on a model > 800 tokens | tokens over |
- What a client controls (its prompt, its tools) is reported per
client, with `client` set; what the server did (cache losses,
evictions, side-calls on the model) is per model, with `client`
unset. The tool findings have no model and no rows, since rows don't
record which tools they offered; `unused_tool`'s days and sessions are
the client's own.
- The side-call finding is raised even at size 0: that is a real result
(the server kept the conversation, e.g. with several slots or
llama.cpp's host-memory prompt cache).
- **Confidence** of a loss finding counts its rows with a measured
loss: ≥ 5 `high`, 1–4 `medium`, none `low` (certain, size unknown).
`unused_tool` rests on absence: `high` at ≥ 14 days and ≥ 5 sessions,
`medium` at ≥ 7 days. The size findings are `high` when sized by the
server's total or the tokenizer, `medium` from characters.
- Sorted by confidence, then `lost_ms`, then `context_tokens`, then
kind, model, client and evidence, so the order is deterministic.
The report's `time.lost_ms` is `null` unless some row has
`loss_measured`; `servers[].fields` lists only fields with source
`server`, so the host knows which numbers exist. `analyze` sees records,
not lines, so the caller that read the file sets `coverage.bad_lines`.
The **summary** is at most about 15 lines: turns and conversations, the
clients when there is more than one (and each client-scoped finding then
names its client), time lost and its share of prefill, the top three
causes, coverage, the top three findings. Its
labels are defined once, in `report.rs`.
**CLI.** `analyze` sorts records by `(t_start_ms, file, id)`. Ids restart
at 1 in every recording, so with several files the records are
renumbered 1… in merged order; with one file they are kept.
## 8. Proxy
- tokio + hyper 1.x, HTTP/1.1, one task per client connection, keep-alive.
Each client connection gets its own upstream connection, opened on
its first request. A reused upstream connection found closed before
the request was written is reopened and the request sent once more;
a request that may have reached the server is never sent twice.
- The request body is read fully, then forwarded; the response is
streamed to the client chunk by chunk, with a copy kept up to
`max_body_bytes`. Hop-by-hop headers are dropped per RFC 9110;
everything else passes unchanged. Only four headers are recorded,
never auth.
- Failures still produce a record:
| `error` | When | Client gets |
|---|---|---|
| `upstream_unreachable` | connecting failed | `502` |
| `timeout` | connecting took > 10 s | `502` |
| `upstream_reset` | upstream failed before or during the response | `502`, or a cut-off body |
| `client_disconnected` | the client left before the whole response | — |
| `body_too_large` | request body over the limit: forwarded, not stored | the response |
There is no timeout on a response: long prefills are normal, and the
client decides how long to wait.
- The record is built after the exchange, off the connection task (parse,
merge, digest), and handed to the sink on tokio's blocking pool.
`JsonlSink` writes from its own thread through a bounded channel
(1024) and flushes each line. When the channel is full the send
waits rather than drop: the next request on that connection waits
for the previous record, a response under way never does.
- Shutdown stops accepting, closes idle connections, and returns only
once the exchanges under way and their records are done. Ids are
taken on arrival.
## 9. Tokenize
`enrich` counts every distinct message and tool without a count, using
the same texts as the analysis (so counts match by hash): llama.cpp
`POST /tokenize {"content", "add_special": false}`, then Ollama `POST
/api/tokenize {"model", "text"}` if the first is 404/405. The Ollama
shape is a guess: current releases have no such endpoint. Counts are
raw, without template tokens; scaling to `prompt_tokens` absorbs that.
Empty texts count 0 without a request; counted texts are not sent
again. A bad URL, unreachable server or missing endpoint is a warning
and the estimates stay; a lost connection keeps what was counted.
Digest records are skipped with a warning. `enrich` blocks, running on
its own thread and runtime so an async host can call it. Only `http://`;
a URL path is kept as a prefix.
## 10. Testing and code rules
- Unit tests per module cover the rules above with their edge cases.
- `tests/fixtures/` holds real recordings, made with `drive.py` and
redacted with `redact.py` (see its README). `tests/fixtures.rs`
compares each report, as text, with a golden `*.report.json`
(`LLMP_UPDATE_GOLDEN=1` rewrites them; explain the change in the
commit) and checks invariants on every fixture: `lost ≤ evaluated`,
finite numbers and no `-0.0`, measured losses have all their counters,
a stream merges to the same totals as its non-streamed twin, and the
full and digest recordings of the same traffic agree.
- `parallel.jsonl` and `parallel-nocache.jsonl` record four agents at
once through one proxy; a test checks that every agent is exactly one
conversation, that interleaving produces no false break, and the
clients.
- `tests/proxy.rs` runs the proxy against a scripted fake upstream;
`tests/analyze.rs` and `tests/tokenize.rs` cover the CLI end to end.
- No comments or docstrings: names carry the meaning, and the reasons
live in this file. No `unwrap()` outside tests; `cargo fmt`
and `cargo clippy --all-targets -- -D warnings` pass on all three
feature sets. The public API is what `lib.rs` re-exports; everything
else is `pub(crate)`.