# llmp
[](https://crates.io/crates/llmp)
[](https://docs.rs/llmp)
[](https://github.com/rssndr/llmp/actions)
Measures **how well a local LLM setup uses its model.** llmp sits between
any client (a coding agent, a chat UI, a script) and any local model
server, records the traffic, and reports how much time goes to re-processing
prompts the server had already seen, why that happened, and what each request
is made of.
```
$ llmp analyze session.jsonl
llmp: 9 requests over 18.3 s, 8 analysed, 7 conversation turns
time lost re-processing prompts: 411 ms of 1.5 s prefill (27%), 594 tokens
by cause: system prompt changes 411 ms (594 tokens)
coverage: 8 of 9 records analysed (1 not chat); reuse reported on 7 of 7 turns, timing on 7
findings:
[medium] 1 side-call on the conversation model gemma: 0 tokens lost
[low] tool write_file offered on 7 requests, never called (60 tokens each)
[low] tool list_dir offered on 7 requests, never called (51 tokens each)
```
That is a real recording (`tests/fixtures/llama.cpp`). The client put
the date at the end of its system prompt. When the date changed,
llama.cpp re-evaluated the whole prompt, and that one change cost
more than a quarter of the session's prefill time.
## Install
```
cargo install llmp
```
As a library:
```
cargo add llmp # proxy + analysis
cargo add llmp --no-default-features # analysis only: pulls in just serde
```
## Usage
**1. Record.** Put the proxy in front of your server and point the
client at the proxy instead:
```
$ llmp proxy --upstream 127.0.0.1:8081
llmp: 127.0.0.1:8080 -> 127.0.0.1:8081, recording to llmp-1789988955.jsonl
```
The proxy forwards everything unchanged and streams responses as they
arrive. It never modifies a request or a response. Stop it with Ctrl+C;
it finishes the exchanges under way and prints how many records it
wrote.
| `--upstream HOST:PORT` | required | the model server |
| `--listen HOST:PORT` | `127.0.0.1:8080` | where clients connect |
| `--out FILE` | `./llmp-<unix seconds>.jsonl` | the recording; appended to |
| `--digest` | off | keep hashes and sizes instead of message text |
| `--max-body BYTES` | 64 MiB | bodies above this are forwarded whole but not stored |
A full recording holds every prompt and reply. Use `--digest` for long
captures, or when the text must not be kept: the analysis still works,
and only estimates of edited messages get rougher.
**2. Analyse.**
```
llmp analyze FILE... [--json] [--tokenize URL] [--gap-seconds N]
```
- Several files are merged in time order.
- `--json` prints the full report instead of the summary.
- `--tokenize URL` asks the server's tokenize endpoint for exact token
counts (llama.cpp), instead of estimating from characters.
- `--gap-seconds N` sets how long a pause splits two sessions
(default 60).
Exit codes: `0` success, `1` runtime error, `2` usage error.
## As a library
Analyse a recording:
```rust
use std::fs::File;
use std::io::BufReader;
let read = llmp::read_jsonl(BufReader::new(File::open("session.jsonl")?));
let mut report = llmp::analyze(&read.records, &llmp::Config::default());
report.coverage.bad_lines = read.bad_lines.len() as u64;
print!("{}", report.summary());
for finding in &report.findings {
println!("{:?} on {:?}: {:?} ms lost", finding.kind, finding.model, finding.lost_ms.map(|m| m.value));
}
```
`analyze` is pure: no I/O, and the same records always give the same
report. Every type is serde-serialisable.
Run the proxy inside your own program (on a tokio runtime), with the
records going wherever you want. A sink can be a closure, a
`std::sync::mpsc::Sender<Record>`, or `JsonlSink`:
```rust
use llmp::proxy::{BodyMode, ProxyConfig};
let config = ProxyConfig {
listen: "127.0.0.1:8080".parse()?,
upstream: "127.0.0.1:8081".parse()?,
body_mode: BodyMode::Full,
max_body_bytes: 64 << 20,
};
let (stop, stopped) = tokio::sync::oneshot::channel::<()>();
```
`serve` returns once `shutdown` resolves and the exchanges under way have
finished; `serve_blocking` does the same on its own runtime until Ctrl+C.
## What it measures
Everything comes from one table with one row per request, which is part
of the report. Every other number is computed from it, so two outputs
can never disagree about the same request.
For each conversation turn, llmp works out how much of the prompt the
server *could* have reused from the previous turn of the same
conversation, and compares it
with what the server *says* it reused. When less was reused than
could have been, the difference is **time lost**, and it is charged to
a cause:
| `system_changed`, `tools_changed`, `history_rewritten` | the client changed something it had sent before (a date in the system prompt, a reordered tool list, an edited message) |
| `side_call` | another request on the same model ran in between (a title, a summary, a classifier) and evicted the conversation |
| `other_conversation` | another conversation on the same model ran in between (a subagent, a parallel task, another agent) and evicted it |
| `other_model` | a request to another model ran in between |
| `server_empty`, `server_head_only`, `server_rollback` | the server dropped its cache: all of it, all but the system prompt, or part of the history |
A loss is never larger than what the server actually re-evaluated.
Context compactions are recognised and not blamed on anyone.
Several agents can share one proxy. Each turn is compared with the
conversation it belongs to, not with whatever request came before it,
and requests are grouped into **clients** by user-agent, and by tool set
when harnesses share a user-agent. Tools and the findings a client
controls (its prompt, its tools) are reported per client.
Each request is also split into system prompt, tool schemas, history
and new content. The findings are typed data (kind, evidence, affected
requests, size, confidence) for a host to rank, word and act on:
side-calls on the conversation model, frequent prefix breaks, repeated
cache losses, eviction by other conversations or models, tools never
called, oversized
tool schemas and system prompts.
**"Not measured" is not zero.** A number the server did not report is
`null`, and every number says where it came from: `server`,
`tokenizer`, `proxy` (the proxy's own clock) or `estimate`.
## Servers
| llama.cpp | `/v1/chat/completions` | verified (b10015) | everything, including reuse; the only one where time lost can be measured |
| Ollama | `/api/chat` | verified (0.32.14) | prompt size and prefill time, but not reuse |
| Ollama | `/v1/chat/completions` | verified (0.32.14) | prompt size only in non-streamed replies; prefill time from the proxy |
| vLLM | `/v1/chat/completions` | **unverified** | expected: prompt size and reuse (`cached_tokens`); prefill time from the proxy |
| LM Studio | `/v1/chat/completions` | **unverified** | expected: as vLLM |
"Verified" means a real recording is checked in under `tests/fixtures`
and tested. On a server that does not report reuse, llmp still records
everything and analyses the composition and the findings that don't need
counters, and it says that time lost was not measured.
## Formats
The recording (JSON Lines, one exchange per line) and the report (JSON)
are versioned and documented in [docs/format.md](docs/format.md). They
are the contract: a tool in any language can produce or consume them
without linking the crate.
## Out of scope
- Launching agents, or setting environment variables for them
- Advice about particular agents, servers or config keys (findings are
data; the host words them)
- Hosted APIs over HTTPS, and Anthropic-format `/v1/messages` clients
- Several models loaded on one server (residency, eviction between
models)
- Before/after comparison of two recordings
- Editing anyone's config files
- Naming the product behind a client: a client is labelled by its
user-agent, and the host maps that to a product
## License
MIT