llmp

Measures how well a local LLM setup uses its model. llmp sits between any client (a coding agent, a chat UI, a script) and any local model server, records the traffic, and reports how much time goes to re-processing prompts the server had already seen, why that happened, and what each request is made of.

$ llmp analyze session.jsonl
llmp: 9 requests over 18.3 s, 8 analysed, 7 conversation turns
time lost re-processing prompts: 411 ms of 1.5 s prefill (27%), 594 tokens
  by cause: system prompt changes 411 ms (594 tokens)
coverage: 8 of 9 records analysed (1 not chat); reuse reported on 7 of 7 turns, timing on 7
findings:
  [medium] 1 side-call on the conversation model gemma: 0 tokens lost
  [low] tool write_file offered on 7 requests, never called (60 tokens each)
  [low] tool list_dir offered on 7 requests, never called (51 tokens each)

That is a real recording (tests/fixtures/llama.cpp). The client put the date at the end of its system prompt. When the date changed, llama.cpp re-evaluated the whole prompt, and that one change cost more than a quarter of the session's prefill time.

Install

cargo install llmp

As a library:

cargo add llmp                                      # proxy + analysis
cargo add llmp --no-default-features                # analysis only: pulls in just serde

Usage

1. Record. Put the proxy in front of your server and point the client at the proxy instead:

$ llmp proxy --upstream 127.0.0.1:8081
llmp: 127.0.0.1:8080 -> 127.0.0.1:8081, recording to llmp-1789988955.jsonl

The proxy forwards everything unchanged and streams responses as they arrive. It never modifies a request or a response. Stop it with Ctrl+C; it finishes the exchanges under way and prints how many records it wrote.

A full recording holds every prompt and reply. Use --digest for long captures, or when the text must not be kept: the analysis still works, and only estimates of edited messages get rougher.

2. Analyse.

llmp analyze FILE... [--json] [--tokenize URL] [--gap-seconds N]

• Several files are merged in time order.
• --json prints the full report instead of the summary.
• --tokenize URL asks the server's tokenize endpoint for exact token counts (llama.cpp), instead of estimating from characters.
• --gap-seconds N sets how long a pause splits two sessions (default 60).

Exit codes: 0 success, 1 runtime error, 2 usage error.

As a library

Analyse a recording:

use std::fs::File;
use std::io::BufReader;

let read = llmp::read_jsonl(BufReader::new(File::open("session.jsonl")?));
let mut report = llmp::analyze(&read.records, &llmp::Config::default());
report.coverage.bad_lines = read.bad_lines.len() as u64;

print!("{}", report.summary());
for finding in &report.findings {
    println!("{:?} on {:?}: {:?} ms lost", finding.kind, finding.model, finding.lost_ms.map(|m| m.value));
}

analyze is pure: no I/O, and the same records always give the same report. Every type is serde-serialisable.

Run the proxy inside your own program (on a tokio runtime), with the records going wherever you want. A sink can be a closure, a std::sync::mpsc::Sender<Record>, or JsonlSink:

use llmp::proxy::{BodyMode, ProxyConfig};

let config = ProxyConfig {
    listen: "127.0.0.1:8080".parse()?,
    upstream: "127.0.0.1:8081".parse()?,
    body_mode: BodyMode::Full,
    max_body_bytes: 64 << 20,
};
let (stop, stopped) = tokio::sync::oneshot::channel::<()>();
let sink = |record: llmp::Record| println!("{} {} took {} ms", record.id, record.request.path, record.t_end_ms - record.t_start_ms);
let proxy = tokio::spawn(llmp::proxy::serve(config, sink, async { let _ = stopped.await; }));

serve returns once shutdown resolves and the exchanges under way have finished; serve_blocking does the same on its own runtime until Ctrl+C.

What it measures

Everything comes from one table with one row per request, which is part of the report. Every other number is computed from it, so two outputs can never disagree about the same request.

For each conversation turn, llmp works out how much of the prompt the server could have reused from the previous turn of the same conversation, and compares it with what the server says it reused. When less was reused than could have been, the difference is time lost, and it is charged to a cause:

A loss is never larger than what the server actually re-evaluated. Context compactions are recognised and not blamed on anyone.

Several agents can share one proxy. Each turn is compared with the conversation it belongs to, not with whatever request came before it, and requests are grouped into clients by user-agent, and by tool set when harnesses share a user-agent. Tools and the findings a client controls (its prompt, its tools) are reported per client.

Each request is also split into system prompt, tool schemas, history and new content. The findings are typed data (kind, evidence, affected requests, size, confidence) for a host to rank, word and act on: side-calls on the conversation model, frequent prefix breaks, repeated cache losses, eviction by other conversations or models, tools never called, oversized tool schemas and system prompts.

"Not measured" is not zero. A number the server did not report is null, and every number says where it came from: server, tokenizer, proxy (the proxy's own clock) or estimate.

Servers

"Verified" means a real recording is checked in under tests/fixtures and tested. On a server that does not report reuse, llmp still records everything and analyses the composition and the findings that don't need counters, and it says that time lost was not measured.

Formats

The recording (JSON Lines, one exchange per line) and the report (JSON) are versioned and documented in docs/format.md. They are the contract: a tool in any language can produce or consume them without linking the crate.

Out of scope

• Launching agents, or setting environment variables for them
• Advice about particular agents, servers or config keys (findings are data; the host words them)
• Hosted APIs over HTTPS, and Anthropic-format /v1/messages clients
• Several models loaded on one server (residency, eviction between models)
• Before/after comparison of two recordings
• Editing anyone's config files
• Naming the product behind a client: a client is labelled by its user-agent, and the host maps that to a product

License

MIT