llmp 0.1.0

Measures how well a local LLM setup uses its model: a recording proxy and a prompt-reuse analysis
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

# llama.cpp

- Server: llama.cpp version 10015 (5d3321c), `llama-server -m gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf -c 8192`
  (defaults otherwise: 4 slots, unified KV cache)
- Model: gemma-4-E2B-it, QAT Q4_K_XL, on an RTX 3070 Ti Laptop GPU
- Session: `drive.py --api openai --model gemma`

Both files record the same traffic: the driver talked to a digest-mode
proxy (`session.digest.jsonl`), which forwarded to a full-mode proxy
(`session.jsonl`), which forwarded to the server.

What it shows:

- Streamed responses carry `timings` but no `usage` (the client did not
  ask for `stream_options.include_usage`), so `prompt_tokens` comes from
  `cache_n + prompt_n` and `completion_tokens` from `predicted_n`.
- Changing the date at the end of the system prompt made the server
  re-evaluate the whole prompt (`cache_n` 0 on turn 4), likely because
  of the model's sliding-window attention. All 594 tokens are charged to
  `system_changed`.
- The side-call cost nothing: with 4 slots it did not evict the
  conversation.
- The server answers a request for an unknown model name with its one
  model, so there is no non-2xx record.

## Parallel agents

`parallel.jsonl` and `parallel-nocache.jsonl` record four agents running
at once through one proxy, against `llama-server ... -c 16384 -np 1`
(one slot); `parallel-nocache` adds `--cache-ram 0`. The drivers:

```
drive.py --role "a coding agent" --tools read_file,list_dir,write_file
drive.py --role "a subagent that only reads files" --tools read_file --script turns
drive.py --role "a test runner in another harness" --tools search_code,run_tests --script turns
drive.py --user-agent other-agent/2 --role "a reviewer" --tools read_file,list_dir,write_file --script turns
```

The first three share the user-agent `llmp-drive/1`. The subagent's
tools are a subset of the coding agent's, so both are client
`llmp-drive/1 #1`; the test runner has other tools and is
`llmp-drive/1 #2`; the reviewer is `other-agent/2`. The title
side-call has no tools and keeps the plain `llmp-drive/1`.

What they show:

- The turns of the four agents interleave (the coding agent's turns are
  up to six requests apart), and each agent is linked into exactly one
  conversation, with no false break.
- With the host-memory prompt cache on (the default, `--cache-ram
  8192`), one slot costs almost nothing: the server restores each
  conversation's cache when it comes back (reuse ~95%).
- With it off, the same traffic loses 77% of its prefill time, almost
  all of it to other conversations.