logprobe 0.4.0

Detect normalization errors, entropy bias, and truncation artifacts in LLM logprob data
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133

# logprobe demo

Real logprob data from GPT-4o-mini, GPT-4.1-nano, and GPT-2, plus a script to generate your own.

## Real API responses

These files are actual OpenAI API responses (April 2025), not synthetic data.

### GPT-4o-mini: creative writing (temperature=0.7, top-20)

```bash
logprobe diagnose demo/gpt4o_mini_creative.json
```

```
Normalization:  pass (log mass = -0.0119)
Missing mass:   0.0115 (150 positions)
Entropy bias:   +0.0155 bits (partial: 1.3450, normalized: 1.3605)
BPB:            byte data available

Validation: all validation checks passed (150 tokens)
```

Even with `top_logprobs=20`, 1.15% of probability mass is missing on average. The entropy bias is small (+0.016 bits) because top-20 captures most of the distribution — but it's still measurable and non-zero. At temperature=0.7 with creative content, 37 of 150 tokens fall below -1.0 logprob.

### GPT-4o-mini: code generation (temperature=0, top-5)

```bash
logprobe diagnose demo/gpt4o_mini_code.json
```

```
Normalization:  pass (log mass = -0.0000)
Missing mass:   0.0000 (56 positions)
Entropy bias:   -0.0000 bits (partial: 0.0083, normalized: 0.0083)
BPB:            byte data available

Validation: all validation checks passed (56 tokens)
```

Code generation at temperature=0 is nearly deterministic — top-5 captures effectively 100% of the mass. Perplexity: 1.0016. Missing mass and entropy bias are both negligible.

### GPT-2 vs GPT-4o-mini: the truncation gap

```bash
logprobe summary demo/gpt2_openai.json
logprobe summary demo/gpt4o_mini_creative.json
```

| Metric | GPT-2 (top-5) | GPT-4o-mini (top-20) |
|--------|---------------|---------------------|
| Perplexity | 5.39 | 1.95 |
| Missing mass | 30.0% | 1.15% |
| BPB | 0.533 | 0.202 |
| Entropy bias | +0.374 bits | +0.016 bits |

GPT-2's top-5 misses **30% of the probability mass**. Two positions have >50% missing, flagged UNRELIABLE. GPT-4o-mini is far more concentrated, but the bias is still non-zero — and at top-5 (the default for most API calls), it would be higher.

### Catching raw logits

```bash
logprobe diagnose demo/gpt2_logits_openai.json
```

```
Normalization:  FAIL (log mass = 12.0840 — likely raw logits)
Missing mass:   0.0000 (9 positions)
Entropy bias:   +11047975.3230 bits (partial: -11047973.7919, normalized: 1.5311)
BPB:            byte data available

Validation: 63 error(s) found
  [ERROR] nonpositive_logprob (position 0): token " quick" has positive logprob 4.2831 ...
  [ERROR] mass_exceeds_one (position 0): top_logprobs mass at position 0 is 4915.949726 ...
  ...
```

Raw logits passed as logprobs. Log mass = 12.08 (should be ~0 or negative). Every token has a positive "logprob", which is impossible for actual probabilities. Most tools would silently compute perplexity from these and give garbage.

## Other commands

```bash
# Per-token entropy breakdown
logprobe entropy demo/gpt4o_mini_creative.json

# Find low-confidence tokens with context
logprobe confidence -t -1.0 demo/gpt4o_mini_creative.json

# Sequence summary (mean logprob, perplexity, missing mass)
logprobe summary demo/gpt4o_mini_natural.json

# BPB
logprobe bpb demo/gpt4o_mini_natural.json

# Terminal visualization (color-coded by confidence)
logprobe highlight demo/gpt4o_mini_creative.json

# JSON output for piping to other tools
logprobe diagnose demo/gpt4o_mini_creative.json --json
```

## GPT-2 fixtures (reproducible, no API key needed)

The `gpt2_*.json` files are generated by running GPT-2 (124M) locally. Anyone can reproduce them:

```bash
pip install torch transformers
python demo/generate_logprobs.py \
    --score "The quick brown fox jumps over the lazy dog." \
    --format openai \
    --output my_logprobs.json
logprobe diagnose my_logprobs.json
```

## What's in each file

| File | Model | Format | Tokens | What it shows |
|------|-------|--------|--------|---------------|
| `gpt4o_mini_natural.json` | GPT-4o-mini | OpenAI | 59 | Factual response, top-5, temp=0 |
| `gpt4o_mini_creative.json` | GPT-4o-mini | OpenAI | 150 | Creative writing, top-20, temp=0.7 |
| `gpt4o_mini_code.json` | GPT-4o-mini | OpenAI | 56 | Code generation, top-5, temp=0 |
| `gpt41_nano.json` | GPT-4.1-nano | OpenAI | 1 | Single-token factual answer |
| `gpt2_openai.json` | GPT-2 | OpenAI | 9 | High missing mass (30%), 2 UNRELIABLE positions |
| `gpt2_logits_openai.json` | GPT-2 | OpenAI | 9 | Raw logits — logprobe catches this |
| `gpt2_vllm.json` | GPT-2 | vLLM | 9 | Same data in flat token-array format |
| `gpt2_stream.jsonl` | GPT-2 | JSONL | 9 | Minimal per-token, no top-k |

## Verifying fixture consistency

```bash
python demo/verify_fixtures.py
```

Checks that all byte arrays match UTF-8 encodings, probabilities sum to <= 1 for normalized files, top_logprobs are sorted, and the logits file has the expected positive values.