# unigram
A bijective codec between bytes and words that cost exactly one LLM token.
The same four bytes, twice:
```text
hex a14ed61a 6 tokens
unigram people error social career 4 tokens
```
The same sixteen:
```text
hex 8623a771b764ce50bb85371ff65aebe9 21 tokens
unigram login city population income question head season
example region location count century update
football task table 16 tokens
```
Now corrupt one character of each.
```text
hex a14ed61a -> a14ed61b still a valid digest, and nothing can tell
unigram people error social career -> people errer social career
Err(UnknownWord { position: 1, word: "errer" })
```
That is the whole pitch. Machine identifiers are routinely handed to a language model
and asked back — an acknowledgement token, a digest, a correlation id — and hex is
the worst available carrier for that trip. It is expensive, because a hex run shreds
into a fragment every character or two under every tokenizer. And it is *silently*
fragile, because every character is drawn from the same sixteen, so every corruption
of a hex digest is another hex digest.
## Using it
```rust
let words = unigram::encode(&[0x3d, 0x9a, 0x00, 0xff]);
assert_eq!(words, "department number access world");
assert_eq!(unigram::decode(&words)?, vec![0x3d, 0x9a, 0x00, 0xff]);
unigram::mint(4); // 32 fresh bits from the OS CSPRNG, 4 tokens
unigram::matches(issued, presented); // comparison that forgives a round trip
```
`decode` is liberal in what it accepts: any run of characters that is not an ASCII
letter separates words, and case is ignored — so a value that came back hyphenated,
re-wrapped across lines, comma-joined, quoted, or shouted still decodes to the bytes
that were sent. It is exact in what it returns, though: an unknown word is refused
and named, never skipped or guessed at.
`matches` compares decoded bytes when both sides are encoded values, and normalized
strings otherwise — so values issued in some older format keep matching themselves
without a migration.
## What it costs
One word is one byte, and one word is one token, so an encoded value costs one token
per byte — the same for every value. Against hex of the same payload, under Claude:
| 4 bytes | 32 | 6.0 / 8 | 4 |
| 8 bytes | 64 | 11.1 / 14 | 8 |
| 16 bytes | 128 | 21.5 / 25 | 16 |
| 32 bytes | 256 | 42.2 / 49 | 32 |
Roughly a quarter cheaper on average — but the flat cost matters more than the mean.
Hex swings with the value, so a token budget built on it has to assume the worst
case. This one is known before the value is minted.
The margin narrows under the GPT-4 vocabularies, where 32 bytes of hex average 37.1,
and widens sharply under Llama's SentencePiece, where the same payload averages 58.2
against the same flat 32.
## Choosing a length
Length is the entropy budget and the token budget at once — the two cannot drift
apart, which is most of why this is easier to size than hex.
| 2 | 16 | 65,536 | fewer than 1 |
| 3 | 24 | 16.8 million | 5 |
| 4 | 32 | 4.3 billion | 92 |
| 6 | 48 | 281 trillion | 23,700 |
| 8 | 64 | 1.8 × 10¹⁹ | 6 million |
| 16 | 128 | 3.4 × 10³⁸ | 2.6 × 10¹⁶ |
| 32 | 256 | 1.2 × 10⁷⁷ | 4.8 × 10³⁵ |
The right column is the birthday bound, `k ≈ √(2·N·p)`, and it is the column to size
against: collisions arrive at the square root of the space, not at the space. Sixteen
words is a UUID's width, thirty-two a SHA-256's.
Two questions hide in that table and it answers only one. **Collision** is the right
column — how many values may be outstanding before two coincide. **Guessing** is
separate: `mint` draws from the OS CSPRNG, so every bit is unpredictable, but four
words is 4.3 billion candidates, which is an afternoon for anything that can ask
freely. Four words suits a value that is scoped, short-lived, and rate-limited — an
acknowledgement nonce, a correlation id. A value a stranger can grind at wants eight
or more, and at equal entropy the words are still the cheaper carrier.
## The join is a space, deliberately
Tokenizer vocabularies hold their canonical word entries space-prefixed, so the space
between two words is absorbed into the word that follows it and costs nothing. No
other separator is free. Measured across all five families, an eight-byte value:
| space | 8 | 8 | 8 | 8 | 8 | 8 |
| `_` `.` | 8 | 8 | 15 | 15 | 15 | 15 |
| `-` | 11 | 9 | 15 | 15 | 15 | 15 |
| `,` `\n` | 13–15 | 12–15 | 15 | 15 | 15 | 15 |
The join would cost almost as much as the payload. Encoded values travel inside
quoted strings in practice, where embedded spaces are free — and `decode` accepts
every one of those separators anyway, so a value that comes back joined differently
is not a value that is lost.
## The alphabet
256 entries of lowercase ASCII English, 4 to 11 characters, chosen under four
constraints:
- **One token** under Claude, GPT-2/3 (`r50k`, `p50k`), GPT-3.5/4 (`cl100k`), GPT-4o
(`o200k`), and Llama's SentencePiece — spanning both the BPE and SentencePiece
families.
- **No two entries within one character edit of each other.** This is what puts a
slipped character outside the alphabet instead of on a different valid word, and it
is the property hex cannot have at any length.
- **Nothing charged** — no death, violence, race, gender, religion, or politics.
These strings surface unbidden in transcripts, logs, and user-facing errors.
- **No entry is an inflection of another**, so a dropped plural cannot silently
decode to a different byte.
The table is indexed by the byte each word encodes, so it is appended to, never
rearranged: reordering an entry changes what every previously issued value decodes to.
## Verifying it
The crate depends on nothing but the OS CSPRNG, at runtime or under test, and never
tokenizes. `cargo test` covers the codec and the table's structure; it says nothing
about cost.
Every number on this page is printed by `verify-alphabet.py`, which reads the
alphabet straight out of `src/lib.rs` and re-measures it against all five families:
```bash
uv run verify-alphabet.py
```
Run it after any edit to the table. A green test suite alone establishes none of what
this crate is named for.
## License
MIT.