Crate unigram 

unigram — a bijective codec between bytes and words that cost one LLM token.

Machine identifiers spend their lives being looked at: handed to a language model and asked back, printed in a log, quoted in an error, read off a page by whoever is debugging at the time. This crate carries them as words, so that an id becomes something a reader can hold — created office access world can be said out loud, told apart from its neighbour at a glance, and recognised again an hour later, which is what a name is for.

use unigram::UnigramId;

let id = UnigramId::from_bytes([0x3d, 0x9a, 0x00, 0xff]);
assert_eq!(id.to_string(), "created office access world");
assert_eq!(UnigramId::<4>::parse("created office access world").unwrap(), id);

The words come from a fixed alphabet of 256, and two properties follow from that size — they are the whole design:

• One word is exactly one byte. Encoding is a table lookup per byte with no bit-packing, no padding, and no length convention; decoding is its inverse. Every nonempty byte string has exactly one encoding, and every sequence of alphabet words decodes.
• Every word is one token, wherever a space precedes it. An encoded value therefore costs one token per byte. Against hex of the same payload, under Claude:

payload	bits	hex (mean / sample max)	unigram
4 bytes	32	6.0 / 8	4
8 bytes	64	11.1 / 14	8
16 bytes	128	21.5 / 25	16
32 bytes	256	42.2 / 49	32

Roughly a quarter cheaper on average, but the flat cost matters more than the mean: hex cost swings with the value, so a budget built on it has to assume the worst case. Those are Claude’s numbers over 64 deterministic payloads per size, so the right-hand column is a sample maximum rather than a proven bound. The margin narrows under the GPT-4 vocabularies, where 32 bytes of hex average 37.1, and widens sharply under Llama’s SentencePiece, at 58.2 against the same flat 32. verify-alphabet.py prints the table for every family it checks.

What the token claim covers

Every entry costs one token space-prefixed and bare, in all five families, so an N-byte value costs exactly N tokens at the start of a string, after a space, inside a JSON string, and mid-sentence. The only surcharge is a punctuation character immediately before the value — a backtick or an open parenthesis — which adds one token, once, and is really the punctuation paying for itself. Measured for a 4-byte value against an ideal of 4, sweeping every one of the 256 entries through the opening and closing positions and keeping the worst:

context	GPT-4o	GPT-3.5/4	GPT-3	GPT-2	Llama	Claude
start of string	+0	+0	+0	+0	+0	+0
in prose, X.	+0	+0	+0	+0	+0	+0
JSON "id":"X"	-1	+0	+0	+0	+0	+1
after a newline	+0	+0	+0	+0	+0	+1
after id:	-1	-1	-1	-1	-1	+0
markdown `X`	+1	+1	+1	+1	+1	+0
after (	+1	+1	+1	+1	+1	+0

So the guarantee is one token per byte, plus at most one for punctuation immediately before it — a constant, never anything that scales with the payload, and negative where the context ends in a space the value absorbs.

This is a property of the table rather than a happy accident, and it was not free. 0.2.0 shipped an alphabet in which 22 entries cost two or three tokens bare, so a value beginning with council cost N+2 at the start of a string — and its verifier tested one payload whose opening word happened to be cheap. Both the table and the sweep are fixed. The sweep is why the claim above needs no list of exceptions.

Size

This encoding’s weakness is size. Mean marginal tokens saved against each alternative, over 200 deterministic payloads per cell, ranged across all five tokenizers — positive means unigram is cheaper:

payload	vs hex	vs base64url	vs base58
4 bytes	+1.1 … +3.9	+0.6 … +2.3	+0.8 … +2.6
8 bytes	+1.9 … +7.1	+0.1 … +2.8	−0.0 … +2.9
16 bytes	+3.2 … +13.4	−0.7 … +5.3	−0.6 … +5.2
32 bytes	+5.6 … +26.2	−2.5 … +9.2	−1.6 … +10.0

Hex loses everywhere. At 4 bytes so does everything else, in every family. Above 16 bytes base64url costs a token or two less under the GPT vocabularies, which have memorised base64 fragments; under Claude it costs five to nine more.

It was built for nonce and correlation-id sizes, where it wins outright. A 32-byte digest is a worse fit — the token margin is gone, and the value is 224 characters across three wrapped lines rather than something read at a glance.

Whatever the size, no alternative has the flat column: every value of a given width costs the same, so a budget is known before minting, where hex and base64 must both be provisioned for their worst case.

The alphabet

256 entries of lowercase ASCII English, 4 to 10 characters, under five constraints:

• One token, space-prefixed and bare, under every tokenizer the verifier pins: OpenAI’s r50k_base, p50k_base, cl100k_base, and o200k_base; the hf-internal-testing/llama-tokenizer SentencePiece artifact at revision d02ad6cb; and ctok 1.0.0’s "5.0" counter, an offline reconstruction of Claude’s tokenizer rather than Anthropic’s own. ctok’s figures for this table have been checked against Anthropic’s official count_tokens endpoint on claude-opus-5 — all 256 entries, spaced and bare, and the cost table — and they agree exactly; verify-claude.py reruns that. Those exact artifacts are the claim, not every past or future model sharing a name, and in particular not Llama 3, which tokenizes with tiktoken rather than the SentencePiece model here.
• No two entries within one character edit, and none a prefix or a suffix-derivative of another. A slipped character, a dropped suffix, or a completed word lands outside the alphabet rather than on a different valid entry. For a value that must prove it is intact, see CheckedUnigramId.
• Nothing charged — no death, violence, race, gender, religion, or politics. These strings surface unbidden in transcripts, logs, and user-facing errors.
• No function words. A value made of that, which, and would reads as damaged prose rather than as a name.
• Frozen, which is the next section.

Why the join is a space

Tokenizer vocabularies hold their canonical word entries space-prefixed, so the space between two words is absorbed into the word that follows it and costs nothing. No other separator is free. Measured across all five families, a hyphen, comma, pipe, slash, or newline becomes a token of its own in every one of them, taking an eight-byte value from 8 tokens to 15 — the join costing almost as much as the payload. GPT-3.5/4 and GPT-4o absorb _ and . for free; no other family absorbs anything. Encoded values travel inside quoted strings in practice, where embedded spaces are free.

Reading a value back

Two parsers, because they answer different questions.

UnigramId::parse and decode are canonical: lowercase alphabet words joined by exactly one space, nothing else. That is what belongs at a boundary where the value is about to be trusted — a database key, an API parameter, an authorization check — because a canonical parser has exactly one accepted spelling per value, and cannot be talked into treating some other string as one.

UnigramId::recover and decode_recovered are tolerant: any run of characters that is not an ASCII letter separates words, and case is ignored, so a value that came back hyphenated, re-wrapped, comma-joined, quoted, or shouted still yields the bytes that were sent. It reads the whole input, so isolate the candidate first.

Both refuse an unknown word and name it.

Why the alphabet is 256 and not larger

A wider alphabet would carry more bits per token, so it is worth saying why this one stops where it does. Of the roughly 65,000 space-prefixed lowercase words in the largest vocabulary, 6,654 are single-token in all five families; 5,452 of those are 4 to 11 ASCII characters; and 640 of those survive Claude, whose tokenizer is by far the narrowest of the five. Spacing them a character edit apart leaves about 509.

So the ceiling is 512 entries — log2(509) ≈ 8.99 bits per token against the 8 here, and 9 does not divide 8. Bit-packing 9-bit symbols would save nothing at all on a 4-byte value (32 bits still needs 4 words), one token on a 16-byte value, and three on a 32-byte one, in exchange for the byte-indexed table, the claim that one word is one byte, and a codec that can be described in a sentence. It is not a trade worth making, and this is therefore not the densest possible encoding — it is the densest byte-aligned one.

Other scripts do not change this. CJK is denser on the page but agrees across families far less: 39 characters are single-token in all five, which does not reach even 256. Accented Latin is worse — 5 words survive. The binding constraint was never English; it is the intersection itself.

Nor does an existing wordlist. BIP39 holds 2048 words, which would be 11 bits each, but it was chosen for human transcription rather than for tokenizers: only 349 of them are single-token both ways across all five families, Claude again being the narrowest at 366. Rounded down to a power of two that is 256 entries and 8 bits per token — the same density this reaches, from a list that carries no bare-cost or surrounding-context guarantee.

The alphabet is the wire format

ALPHABET is frozen. Byte n is ALPHABET[n], all 256 slots are occupied, and changing any entry changes what every previously issued value decodes to. There is no append: the array is full. A test pins the table’s digest so that an edit has to be deliberate, and if a different table is ever wanted it belongs beside this one under a new name and a new FORMAT_VERSION, with this decoder kept forever.

Nothing in an encoded value says which table produced it, so a system that stores these must record the format version alongside them, or accept that it can never change tables.

Changing the alphabet

Nothing here tokenizes, at runtime or under test: the OS CSPRNG is this crate’s only dependency at any stage. So cargo test covers the codec’s behaviour and the table’s structural properties — 256 entries, sorted, unique, 4 to 11 lowercase ASCII characters, no two within one character edit, no entry reachable from another by adding or removing a suffix, and the frozen digest — and says nothing about cost.

Every cost claim above is checked instead by verify-alphabet.py, beside this file. It reads ALPHABET straight out of this source — a copy would drift — and re-measures each entry against all five tokenizer families, along with the composed per-byte cost in each surrounding context, the margin over hex, and the choice of separator:

uv run verify-alphabet.py

Run it after any edit to ALPHABET. A green test suite alone establishes none of what this crate is named for, and an edit that satisfies every test here can still break every cost claim above.

Structs

CheckedUnigramId

An identifier of N bytes carrying a trailing check word, rendered as N + 1 alphabet words.

UnigramId

An identifier of N bytes, rendered as N alphabet words.

Enums

DecodeError

Why a sequence of words could not be decoded.

ParseError

Why a string could not be parsed as a UnigramId of a particular width.

Constants

ALPHABET

The 256-word alphabet, sorted, indexed by the byte each word encodes.

FORMAT_VERSION

The version of the encoding this crate implements.

Functions

decode

Decode the canonical form: lowercase alphabet words joined by single spaces.

decode_recovered

Decode tolerantly, forgiving the reformatting a round trip introduces.

encode

Encode bytes as space-joined alphabet words, one word per byte.

try_mint

Mint bytes bytes of fresh entropy, encoded.