Module tokens 

The canonical ByT5 byte table (SPEC.md §4) — fixed, no vocab files.

UTF-8 byte b maps to id b + 3; one trailing EOS (id 1) is appended on encode; pad = 0, unk = 2. Ids are NOT raw bytes — feeding text.as_bytes() directly produces silent garbage on real models (the classic ByT5 interop bug; SPEC.md §4 warning).

Constants

BYTE_OFFSET

Byte offset: id = byte + 3.

EOS_ID

Special id 1: end of sequence (stops decode; appended on encode).

PAD_ID

Special id 0: padding (skipped on decode).

UNK_ID

Special id 2: unknown (skipped on decode).

Functions

decode

tokens.decode: stop at EOS; skip pad/unk; (id - 3) mod 256 per byte; reassemble as UTF-8 (lossy, like the reference loader’s errors="replace").

encode

tokens.encode: UTF-8 bytes b -> ids b+3, then one trailing EOS.