Module byte_map 

The GPT-2 “byte-to-unicode” alphabet.

Why this exists

Byte-level BPE’s whole selling point is that the base alphabet is the 256 possible byte values, so any input – valid UTF-8, invalid UTF-8, raw binary – is representable with no <UNK> token. But a tokenizer.json is itself a JSON document, and JSON strings cannot contain raw control bytes (0x00-0x1F), the byte 0x22 (") needs escaping, whitespace bytes collide with JSON’s own whitespace handling in careless implementations, and a lone unpaired continuation byte (0x80-0xBF) is not valid UTF-8 on its own – so it cannot appear literally inside a JSON string at all.

GPT-2’s encoder.py solves this with a reversible byte <-> char bijection: bytes that are already “nice” printable characters (roughly ASCII !..~ plus a couple of Latin-1 supplement ranges) map to themselves, and the remaining ~68 awkward bytes (controls, space, DEL, and the unassigned/continuation bytes in 0x7F-0xA0/0xAD) are remapped to otherwise-unused codepoints starting at U+0100. Every one of the 256 resulting characters is then representable as an ordinary JSON string character, so a tokenizer.json vocab can hold "Ġhello" (where Ġ stands in for the space byte 0x20) instead of needing binary escapes.

This mapping is a serialization concern only. Once a vocab is loaded, crate::vocab::Vocab stores each token’s canonical raw bytes, and the rest of this crate (encode, decode, merges) never looks at mapped characters again – see crate::loader for the one place this map is actually used.

Reference: https://github.com/openai/gpt-2/blob/master/src/encoder.py#L9

Functions

byte_to_unicode

Maps a raw byte to its GPT-2 byte-level-alphabet character.

decode_mapped_token

Decodes a mapped-alphabet token string (as it appears verbatim in a tokenizer.json vocab key) back into the raw bytes it represents.

unicode_to_byte

Inverse of byte_to_unicode: recovers the raw byte a mapped character stands for, or None if c is not one of the 256 characters in the byte-level alphabet (which means the vocab this came from is not a valid byte-level BPE vocab).