Expand description
Character-level tokenizer, matching nanoGPT’s shakespeare_char prepare
step: the vocabulary is the sorted set of distinct characters in the
corpus, and a token id is that character’s index in the sorted order.
For Tiny Shakespeare this yields 65 tokens, which shrinks the (weight-tied) embedding table from 9.65M parameters at the GPT-2 BPE vocab to 24,960 — the difference between a 45.9 MB checkpoint that is 84% untrained random rows and a 43.1 MB one where every parameter does work.