snaptokens 0.2.2

Fast BPE tokenization from local Hugging Face tokenizer.json files
Documentation