tiktoken 4.0.1

A high-performance pure-Rust implementation of OpenAI's tiktoken BPE tokenizer
Documentation