tiktoken-rust 0.2.0

a fast BPE tokeniser for use with OpenAI's models
Documentation