Crate toktrie_tiktoken 

This crate integrates the tiktoken BPE tokenizer (used by OpenAI models) with toktrie, providing a TokenizerEnv implementation backed by tiktoken’s CoreBPE.

Structs

TikTokenBPE

A tiktoken BPE tokenizer paired with a TokTrie for efficient constrained-decoding support. Implements TokenizerEnv.