Struct FastTokenizer 

pub struct FastTokenizer { /* private fields */ }

Hybrid tokenizer: fast BPE encoding via fastokens, decoding via HuggingFace.

Both backends are loaded from the same tokenizer.json file.

Implementations

impl FastTokenizer

pub fn from_file(path: &str) -> Result<Self>

Trait Implementations

impl Decoder for FastTokenizer

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for FastTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

fn encode_segments(&self, segments: &[EncodeSegment<'_>]) -> Result<Encoding>

Encode Kimi K3-style renderer segments while preserving trusted control-token and untrusted content boundaries.

impl Tokenizer for FastTokenizer

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache.

fn vocab_size(&self) -> Option<usize>

Vocabulary cardinality including added tokens, when the backend can expose one. None for backends without a bounded id space or vocabulary introspection.

fn token_to_id(&self, token: &str) -> Result<Option<TokenIdType>>

Resolve a token string to its vocabulary id, when the backend supports lookup. Ok(None) when the token is not in the vocabulary. Err when this backend cannot do id lookup at all — kept distinct from a genuine vocabulary miss so callers can tell “unsupported” from “looked up, not found”.

fn special_token_ids(&self) -> Result<Vec<TokenIdType>>

Ids of added tokens marked special (e.g. BOS/EOS/PAD/control tokens), as distinct from ordinary vocabulary tokens. Ok(vec![]) for backends that genuinely have none, or that do not distinguish special from ordinary vocabulary ids. Err when the backend cannot enumerate added tokens at all — an empty Vec alone cannot carry that distinction.

fn num_special_tokens_added(&self) -> Result<usize>

Count of special tokens encode’s add_special_tokens: true path would add to a bare encoding (e.g. BOS/EOS), available without performing an encode. Ok(0) for backends that genuinely add none. Err when the backend cannot determine this at all — a plain 0 alone cannot carry that distinction, and this value feeds token-budget accounting where a silent 0 would under-count rather than fail loudly.

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.