Crate llama_tokenizer 

llama-tokenizer

Deterministic tokenization for llama.rs.

This crate provides:

• A Tokenizer trait for pluggable tokenization backends
• A reference whitespace tokenizer for testing
• Streaming decoding with UTF-8 handling
• Chat template support (future)

Structs

DecodingState

Streaming decoding state for handling partial UTF-8 sequences.

WhitespaceTokenizer

Reference whitespace tokenizer for Milestone A testing.

Enums

TokenizerError

Error type for tokenization operations.

Traits

Tokenizer

Core tokenizer trait. Implementations can be swapped without changing app code.

Type Aliases

TokenizerResult