# dynamo-tokenizers
Efficient, versatile tokenization for LLM inference. Wraps HuggingFace and TikToken tokenizers (plus a FastTokenizer hybrid mode) behind a small encode/decode/sequence API designed for streaming detokenization.
## Features
- **Multiple backends.** HuggingFace `tokenizers`, OpenAI `tiktoken`, and a FastTokenizer hybrid behind one trait.
- **Streaming-friendly.** `Sequence` tracks incremental token-id appends and emits text deltas without re-decoding the full prefix.
- **Prefix caching.** `CachedTokenizer` records prefix tokenizations at special-token boundaries; repeated prompts that share a system prefix re-encode only the trailing suffix, turning O(N) work into O(suffix_len).
- **Hash verification.** Detect tokenizer drift across model versions.
## Quick start
```rust
use dynamo_tokenizers::hf::HuggingFaceTokenizer;
use dynamo_tokenizers::traits::{Encoder, Decoder};
// tokenizer.json downloaded from any HuggingFace model repo
let tokenizer = HuggingFaceTokenizer::from_file("/path/to/tokenizer.json")
.expect("load tokenizer");
let encoding = tokenizer.encode("Your sample text here")
.expect("encode");
println!("{:?}", encoding);
let decoded = tokenizer.decode(&encoding.token_ids, false)
.expect("decode");
assert_eq!(decoded, "Your sample text here");
```
## Streaming detokenization with `Sequence`
```rust
use dynamo_tokenizers::{Sequence, Tokenizer};
use std::sync::Arc;
let tokenizer = Tokenizer::from(Arc::new(tokenizer));
let mut sequence = Sequence::new(tokenizer.clone());
sequence.append_text("Your sample text here")
.expect("append text");
// As each new token id is produced by the engine, append it
// and get back just the incremental text delta:
let delta = sequence.append_token_id(1337)
.expect("append token_id");
```
## Prefix caching with `CachedTokenizer`
Multi-turn chat workloads re-tokenize a large shared prefix (system prompt +
prior turns) on every request. `CachedTokenizer` wraps any tokenizer and caches
prefix tokenizations at special-token boundaries (e.g. `<|im_start|>`,
`<|im_end|>`, `<s>`, `</s>`). On a hit it merges the cached prefix tokens with a
fresh encode of the trailing suffix only.
Boundaries are taken **only** immediately after a registered special token —
those are atomic in BPE, so the merge is exact:
`encode(prefix) + encode(suffix) == encode(prefix + suffix)`. There is no
whitespace/punctuation fallback; the cache prefers a miss over a corrupt split.
```rust
use dynamo_tokenizers::{CachedTokenizer, HuggingFaceTokenizer};
use dynamo_tokenizers::traits::{Encoder, Tokenizer};
use std::sync::Arc;
let hf = HuggingFaceTokenizer::from_file("/path/to/tokenizer.json")
.expect("load tokenizer");
let inner: Arc<dyn Tokenizer> = Arc::new(hf);
// The atomic special tokens the model uses as turn delimiters.
// An empty list disables caching: encode/encode_batch pass straight through.
let specials = vec!["<|im_start|>".to_string(), "<|im_end|>".to_string()];
let cached = CachedTokenizer::new(inner, specials, 256 * 1024 * 1024); // 256 MiB budget
let stats = cached.cache_stats();
println!("hits={} misses={} hit_rate={:.2}", stats.hits, stats.misses, stats.hit_rate);
```
TikToken models expose the exact special-token strings registered with their BPE, so
the same cache can be constructed without duplicating model metadata:
```rust
use dynamo_tokenizers::{CachedTokenizer, TikTokenTokenizer};
use dynamo_tokenizers::traits::Tokenizer;
use std::sync::Arc;
let tiktoken = TikTokenTokenizer::from_file_auto("/path/to/tiktoken.model")
.expect("load tokenizer");
let specials = tiktoken.special_tokens().to_vec();
let inner: Arc<dyn Tokenizer> = Arc::new(tiktoken);
let cached = CachedTokenizer::new(inner, specials, 256 * 1024 * 1024);
```
Entries are evicted by approximate LRU once `max_memory_bytes` is exceeded. The
cache lives as long as the `CachedTokenizer` instance. Use `.with_observer(...)`
to push request-level hit/miss events into your metrics. Use
`.with_token_observer(...)` to receive exact cached and uncached token counts
after each successful encode while L1 is active. Partial hits report both
categories, so consumers can increment `cached_tokens_total` and
`uncached_tokens_total` counters and derive a token-level reuse ratio.