llm-tokenizer-1.4.0
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

llm-tokenizer 1.4.0

LLM tokenizer library with caching and chat template support

Crate
Source
Builds
Feature flags

..
deepseek_v32.rs
deepseek_v4.rs
kimi_k25_tools.rs
mod.rs

1
2
3

pub mod deepseek_v32;
pub mod deepseek_v4;
pub mod kimi_k25_tools;