pretokie-0.0.3
Platform
Feature flags

docs.rs

Rust

pretokie 0.0.3

Fast, zero-allocation pretokenizers for BPE tokenizers

Crate
Source
Builds
Feature flags

..
bert.rs
cl100k.rs
deepseek.rs
gpt2.rs
mod.rs
o200k.rs
qwen.rs
regex.rs
smollm.rs
voyage.rs