pretokie-0.0.2
Platform
Feature flags

docs.rs

Rust

pretokie 0.0.2

Fast, zero-allocation pretokenizers for BPE tokenizers

Crate
Source
Builds
Feature flags

..
bert.rs
cl100k.rs
deepseek.rs
gpt2.rs
mod.rs
o200k.rs
qwen.rs
regex.rs
smollm.rs
voyage.rs