splintr 0.19.1

Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings
Documentation

splintr

There is currently very little information to present on this page because Docs.rs has only limited support for extracting structured feature metadata from Cargo crates. This issue is tracked in Rust RFC #3416. Check this library's main docs, readme, and Cargo.toml in case its authors have documentation for features available there instead.

This version has 22 feature flags, 2 of them enabled by default.

default

rayon (default)

regexr-jit (default)

pcre2

python

vocab-cl100k

vocab-deepseek

vocab-gemma2

vocab-gemma3

vocab-gemma4

vocab-glm

vocab-gpt-oss

vocab-kimi

vocab-llama2

vocab-llama3

vocab-mistral

vocab-modernbert

vocab-o200k

vocab-olmo2

vocab-phi4

vocab-qwen

vocab-whisper

vocabs