splintr 0.18.0

Fast Rust tokenizer (BPE + SentencePiece + WordPiece) with Python bindings
Documentation

splintr

There is currently very little information to present on this page because Docs.rs has only limited support for extracting structured feature metadata from Cargo crates. This issue is tracked in Rust RFC #3416. Check this library's main docs, readme, and Cargo.toml in case its authors have documentation for features available there instead.

This version has 15 feature flags, 2 of them enabled by default.

default

rayon (default)

regexr-jit (default)

pcre2

python

vocab-cl100k

This feature flag does not enable additional features.

vocab-deepseek

This feature flag does not enable additional features.

vocab-glm

This feature flag does not enable additional features.

vocab-gpt-oss

vocab-kimi

This feature flag does not enable additional features.

vocab-llama3

This feature flag does not enable additional features.

vocab-mistral

This feature flag does not enable additional features.

vocab-o200k

This feature flag does not enable additional features.

vocab-qwen

This feature flag does not enable additional features.

vocab-whisper

This feature flag does not enable additional features.

vocabs