Skip to main content

Crate simd_tokenizer

Crate simd_tokenizer 

Source
Expand description

SIMD-style token counting without unsafe code.

Provides a pure-Rust, SWAR-boosted token estimator — extracted from clawdius — for environments where tiktoken-rs is unavailable (WASM builds, minimal builds, or when the C/C++/Python tokenizer runtimes cannot be linked), or where a fast estimate beats a slow exact count.

§Architecture

TypeDescription
TokenCounterTrait abstracting token counting backends.
SimdWhitespaceTokenizerSWAR byte-scan for whitespace boundaries.
TokenEstimatorEnum dispatching to the best available backend.
[TiktokenCounter]Exact cl100k_base / o200k_base counting (feature tiktoken).

§The “SIMD” in the name, honestly

clawdius’s original scanner used std::arch SSE2/NEON intrinsics behind unsafe. This port replaces it with SWAR (“SIMD Within A Register”): 8 bytes are loaded per iteration into a u64 and classified with branch-free bit tricks, then run boundaries are counted with popcount. It is fully portable (no cfg(target_feature), no runtime detection) and lets the whole crate declare #![forbid(unsafe_code)] while staying on stable Rust.

Performance characteristics, measured honestly:

  • Several times faster than a naive branchy per-byte loop on large inputs, and much faster than str::split_whitespace().count() (which allocates UTF-8-boundary-aware iterators).
  • Not as fast as hand-written SSE2/AVX2 for huge buffers; SWAR moves 8 bytes per instruction-stream step vs 16-32 for vector ISAs.
  • For token estimation granularity (the result feeds a heuristic blend, not an exact invoice) this trade is usually invisible end-to-end.

§Estimate semantics

SimdWhitespaceTokenizer splits on any byte ≤ 0x20 (space, tab, newline, CR, and other C0 controls), then blends the word count with a ~4 chars/token heuristic and a punctuation overhead term — see estimate_from_whitespace_splits. It approximates BPE behavior for budgeting and rate-limit decisions; it is not a tokenizer table.

§Feature flags

  • tiktoken (default off) — adds [TiktokenCounter] for exact cl100k_base and o200k_base counts ([TiktokenCounter::new] and [TiktokenCounter::o200k]) and makes TokenEstimator::new prefer tiktoken (cl100k_base).

§Guarantees

  • #![forbid(unsafe_code)] — including the SIMD-style scan.
  • #![deny(missing_docs)] — the entire public API is documented.
  • Counting is deterministic, allocation-free, and never panics on any input (verified by property tests over arbitrary byte strings, including non-UTF-8 and truncated UTF-8 sequences).

Structs§

SimdWhitespaceTokenizer
SIMD-style (SWAR) whitespace-split token estimator.

Enums§

TokenEstimator
Enum that dispatches to the best available token counter.

Traits§

TokenCounter
Abstraction over token counting backends.

Functions§

estimate_from_whitespace_splits
Estimate a BPE-approximate token count from whitespace split statistics.
scalar_count_splits
Portable scalar reference implementation of [SimdWhitespaceTokenizer:: count_splits]. Public so callers (and tests) can cross-verify the SWAR path byte-for-byte.