Expand description
SIMD-style token counting without unsafe code.
Provides a pure-Rust, SWAR-boosted token estimator — extracted from
clawdius — for environments where
tiktoken-rs is unavailable (WASM builds, minimal builds, or when the
C/C++/Python tokenizer runtimes cannot be linked), or where a fast
estimate beats a slow exact count.
§Architecture
| Type | Description |
|---|---|
TokenCounter | Trait abstracting token counting backends. |
SimdWhitespaceTokenizer | SWAR byte-scan for whitespace boundaries. |
TokenEstimator | Enum dispatching to the best available backend. |
[TiktokenCounter] | Exact cl100k_base / o200k_base counting (feature tiktoken). |
§The “SIMD” in the name, honestly
clawdius’s original scanner used std::arch SSE2/NEON intrinsics behind
unsafe. This port replaces it with SWAR (“SIMD Within A Register”):
8 bytes are loaded per iteration into a u64 and classified with
branch-free bit tricks, then run boundaries are counted with popcount.
It is fully portable (no cfg(target_feature), no runtime detection) and
lets the whole crate declare #![forbid(unsafe_code)] while staying on
stable Rust.
Performance characteristics, measured honestly:
- Several times faster than a naive branchy per-byte loop on large inputs,
and much faster than
str::split_whitespace().count()(which allocates UTF-8-boundary-aware iterators). - Not as fast as hand-written SSE2/AVX2 for huge buffers; SWAR moves 8 bytes per instruction-stream step vs 16-32 for vector ISAs.
- For token estimation granularity (the result feeds a heuristic blend, not an exact invoice) this trade is usually invisible end-to-end.
§Estimate semantics
SimdWhitespaceTokenizer splits on any byte ≤ 0x20 (space, tab,
newline, CR, and other C0 controls), then blends the word count with a
~4 chars/token heuristic and a punctuation overhead term — see
estimate_from_whitespace_splits. It approximates BPE behavior for
budgeting and rate-limit decisions; it is not a tokenizer table.
§Feature flags
tiktoken(default off) — adds [TiktokenCounter] for exactcl100k_baseando200k_basecounts ([TiktokenCounter::new] and [TiktokenCounter::o200k]) and makesTokenEstimator::newprefer tiktoken (cl100k_base).
§Guarantees
#![forbid(unsafe_code)]— including the SIMD-style scan.#![deny(missing_docs)]— the entire public API is documented.- Counting is deterministic, allocation-free, and never panics on any input (verified by property tests over arbitrary byte strings, including non-UTF-8 and truncated UTF-8 sequences).
Structs§
- Simd
Whitespace Tokenizer - SIMD-style (SWAR) whitespace-split token estimator.
Enums§
- Token
Estimator - Enum that dispatches to the best available token counter.
Traits§
- Token
Counter - Abstraction over token counting backends.
Functions§
- estimate_
from_ whitespace_ splits - Estimate a BPE-approximate token count from whitespace split statistics.
- scalar_
count_ splits - Portable scalar reference implementation of [
SimdWhitespaceTokenizer:: count_splits]. Public so callers (and tests) can cross-verify the SWAR path byte-for-byte.