1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
//! SIMD-style token counting without unsafe code.
//!
//! Provides a pure-Rust, SWAR-boosted token estimator — extracted from
//! [clawdius](https://github.com/WyattAu/clawdius) — for environments where
//! `tiktoken-rs` is unavailable (WASM builds, minimal builds, or when the
//! C/C++/Python tokenizer runtimes cannot be linked), or where a fast
//! *estimate* beats a slow *exact* count.
//!
//! # Architecture
//!
//! | Type | Description |
//! | --- | --- |
//! | [`TokenCounter`] | Trait abstracting token counting backends. |
//! | [`SimdWhitespaceTokenizer`] | SWAR byte-scan for whitespace boundaries. |
//! | [`TokenEstimator`] | Enum dispatching to the best available backend. |
//! | [`TiktokenCounter`] | Exact `cl100k_base` / `o200k_base` counting (feature `tiktoken`). |
//!
//! # The "SIMD" in the name, honestly
//!
//! clawdius's original scanner used `std::arch` SSE2/NEON intrinsics behind
//! `unsafe`. This port replaces it with **SWAR** ("SIMD Within A Register"):
//! 8 bytes are loaded per iteration into a `u64` and classified with
//! branch-free bit tricks, then run boundaries are counted with `popcount`.
//! It is fully portable (no `cfg(target_feature)`, no runtime detection) and
//! lets the whole crate declare `#![forbid(unsafe_code)]` while staying on
//! stable Rust.
//!
//! Performance characteristics, measured honestly:
//!
//! - Several times faster than a naive branchy per-byte loop on large inputs,
//! and much faster than `str::split_whitespace().count()` (which allocates
//! UTF-8-boundary-aware iterators).
//! - Not as fast as hand-written SSE2/AVX2 for huge buffers; SWAR moves 8
//! bytes per instruction-stream step vs 16-32 for vector ISAs.
//! - For token *estimation* granularity (the result feeds a heuristic blend,
//! not an exact invoice) this trade is usually invisible end-to-end.
//!
//! # Estimate semantics
//!
//! [`SimdWhitespaceTokenizer`] splits on any byte ≤ `0x20` (space, tab,
//! newline, CR, and other C0 controls), then blends the word count with a
//! ~4 chars/token heuristic and a punctuation overhead term — see
//! [`estimate_from_whitespace_splits`]. It approximates BPE behavior for
//! budgeting and rate-limit decisions; it is **not** a tokenizer table.
//!
//! # Feature flags
//!
//! - `tiktoken` (default **off**) — adds [`TiktokenCounter`] for exact
//! `cl100k_base` and `o200k_base` counts ([`TiktokenCounter::new`] and
//! [`TiktokenCounter::o200k`]) and makes [`TokenEstimator::new`] prefer
//! tiktoken (`cl100k_base`).
//!
//! # Guarantees
//!
//! - `#![forbid(unsafe_code)]` — including the SIMD-style scan.
//! - `#![deny(missing_docs)]` — the entire public API is documented.
//! - Counting is deterministic, allocation-free, and never panics on any
//! input (verified by property tests over arbitrary byte strings,
//! including non-UTF-8 and truncated UTF-8 sequences).
pub use ;
pub use ;
/// Abstraction over token counting backends.
/// Estimate a BPE-approximate token count from whitespace split statistics.
///
/// Deliberately kept simple: split on whitespace, add a small overhead for
/// punctuation that would normally become separate tokens in BPE, and apply
/// a ~4 chars/token heuristic for whitespace-dense text. The blend trusts
/// word count when there are clear word boundaries and falls back to the
/// character-based estimate for dense text.
/// Enum that dispatches to the best available token counter.
///
/// With the `tiktoken` feature enabled (on non-WASM targets) this wraps a
/// [`TiktokenCounter`]; otherwise it uses [`SimdWhitespaceTokenizer`].
// `SimdWhitespaceTokenizer` is a zero-sized type, so the tiktoken variant is
// inherently ~200+ bytes larger (CoreBPE's tables state). Boxing would
// allocate on every construction for no real benefit; the size is bounded and
// known.