pub struct SimdWhitespaceTokenizer { /* private fields */ }Expand description
SIMD-style (SWAR) whitespace-split token estimator.
Splits on any byte ≤ 0x20 (covers space, tab, newline, CR, and other C0
controls) using 8-byte SWAR loads, then blends word count + character
based heuristics via estimate_from_whitespace_splits for a
BPE-approximate token count.
Implementations§
Source§impl SimdWhitespaceTokenizer
impl SimdWhitespaceTokenizer
Sourcepub fn count_splits(&self, bytes: &[u8]) -> usize
pub fn count_splits(&self, bytes: &[u8]) -> usize
Count whitespace-delimited segments (maximal runs of non-whitespace bytes).
Uses the SWAR path (8 bytes per step); the portable scalar reference
is available as scalar_count_splits. Both produce identical
results for every input.
Trait Implementations§
Source§impl Clone for SimdWhitespaceTokenizer
impl Clone for SimdWhitespaceTokenizer
Source§impl Debug for SimdWhitespaceTokenizer
impl Debug for SimdWhitespaceTokenizer
Source§impl Default for SimdWhitespaceTokenizer
impl Default for SimdWhitespaceTokenizer
Auto Trait Implementations§
impl Freeze for SimdWhitespaceTokenizer
impl RefUnwindSafe for SimdWhitespaceTokenizer
impl Send for SimdWhitespaceTokenizer
impl Sync for SimdWhitespaceTokenizer
impl Unpin for SimdWhitespaceTokenizer
impl UnsafeUnpin for SimdWhitespaceTokenizer
impl UnwindSafe for SimdWhitespaceTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more