Struct SimdTokenizer 

pub struct SimdTokenizer { /* private fields */ }

SIMD-optimized tokenization utilities for improved performance

Implementations

impl SimdTokenizer

pub fn new() -> Self

Create a new SIMD tokenizer

pub fn classify_ascii_chars(&self, text: &[u8]) -> Vec<u8>

Fast ASCII character classification using SIMD

pub fn find_whitespace_boundaries(&self, text: &[u8]) -> Vec<usize>

Fast whitespace detection using SIMD

pub fn validate_utf8_fast(&self, bytes: &[u8]) -> Result<()>

Fast byte-to-UTF8 validation using SIMD

pub fn to_lowercase_ascii(&self, text: &[u8]) -> Vec<u8>

Fast case conversion using SIMD

pub fn preprocess_text(&self, text: &str) -> Result<Vec<String>>

High-performance text preprocessing pipeline

Trait Implementations

impl Default for SimdTokenizer

fn default() -> Self

Returns the “default value” for a type.