Struct WhitespaceTokenizer 

pub struct WhitespaceTokenizer;

Unicode-scalar whitespace tokenizer.

Splits on any sequence of Unicode whitespace via str::split_whitespace and discards empty spans. Zero-size; free to copy.

Token counts produced by this type are a structural estimate — see the module documentation for how they relate to subword model tokenizers.

Performance

Both tokenize and token_count are O(n) single-pass scans with no internal allocation beyond the returned Vec. An LRU cache would add memory pressure and synchronisation overhead that outweighs any benefit at these text sizes.

Trait Implementations

impl Clone for WhitespaceTokenizer

fn clone(&self) -> WhitespaceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WhitespaceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for WhitespaceTokenizer

fn default() -> WhitespaceTokenizer

Returns the “default value” for a type.

impl Tokenizer for WhitespaceTokenizer

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

Split text into tokens, returning slices into the original string.

fn token_count(&self, text: &str) -> usize

Count the number of tokens in text.

impl Copy for WhitespaceTokenizer