Struct WhitespaceTokenizer 

pub struct WhitespaceTokenizer { /* private fields */ }

Simple whitespace-based tokenizer (English, Spanish, French, etc.).

Splits on whitespace and punctuation. Works for languages with clear word boundaries.

Implementations

impl WhitespaceTokenizer

pub fn new() -> Self

Create a new whitespace tokenizer.

pub fn with_punctuation(self, include: bool) -> Self

Create with punctuation handling.

Trait Implementations

impl Default for WhitespaceTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for WhitespaceTokenizer

fn tokenize(&self, text: &str, _language: Option<&Language>) -> Vec<Token>

Tokenize text into a sequence of tokens.

fn name(&self) -> &'static str

Get the tokenizer name/identifier.

fn is_stopword(&self, token: &Token, language: Option<&Language>) -> bool

Check if a token is a stopword (common function words to ignore).