Struct WhitespaceTokenizer 

pub struct WhitespaceTokenizer;

Whitespace tokenizer that splits text on Unicode whitespace characters.

This is the simplest tokenizer, splitting on any Unicode whitespace (spaces, tabs, newlines, etc.). It preserves punctuation attached to words.

Examples

use aprender::text::{Tokenizer, tokenize::WhitespaceTokenizer};

let tokenizer = WhitespaceTokenizer::new();

// Basic tokenization
let tokens = tokenizer.tokenize("Hello, world!").expect("tokenize should succeed");
assert_eq!(tokens, vec!["Hello,", "world!"]);

// Handles multiple spaces
let tokens = tokenizer.tokenize("foo   bar").expect("tokenize should succeed");
assert_eq!(tokens, vec!["foo", "bar"]);

// Handles newlines and tabs
let tokens = tokenizer.tokenize("line1\nline2\ttab").expect("tokenize should succeed");
assert_eq!(tokens, vec!["line1", "line2", "tab"]);

Implementations

impl WhitespaceTokenizer

pub fn new() -> WhitespaceTokenizer

Create a new whitespace tokenizer.

Examples

use aprender::text::tokenize::WhitespaceTokenizer;

let tokenizer = WhitespaceTokenizer::new();

Trait Implementations

impl Clone for WhitespaceTokenizer

fn clone(&self) -> WhitespaceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WhitespaceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for WhitespaceTokenizer

fn default() -> WhitespaceTokenizer

Returns the “default value” for a type.

impl Tokenizer for WhitespaceTokenizer

fn tokenize(&self, text: &str) -> Result<Vec<String>, AprenderError>

Tokenize the input text into a vector of string tokens.