Struct RegexTokenizer 

pub struct RegexTokenizer { /* private fields */ }

Regular expression based tokenizer

Implementations

impl RegexTokenizer

pub fn new(pattern: &str, gaps: bool) -> Result<Self>

Create a new regex tokenizer

Arguments

• pattern - The regex pattern to use
• gaps - If true, the pattern matches token separators. If false, it matches tokens.

Trait Implementations

impl Clone for RegexTokenizer

fn clone(&self) -> RegexTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RegexTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Tokenizer for RegexTokenizer

fn tokenize(&self, text: &str) -> Result<Vec<String>>

Tokenize the input text into tokens

fn clone_box(&self) -> Box<dyn Tokenizer + Send + Sync>

Clone the tokenizer (for use in parallel processing)

fn tokenize_batch(&self, texts: &[&str]) -> Result<Vec<Vec<String>>>

Tokenize batch of text