Tokenizer

scirs2_text::tokenize

Trait Tokenizer 

Source 
pub trait Tokenizer {
    // Required methods
    fn tokenize(&self, text: &str) -> Result<Vec<String>>;
    fn clone_box(&self) -> Box<dyn Tokenizer + Send + Sync>;

    // Provided method
    fn tokenize_batch(&self, texts: &[&str]) -> Result<Vec<Vec<String>>> { ... }
}
Expand description

Trait for tokenizing text

Required Methods§

Source

fn tokenize(&self, text: &str) -> Result<Vec<String>>

Tokenize the input text into tokens

Source

fn clone_box(&self) -> Box<dyn Tokenizer + Send + Sync>

Clone the tokenizer (for use in parallel processing)

Provided Methods§

Source

fn tokenize_batch(&self, texts: &[&str]) -> Result<Vec<Vec<String>>>

Tokenize batch of text

Implementors§