Struct SentenceTokenizer 

pub struct SentenceTokenizer { /* private fields */ }

Tokenizer for splitting text into sentences

Implementations

impl SentenceTokenizer

pub fn new() -> Self

Create a new sentence tokenizer

pub fn withpattern(pattern: &str) -> Result<Self>

Create a new sentence tokenizer with a custom pattern

Trait Implementations

impl Clone for SentenceTokenizer

fn clone(&self) -> SentenceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SentenceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SentenceTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for SentenceTokenizer

fn tokenize(&self, text: &str) -> Result<Vec<String>>

Tokenize the input text into tokens

fn clone_box(&self) -> Box<dyn Tokenizer + Send + Sync>

Clone the tokenizer (for use in parallel processing)

fn tokenize_batch(&self, texts: &[&str]) -> Result<Vec<Vec<String>>>

Tokenize batch of text