Struct vtext::tokenize_sentence::PunctuationTokenizer

pub struct PunctuationTokenizer {
    pub params: PunctuationTokenizerParams,
}

Punctuation sentence tokenizer

This simple tokenizer uses punctuation (default ".", "?", "!") to determine sentence boundaries. Trailing whitespace is also captured in the preceding sentence.

Arguments (PunctuationTokenizerParams)

• punctuation - Punctuation tokens used to determine boundaries. Only the first "character" using the chars method is used.

Fields

params: PunctuationTokenizerParams

Trait Implementations

impl Clone for PunctuationTokenizer

fn clone(&self) -> PunctuationTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PunctuationTokenizer

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Default for PunctuationTokenizer

fn default() -> PunctuationTokenizer

Create a new instance

impl Tokenizer for PunctuationTokenizer

fn tokenize<'a>(
    &'a self,
    text: &'a str
) -> Box<dyn Iterator<Item = &'a str> + 'a>

Tokenize a string