Struct vtext::tokenize::UnicodeWordTokenizer

pub struct UnicodeWordTokenizer {
    pub params: UnicodeWordTokenizerParams,
}

Unicode Segmentation tokenizer

This implementation is a thin wrapper around the unicode-segmentation crate

References

• Unicode® Standard Annex #29

Fields

params: UnicodeWordTokenizerParams

Trait Implementations

impl Clone for UnicodeWordTokenizer

fn clone(&self) -> UnicodeWordTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnicodeWordTokenizer

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Default for UnicodeWordTokenizer

fn default() -> UnicodeWordTokenizer

Create a new instance

impl Tokenizer for UnicodeWordTokenizer

fn tokenize<'a>(&self, text: &'a str) -> Box<dyn Iterator<Item = &'a str> + 'a>

Tokenize a string