Struct IcuTokenizer 

pub struct IcuTokenizer { /* private fields */ }

Implementations

impl IcuTokenizer

pub fn analyzer(self) -> TextAnalyzer

pub fn analyzer_builder(self) -> TextAnalyzerBuilder

Trait Implementations

impl Clone for IcuTokenizer

fn clone(&self) -> IcuTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for IcuTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for IcuTokenizer

type TokenStream<'a> = IcuTokenStream

Stream type emitted by the tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.