Enum Tokenizer 

pub enum Tokenizer {
    Blank,
    Camel,
    Class,
    Punct,
    Segment(SegmentLanguage),
}

Variants

Blank

Camel

Class

Punct

Segment(SegmentLanguage)

Splits a span into words using a morphological dictionary.

Unlike the character-class tokenizers, which decide a boundary from the pair of characters around it, this one needs the whole span at once: the boundaries come from a lattice search over dictionary entries. It therefore runs after the character-class tokenizers have produced spans, segmenting each one further.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Tokenizer

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Tokenizer

impl Hash for Tokenizer

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Tokenizer

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Tokenizer

impl ToSql for Tokenizer

fn fmt_sql(&self, f: &mut String, sql_fmt: SqlFormat)

Format the type to a SQL string.

fn to_sql(&self) -> String

Convert the type to a SQL string.

fn to_sql_pretty(&self) -> String

Convert the type to a pretty-printed SQL string with indentation.