pub enum TokenType {
Show 14 variants Alphanum, Synonym, Word, Hangul, Num, Email, Apostrophe, Double, Katakana, Acronym, Gram, Fingerprint, Shingle, Other(String),
}
Expand description

Type of token

Variants§

§

Alphanum

Alphanumeric token

§

Synonym

Synonym token

§

Word

Word token

§

Hangul

Hangul (Korean alphabet) token

§

Num

Numeric token

§

Email

Email token

§

Apostrophe

Words with apostrophe token

§

Double

CJK (Chinese, Japanese, and Korean) tokens

§

Katakana

Normalized CJK (Chinese, Japanese, and Korean) tokens. Normalizes width differences in CJK (Chinese, Japanese, and Korean) characters as follows: Folds full-width ASCII character variants into the equivalent basic Latin characters Folds half-width Katakana character variants into the equivalent Kana characters

§

Acronym

Acronym token

§

Gram

Gram token

§

Fingerprint

Fingerprint token

§

Shingle

Shingle token

§

Other(String)

Other token

Trait Implementations§

Returns a copy of the value. Read more
Performs copy-assignment from source. Read more
Formats the value using the given formatter. Read more
Returns the “default value” for a type. Read more
Deserialize this value from the given Serde deserializer. Read more
This method tests for self and other values to be equal, and is used by ==. Read more
This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason. Read more
Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations§

Blanket Implementations§

Gets the TypeId of self. Read more
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The resulting type after obtaining ownership.
Creates owned data from borrowed data, usually by cloning. Read more
Uses borrowed data to replace owned data, usually by cloning. Read more
The type returned in the event of a conversion error.
Performs the conversion.
The type returned in the event of a conversion error.
Performs the conversion.