pub enum TokenType {
Show 14 variants Alphanum, Synonym, Word, Hangul, Num, Email, Apostrophe, Double, Katakana, Acronym, Gram, Fingerprint, Shingle, Other(String),
}
Expand description

Type of token

Variants

Alphanum

Alphanumeric token

Synonym

Synonym token

Word

Word token

Hangul

Hangul (Korean alphabet) token

Num

Numeric token

Email

Email token

Apostrophe

Words with apostrophe token

Double

CJK (Chinese, Japanese, and Korean) tokens

Katakana

Normalized CJK (Chinese, Japanese, and Korean) tokens. Normalizes width differences in CJK (Chinese, Japanese, and Korean) characters as follows: Folds full-width ASCII character variants into the equivalent basic Latin characters Folds half-width Katakana character variants into the equivalent Kana characters

Acronym

Acronym token

Gram

Gram token

Fingerprint

Fingerprint token

Shingle

Shingle token

Other(String)

Other token

Trait Implementations

Returns a copy of the value. Read more
Performs copy-assignment from source. Read more
Formats the value using the given formatter. Read more
Returns the “default value” for a type. Read more
Deserialize this value from the given Serde deserializer. Read more
This method tests for self and other values to be equal, and is used by ==. Read more
This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason. Read more
Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more
Immutably borrows from an owned value. Read more
Mutably borrows from an owned value. Read more

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The resulting type after obtaining ownership.
Creates owned data from borrowed data, usually by cloning. Read more
Uses borrowed data to replace owned data, usually by cloning. Read more
The type returned in the event of a conversion error.
Performs the conversion.
The type returned in the event of a conversion error.
Performs the conversion.