Enum TokenType

pub enum TokenType {
    Alphanum,
    Synonym,
    Word,
    Hangul,
    Num,
    Email,
    Apostrophe,
    Double,
    Katakana,
    Acronym,
    Gram,
    Fingerprint,
    Shingle,
    Other(String),
}

Type of token

Variants

Alphanum

Alphanumeric token

Synonym

Synonym token

Word

Word token

Hangul

Hangul (Korean alphabet) token

Num

Numeric token

Email

Email token

Apostrophe

Words with apostrophe token

Double

CJK (Chinese, Japanese, and Korean) tokens

Katakana

Normalized CJK (Chinese, Japanese, and Korean) tokens. Normalizes width differences in CJK (Chinese, Japanese, and Korean) characters as follows: Folds full-width ASCII character variants into the equivalent basic Latin characters Folds half-width Katakana character variants into the equivalent Kana characters

Acronym

Acronym token

Gram

Gram token

Fingerprint

Fingerprint token

Shingle

Shingle token

Other(String)

Other token

Trait Implementations

impl Clone for TokenType

fn clone(&self) -> TokenType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenType

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TokenType

fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for TokenType

fn eq(&self, other: &TokenType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenType

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for TokenType

impl StructuralPartialEq for TokenType