Enum TokenKind 

pub enum TokenKind {
    Thai,
    Latin,
    Number,
    Punctuation,
    Emoji,
    Whitespace,
    Unknown,
}

Classification of a Token’s script / category.

Variants

Thai

Thai script syllable or word.

Latin

Latin / ASCII alphabetic text.

Number

Numeric digits (ASCII or Thai ๐–๙).

Punctuation

Punctuation or symbol.

Emoji

Emoji character sequence.

Whitespace

Whitespace (space, tab, newline).

Unknown

Anything that does not fit the above categories.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for TokenKind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind