Enum wana_kana::tokenize::TokenType

pub enum TokenType {
    En,
    Ja,
    EnNum,
    JaNum,
    EnPunc,
    JaPunc,
    Kanji,
    Hiragana,
    Katakana,
    Space,
    Other,
}

Available on crate feature tokenize only.

The tokenizer assigns each token a TokenType.

In compact tokenization mode only En, Ja and Other exist.

Variants

En

Egnlish token.

Ja

Japanese token

EnNum

English number

JaNum

Japanese number

EnPunc

English punctuation

JaPunc

Japanese punctuation

Kanji

Kanji

Hiragana

Hiragana

Katakana

Katakana

Space

Space

Other

Other

Trait Implementations

impl Clone for TokenType

fn clone(&self) -> TokenType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq<TokenType> for TokenType

fn eq(&self, other: &TokenType) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenType

impl Eq for TokenType

impl StructuralEq for TokenType

impl StructuralPartialEq for TokenType