Enum regen::core::token::TokenRule

pub enum TokenRule {
    IgnoreLiteral(String),
    IgnoreRegExp(String),
    Literal(String, String),
    RegExp(String, String),
}

Definition for a token rule used by the tokenizer

On each tokenization step, the tokenizer will try to match the input against the token rules. The longest token matched will be the next token produced.

If a token cannot be matched, one character is skipped and added to the unmatchable token list.

Variants

IgnoreLiteral(String)

Ignore rule matching a literal

IgnoreRegExp(String)

Ignore rule matching a regular expression

Literal(String, String)

Token rule matching a literal

RegExp(String, String)

Token rule matching a regular expression

Trait Implementations

impl Clone for TokenRule

fn clone(&self) -> TokenRule

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenRule

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.