Struct TokenDef 

pub struct TokenDef {
    pub name: String,
    pub is_extract: bool,
}

Definition for a token type used by the tokenizer

Each token definiton has a name and a flag to indicate if it should be extracted. Extracted tokens are not used in AST generation (for example, comments), while ignored tokens are completely removed from the rest of the parsing process.

Note that this should not be confused with the Token in the SDK, which is the token data structure at runtime when parsing.

Fields

name: String

Token type name

is_extract: bool

If the token type should be extracted

Trait Implementations

impl Clone for TokenDef

fn clone(&self) -> TokenDef

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenDef

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.