Struct earl_lang_syntax::tokenizer::Token

pub struct Token {
    pub id: TokenId,
    pub index: usize,
    pub length: usize,
    pub lexeme: String,
}

A Token's id is its type, but I can't use type as the name of a field in a struct.

A token keeps track of what kind of token it is, i.e. id, and at what byte it was found in the input and how many bytes it spans.

• The field id is what type of token this is.
• The field index is the index of the byte where this token begins.
• The field length is how many bytes this token spans.

Fields

id: TokenId

asdfasdff

index: usize

length: usize

asdfasdff

lexeme: String

Implementations

impl Token

pub fn new(id: TokenId, index: usize, length: usize) -> Token

Creates a new token.

pub fn new_from_code(
    id: TokenId,
    index: usize,
    length: usize,
    code: &str
) -> Token

pub fn finalize(&mut self, code: &str)

Convenience thing

pub fn get_lexeme<'a>(&self, code: &'a str) -> &'a str

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.