Struct Token 

pub struct Token {
    pub token_type: TokenType,
    pub text: Arc<str>,
    pub start: usize,
    pub end: usize,
}

Token with position information

Fields

token_type: TokenType

The type of token

text: Arc<str>

The actual text of the token

start: usize

Start position in the input

end: usize

End position in the input

Implementations

impl Token

pub fn new( token_type: TokenType, text: impl Into<Arc<str>>, start: usize, end: usize, ) -> Self

Create a new token

pub fn len(&self) -> usize

Get the length of the token

pub fn is_empty(&self) -> bool

Check if the token is empty

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.