Struct Token 

pub struct Token {
    pub name: String,
    pub value: String,
    pub position: usize,
}

Represents a lexical token with a name/type and its raw value from the source code.

Used to represent the output of the Tokenizer struct.

Examples

let num = Token { name: "int".into(), value: "23".into(), position: 3 };
let kw = Token::from(("keyword", "if", 0));
assert_eq!(num, Token::from(("int", "23", 3)));

Fields

name: String

The type or category of the token (e.g., “int”, “identifier”, “operator”).

value: String

The actual text/value from the source code that this token represents.

position: usize

The position of the token in the source code.

• For [Tokenizer::tokenize], this is the byte offset from the start of the entire source.
• For [Tokenizer::tokenize_lines], this is the byte offset from the start of each line.

For instance, tokenizing "aa\naa" for the token a will yield positions of [0, 1, 3, 4] and [[0, 1], [0, 1]], respectively.

Trait Implementations

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Into<String>> From<(T, T, usize)> for Token

fn from(value: (T, T, usize)) -> Self

Converts to this type from the input type.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Token

impl StructuralPartialEq for Token