Struct Token 

pub struct Token {
    pub kind: i32,
    pub value: TokenValue,
}

A single lexed token with its semantic payload.

kind is an i32 whose value is either an ASCII byte code (for single-byte tokens like +, -, [) or one of the TK_* constants (for reserved words, multi-char symbols, and literals).

TODO(port): Phase B — replace kind: i32 with a proper TokenKind enum covering both single-byte and named tokens (e.g. TokenKind::Char(u8) + named variants).

Fields

kind: i32

value: TokenValue

Implementations

impl Token

pub fn new(kind: i32) -> Self

Construct a token with no semantic value.

pub fn eos() -> Self

The end-of-stream sentinel token.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.