Struct Token 

pub struct Token {
    pub name: TokenText,
    pub tin: Tin,
    pub val: Value,
    pub src: TokenText,
    pub len: usize,
    pub site: Site,
    pub err: TokenCode,
    pub why: TokenCode,
    pub use_data: Option<Box<HashMap<String, Value>>>,
    pub ignored: Option<Box<Token>>,
    pub val_fn: Option<TokenValFunc>,
}

A single lexical token produced by the lexer.

Fields

name: TokenText

tin: Tin

val: Value

src: TokenText

len: usize

UTF-8 byte length of src, paired with the byte offset site.si.

site: Site

Where in the source this token starts.

err: TokenCode

why: TokenCode

use_data: Option<Box<HashMap<String, Value>>>

Plugin diagnostic details, boxed and absent until something writes one. Prefer use_data() and use_data_mut() to reaching through the Option; the shape is public only so that a Token can still be built with a struct literal. It is boxed because a HashMap is 48 bytes inline, a token is cloned about six times per input construct, and almost no token ever carries a detail. Measured by padding Token, 96 bytes of it is worth 3% to 12% depending on the grammar.

ignored: Option<Box<Token>>

Optional ignored trivia associated with this token. Negotiated re-lexing carries it to the replacement token.

val_fn: Option<TokenValFunc>

Optional semantic value callback. val remains the eagerly produced fallback and the value shown by raw token inspection.

Implementations

impl Token

pub fn new( name: impl AsRef<str>, tin: Tin, val: Value, src: impl Into<TokenText>, pnt: Point, ) -> Self

pub fn use_data(&self) -> &HashMap<String, Value>

Plugin diagnostic details. Empty unless something wrote one.

pub fn use_data_mut(&mut self) -> &mut HashMap<String, Value>

Mutable access, allocating the bag on first write.

pub fn no_token() -> Self

pub fn is_no_token(&self) -> bool

pub fn bad(&mut self, err: &str) -> &mut Self

pub fn bad_with_details( &mut self, err: &str, details: impl IntoIterator<Item = (String, Value)>, ) -> &mut Self

Mark this token bad and deep-merge plugin diagnostic details into its existing use_data bag. This is the typed Rust form of token.bad(code, details).

pub fn with_val_func(self, callback: TokenValFunc) -> Self

Attach an already shared lazy value callback.

pub fn with_lazy_value( self, callback: impl Fn(&mut Rule, &mut Context) -> Value + Send + Sync + 'static, ) -> Self

Attach a lazy value callback without constructing a wrapper first.

pub fn resolve_val(&self, rule: &mut Rule, context: &mut Context) -> Value

Resolve the semantic token value against the live parse state.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Token

fn default() -> Self

Returns the “default value” for a type.

impl Display for Token

fn fmt(&self, formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token