Struct TokenMeta 

pub struct TokenMeta {
    pub token: Token,
    pub category: TokenCategory,
    pub text: String,
    pub preceded_by_space: bool,
    pub preceded_by_newline: bool,
}

Rich metadata attached to a token.

Fields

token: Token

The underlying token.

category: TokenCategory

High-level category.

text: String

Raw source text for this token.

preceded_by_space: bool

true if any whitespace preceded this token on the same line.

preceded_by_newline: bool

true if a newline preceded this token.

Implementations

impl TokenMeta

pub fn new( token: Token, category: TokenCategory, text: impl Into<String>, preceded_by_space: bool, preceded_by_newline: bool, ) -> Self

Construct a new TokenMeta.

pub fn from_token(token: Token, source: &str) -> Self

Convenience: construct from a Token and its source slice.

pub fn span(&self) -> &Span

Return the span of this token.

pub fn kind(&self) -> &TokenKind

Return the TokenKind.

pub fn is_ident(&self) -> bool

true if this token is an identifier.

pub fn is_keyword(&self) -> bool

true if this token is a keyword.

pub fn is_literal(&self) -> bool

true if this token is a literal.

impl TokenMeta

pub fn set_preceded_by_newline(&mut self, v: bool)

Annotate with preceding newline based on source position.

pub fn set_preceded_by_space(&mut self, v: bool)

Annotate with preceding space.

pub fn len(&self) -> usize

Length of the token’s source span.

pub fn is_empty(&self) -> bool

Whether this token’s source span is empty.

pub fn is_numeric(&self) -> bool

Whether this token is a numeric literal.

pub fn is_string(&self) -> bool

Whether this token is a string literal.

pub fn is_operator(&self) -> bool

Whether this token is an operator.

Trait Implementations

impl Clone for TokenMeta

fn clone(&self) -> TokenMeta

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenMeta

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenMeta

fn eq(&self, other: &TokenMeta) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenMeta