Struct Token 

pub struct Token<'a> {
    pub kind: TokenKind,
    pub text: &'a str,
    pub span: Span,
    /* private fields */
}

A lexed token: its kind, source text, and byte-offset span.

Fields

kind: TokenKind

The token kind.

text: &'a str

The raw source text of the token.

span: Span

The byte-offset span of the token in the source input.

Implementations

impl<'a> Token<'a>

pub fn new(kind: TokenKind, text: &'a str, span: Span) -> Self

Creates a token from a kind, source text, and span.

pub fn eof(position: usize) -> Self

Creates the end-of-input token at the given byte offset.

pub fn is_keyword_or_identifier(&self) -> bool

Returns true when this token can stand in for a keyword or identifier.

FLOAT / INTEGER field-type keywords share kinds with numeric literals; they count as identifiers only when the source text starts with a letter.

Trait Implementations

impl<'a> Clone for Token<'a>

fn clone(&self) -> Token<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Copy for Token<'a>

impl<'a> Debug for Token<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Display for Token<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Eq for Token<'a>

impl<'a> PartialEq for Token<'a>

fn eq(&self, other: &Token<'a>) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl<'a> StructuralPartialEq for Token<'a>