Struct Token 

pub struct Token<'t> {
    pub token_type: TerminalIndex,
    pub location: Location,
    pub token_number: TokenNumber,
    /* private fields */
}

The Token<’t> type represents a scanned token. It has a reference to the scanned text in the text member.

The lifetime parameter 't refers to the lifetime of the scanned text.

Fields

token_type: TerminalIndex

The index of the terminal in the augmented terminal list

location: Location

Position information

token_number: TokenNumber

Unique token number that allows ordering of tokens from different contexts, e.g. comment tokens can be intermingled with normal tokens.

Implementations

impl<'t> Token<'t>

pub fn eoi(token_number: TokenNumber) -> Self

Creates an End-Of-Input token

pub fn with<T>( text: T, token_type: TerminalIndex, location: Location, token_number: TokenNumber, ) -> Self

where T: Into<Cow<'t, str>>,

Creates a token with given values.

pub fn with_location(self, location: Location) -> Self

Change the location of the token after it’s creation

pub fn with_type(self, token_type: TerminalIndex) -> Self

Change the location of the token after it’s creation

pub fn is_skip_token(&self) -> bool

Indicates whether the token is normally skipped by the TokenStream. The behavior is independent from the language.

pub fn is_comment_token(&self) -> bool

Indicates whether the token is a comment token.

pub fn text(&self) -> &str

Accesses the token’s scanned text

pub fn to_owned(&self) -> Token<'static>

Creates an owned instance of the token from a shared reference

pub fn into_owned(self) -> Token<'static>

Creates an owned instance of the token and consumes self

Trait Implementations

impl AsRef<str> for Token<'_>

fn as_ref(&self) -> &str

Converts this type into a shared reference of the (usually inferred) input type.

impl<'t> Clone for Token<'t>

fn clone(&self) -> Token<'t>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'t> Debug for Token<'t>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'t> Default for Token<'t>

fn default() -> Token<'t>

Returns the “default value” for a type.

impl Display for Token<'_>

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl FormatToken for Token<'_>

fn format(&self, terminal_names: &'static [&'static str]) -> String

Generates a formatted position which an editor can follow via mouse click.

impl From<&Token<'_>> for Location

fn from(token: &Token<'_>) -> Self

Converts to this type from the input type.

impl From<&Token<'_>> for PTToken

fn from(token: &Token<'_>) -> Self

Converts to this type from the input type.

impl From<&Token<'_>> for Range<usize>

fn from(token: &Token<'_>) -> Self

Converts to this type from the input type.

impl From<&Token<'_>> for Span

fn from(token: &Token<'_>) -> Self

Converts to this type from the input type.

impl<'t> Ord for Token<'t>

fn cmp(&self, other: &Token<'t>) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl<'t> PartialEq for Token<'t>

fn eq(&self, other: &Token<'t>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'t> PartialOrd for Token<'t>

fn partial_cmp(&self, other: &Token<'t>) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl ToSpan for Token<'_>

fn span(&self) -> Span

Calculates the span of the implementing item

impl<'t> Eq for Token<'t>

impl<'t> StructuralPartialEq for Token<'t>