Struct lrpar::lex::Lexeme

pub struct Lexeme<StorageT> { /* private fields */ }

A Lexeme represents a segment of the user’s input that conforms to a known type. All lexemes have a starting position in the user’s input: lexemes that result from error recovery, however, do not have a length (or, therefore, an end). This allows us to differentiate between lexemes that are always of zero length (which are required in some grammars) from lexemes that result from error recovery (where an error recovery algorithm can know the type that a lexeme should have been, but can’t know what its contents should have been).

Implementations

impl<StorageT: Copy> Lexeme<StorageT>

pub fn new(tok_id: StorageT, start: usize, len: Option<usize>) -> Self

Create a new token with ID tok_id and a starting position in the input start. If the token is the result of user input, then Some(n) should be passed to len; if the token is the result of error recovery, then None should be passed to len.

pub fn tok_id(&self) -> StorageT

The token ID.

pub fn start(&self) -> usize

Byte offset of the start of the lexeme

pub fn end(&self) -> Option<usize>

Byte offset of the end of the lexeme, or None if this lexeme is the result of error recovery.

pub fn len(&self) -> Option<usize>

Length in bytes of the lexeme, or None if this lexeme is the result of error recovery.

Trait Implementations

impl<StorageT: Clone> Clone for Lexeme<StorageT>

fn clone(&self) -> Lexeme<StorageT>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<StorageT: Debug> Debug for Lexeme<StorageT>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<StorageT: PartialEq> PartialEq<Lexeme<StorageT>> for Lexeme<StorageT>

fn eq(&self, other: &Lexeme<StorageT>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<StorageT: Copy> Copy for Lexeme<StorageT>

impl<StorageT> StructuralPartialEq for Lexeme<StorageT>