Struct lrpar::lex::Lexeme

pub struct Lexeme<StorageT> { /* fields omitted */ }

A Lexeme represents a segment of the user's input that conforms to a known type. All lexemes have a starting position in the user's input: lexemes that result from error recovery, however, do not have a length (or, therefore, an end). This allows us to differentiate between lexemes that are always of zero length (which are required in some grammars) from lexemes that result from error recovery (where an error recovery algorithm can know the type that a lexeme should have been, but can't know what its contents should have been).

Methods

impl<StorageT: Copy> Lexeme<StorageT>

pub fn new(tok_id: StorageT, start: usize, len: Option<usize>) -> Self

Create a new token with ID tok_id and a starting position in the input start. If the token is the result of user input, then Some(n) should be passed to len; if the token is the result of error recovery, then None should be passed to len.

pub fn tok_id(&self) -> StorageT

The token ID.

pub fn start(&self) -> usize

Byte offset of the start of the lexeme

pub fn end(&self) -> Option<usize>

Byte offset of the end of the lexeme, or None if this lexeme is the result of error recovery.

pub fn len(&self) -> Option<usize>

Length in bytes of the lexeme, or None if this lexeme is the result of error recovery.

pub fn is_empty(&self) -> bool

Returns true if this lexeme is the result of error recovery, or false otherwise.

Trait Implementations

impl<StorageT: Copy> Copy for Lexeme<StorageT>

impl<StorageT: PartialEq> PartialEq<Lexeme<StorageT>> for Lexeme<StorageT>

fn eq(&self, other: &Lexeme<StorageT>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Lexeme<StorageT>) -> bool

This method tests for !=.

impl<StorageT: Clone> Clone for Lexeme<StorageT>

fn clone(&self) -> Lexeme<StorageT>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<StorageT: Eq> Eq for Lexeme<StorageT>

impl<StorageT: Debug> Debug for Lexeme<StorageT>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<StorageT: Hash> Hash for Lexeme<StorageT>

fn hash<__HStorageT: Hasher>(&self, state: &mut __HStorageT)

Feeds this value into the given [Hasher].

fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given [Hasher].