Enum rustlr::lexer_interface::RawToken
source · [−]pub enum RawToken<'t> {
}
Expand description
structure produced by StrTokenizer. TerminalTokens must be created from RawTokens (in the Tokenizer::nextsym function) once the grammar’s terminal symbols and abstract syntax type are known.
Variants
Num(i64)
an unsigned integer, though for convenience it is interpreted as a signed number. Negative numbers must be recognized by higher-level parser. Both decimal and hexadecimal numbers prefixed by 0x are recognized.
Float(f64)
floating point number
BigNumber(&'t str)
Number too large for i64 or f64 - hex numbers will include 0x prefix
Char(char)
single character inside single quotes.
Strlit(&'t str)
String literal, allows for nested quotes
Alphanum(&'t str)
Alphanumeric sequence, staring with an alphabetical character or ‘_’, and followed by arbitrary numbers of alphabetical, numeric or _.
Symbol(&'t str)
non-alphanumeric characters, either identified as triples, doubles, singles, or unrecognized sequences (refering to length of symbol)
Byte(u8)
a single byte (intended for binary data): this variant is currently not recognized by StrTokenizer
Bytes(&'t [u8])
slice of bytes (intended for binary data): this variant is currently not recognized by StrTokenizer
Newline
newline, returned optionally
Whitespace(usize)
number of consecutive whitespaces, returned optionally
Verbatim(&'t str)
usually used to represent comments, if returned optionally
LexError
tokenizer error
Trait Implementations
Auto Trait Implementations
impl<'t> RefUnwindSafe for RawToken<'t>
impl<'t> Send for RawToken<'t>
impl<'t> Sync for RawToken<'t>
impl<'t> Unpin for RawToken<'t>
impl<'t> UnwindSafe for RawToken<'t>
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more