Enum rustlr::lexer_interface::RawToken
source · [−]pub enum RawToken<'t> {
}
Expand description
structure produced by StrTokenizer. TerminalTokens must be created from RawTokens (in the Tokenizer::nextsym function) once the grammar’s terminal symbols and abstract syntax type are known.
Variants
Num(i64)
an unsigned integer, though for convenience it is interpreted as a signed number. Negative numbers must be recognized by higher-level parser. Both decimal and hexadecimal numbers prefixed by 0x are recognized.
Float(f64)
floating point number
BigNumber(&'t str)
Number too large for i64 or f64 - hex numbers will include 0x prefix
Char(char)
single character inside single quotes.
Strlit(&'t str)
String literal, allows for nested quotes. String literals always contain the enclosing double quotes
Alphanum(&'t str)
Alphanumeric sequence, staring with an alphabetical character or ‘_’, and followed by arbitrary numbers of alphabetical, numeric or _.
Symbol(&'t str)
non-alphanumeric characters, either identified as triples, doubles, singles, or unrecognized sequences (refering to length of symbol)
Byte(u8)
a single byte (intended for binary data): this variant is currently not recognized by StrTokenizer
Bytes(&'t [u8])
slice of bytes (intended for binary data): this variant is currently not recognized by StrTokenizer
Newline
newline, returned optionally
Whitespace(usize)
number of consecutive whitespaces, returned optionally
Verbatim(&'t str)
usually used to represent comments, if returned optionally
Custom(&'static str, &'t str)
Custom token type, allows for user extension. The first string should
define the type of the token while the second should carry raw text.
This token type is intended to be enabled with lexattribute add_custom
directives,
which correspond to the function StrTokenizer::add_custom
LexError
tokenizer error
Trait Implementations
Auto Trait Implementations
impl<'t> RefUnwindSafe for RawToken<'t>
impl<'t> Send for RawToken<'t>
impl<'t> Sync for RawToken<'t>
impl<'t> Unpin for RawToken<'t>
impl<'t> UnwindSafe for RawToken<'t>
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more