Enum rustlr::lexer_interface::RawToken
source · pub enum RawToken<'t> {
}
Expand description
structure produced by StrTokenizer. TerminalTokens must be created from RawTokens (in the Tokenizer::nextsym function) once the grammar’s terminal symbols and abstract syntax type are known.
Variants§
Num(i64)
an unsigned integer, though for convenience it is interpreted as a signed number. Negative numbers must be recognized by higher-level parser. Both decimal and hexadecimal numbers prefixed by 0x are recognized.
Float(f64)
floating point number
BigNumber(&'t str)
Numbers too large for i64 or f64 are represented verbatim
Char(char)
single character inside single quotes.
Strlit(&'t str)
String literal, allows for nested quotes. String literals always contain the enclosing double quotes
Alphanum(&'t str)
Alphanumeric sequence, staring with an alphabetical character or ‘_’, and followed by arbitrary numbers of alphabetical, numeric or _.
Symbol(&'t str)
non-alphanumeric characters, either identified as triples, doubles, singles, or unrecognized sequences (refering to length of symbol)
Byte(u8)
a single byte (intended for binary data): this variant is currently not recognized by StrTokenizer
Bytes(&'t [u8])
slice of bytes (intended for binary data): this variant is currently not recognized by StrTokenizer
Newline
newline, returned optionally
Whitespace(usize)
number of consecutive whitespaces, returned optionally
Verbatim(&'t str)
usually used to represent comments, if returned optionally
Custom(&'static str, &'t str)
Custom token type, allows for user extension. The first string should
define the type of the token while the second should carry raw text.
This token type is intended to be enabled with lexattribute add_custom
directives,
which correspond to the function StrTokenizer::add_custom
Skipto(&'t str)
special token triggered by StrTokenizer::skip_to, returns skipped text
Skipmatched(&'t str)
special token triggered by StrTokenizer::skip_match, returns matched text
LexError
tokenizer error