Enum rustlr::lexer_interface::RawToken

pub enum RawToken<'t> {
    Num(i64),
    Float(f64),
    BigNumber(&'t str),
    Char(char),
    Strlit(&'t str),
    Alphanum(&'t str),
    Symbol(&'t str),
    Byte(u8),
    Bytes(&'t [u8]),
    Newline,
    Whitespace(usize),
    Verbatim(&'t str),
    Custom(&'static str, &'t str),
    LexError,
}

structure produced by StrTokenizer. TerminalTokens must be created from RawTokens (in the Tokenizer::nextsym function) once the grammar’s terminal symbols and abstract syntax type are known.

Variants

Num(i64)

an unsigned integer, though for convenience it is interpreted as a signed number. Negative numbers must be recognized by higher-level parser. Both decimal and hexadecimal numbers prefixed by 0x are recognized.

Float(f64)

floating point number

BigNumber(&'t str)

Number too large for i64 or f64 - hex numbers will include 0x prefix

Char(char)

single character inside single quotes.

Strlit(&'t str)

String literal, allows for nested quotes. String literals always contain the enclosing double quotes

Alphanum(&'t str)

Alphanumeric sequence, staring with an alphabetical character or ‘_’, and followed by arbitrary numbers of alphabetical, numeric or _.

Symbol(&'t str)

non-alphanumeric characters, either identified as triples, doubles, singles, or unrecognized sequences (refering to length of symbol)

Byte(u8)

a single byte (intended for binary data): this variant is currently not recognized by StrTokenizer

Bytes(&'t [u8])

slice of bytes (intended for binary data): this variant is currently not recognized by StrTokenizer

Newline

newline, returned optionally

Whitespace(usize)

number of consecutive whitespaces, returned optionally

Verbatim(&'t str)

usually used to represent comments, if returned optionally

Custom(&'static str, &'t str)

Custom token type, allows for user extension. The first string should define the type of the token while the second should carry raw text. This token type is intended to be enabled with lexattribute add_custom directives, which correspond to the function StrTokenizer::add_custom

LexError

tokenizer error

Trait Implementations

impl<'t> Debug for RawToken<'t>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.