Module lexer
Source - unescape
- Utilities for validating string and char literals and turning them into
values they represent.
- Cursor
- Peekable iterator over a char sequence.
- Token
- Parsed token.
It doesn’t contain information about data that has been parsed,
only the type of the token and its size.
- Base
- Base of numeric literal encoding according to its prefix.
- DocStyle
- LiteralKind
- Enum representing the literal types supported by the lexer.
- RawStrError
- TokenKind
- Enum representing common lexeme types.
- is_id_continue
- True if
c is valid as a non-first character of an identifier.
See Rust language reference for
a formal definition of valid identifier name. - is_id_start
- True if
c is valid as a first character of an identifier.
See Rust language reference for
a formal definition of valid identifier name. - is_ident
- The passed string is lexically an identifier.
- is_whitespace
- True if
c is considered a whitespace according to Rust language definition.
See Rust language reference
for definitions of these classes. - strip_shebang
rustc allows files to have a shebang, e.g. “#!/usr/bin/rustrun”,
but shebang isn’t a part of rust syntax.- tokenize
- Creates an iterator that produces tokens from the input string.
- validate_raw_str
- Validates a raw string literal. Used for getting more information about a
problem with a
RawStr/RawByteStr with a None field.