Low-level Rust lexer.
Tokens produced by this lexer are not yet ready for parsing the Rust syntax,
for that see
librustc_parse::lexer, which converts this basic token stream
into wide tokens used by actual parser.
The purpose of this crate is to convert raw sources into a labeled sequence of well-known token types, so building an actual Rust token stream will be easier.
Main entity of this crate is
TokenKind enum which represents common
Utilities for validating string and char literals and turning them into values they represent.
Parsed token. It doesn't contain information about data that has been parsed, only the type of the token and its size.
Base of numeric literal encoding according to its prefix.
Enum represening common lexeme types.
Parses the first token from the provided input string.
Creates an iterator that produces tokens from the input string.