Low-level Rust lexer.
Tokens produced by this lexer are not yet ready for parsing the Rust syntax,
for that see librustc_parse::lexer, which converts this basic token stream
into wide tokens used by actual parser.
The purpose of this crate is to convert raw sources into a labeled sequence of well-known token types, so building an actual Rust token stream will be easier.
Main entity of this crate is [TokenKind] enum which represents common
lexeme types.