rustc-ap-rustc_lexer 637.0.0

Automatically published version of the package `rustc_lexer` in the rust-lang/rust repository from commit 7e393b5b3b543d355ae16c1940cf98b6c7fcb8aa The publishing script for this crate lives at: https://github.com/alexcrichton/rustc-auto-publish

Low-level Rust lexer.

Tokens produced by this lexer are not yet ready for parsing the Rust syntax, for that see librustc_parse::lexer, which converts this basic token stream into wide tokens used by actual parser.

The purpose of this crate is to convert raw sources into a labeled sequence of well-known token types, so building an actual Rust token stream will be easier.

Main entity of this crate is [TokenKind] enum which represents common lexeme types.