Expand description
Tokenizer for the CJC programming language.
This crate converts raw CJC source text into a flat stream of Tokens
that is consumed by [cjc_parser]. The lexer handles all literal forms
(integers, floats, strings, byte strings, raw strings, format strings,
regex literals), keywords, operators, punctuation, and comments (line //
and nested block /* */).
§Usage
ⓘ
use cjc_lexer::Lexer;
let (tokens, diagnostics) = Lexer::new(src).tokenize();The returned DiagnosticBag collects any lexer-level errors (unterminated
strings, invalid escape sequences, unexpected characters, etc.).
Structs§
- Lexer
- Tokenizer that converts CJC source text into a sequence of
Tokens. - Token
- A single token produced by the
Lexer.