Tokenizer for the CJC programming language.

This crate converts raw CJC source text into a flat stream of [Token]s that is consumed by [cjc_parser]. The lexer handles all literal forms (integers, floats, strings, byte strings, raw strings, format strings, regex literals), keywords, operators, punctuation, and comments (line // and nested block /* */).

Usage

use cjc_lexer::Lexer;

let (tokens, diagnostics) = Lexer::new(src).tokenize();

The returned [DiagnosticBag] collects any lexer-level errors (unterminated strings, invalid escape sequences, unexpected characters, etc.).