Module code 

Round 1 (docs/round-1-contract.md §3): CodeBlock’s BUILT-IN tokenizer, a line-by-line port of src/code.ts over catalog/code.json (embedded). One forward scan, first matching rule wins; the tokens split into LINES and adjacent tokens of one kind merge. fixtures/code-tokens.json locks it. The scan walks chars where the TS walks UTF-16 units; the merge step makes the two agree for any text (an astral char is one plain token either way).

Structs

AnsiRun

EXP-1324 ansi: one run of the escape-free text with the SGR colour (a codeToken kind) it was printed in, or none.

CodeToken

LanguageSpec

Functions

ansi_runs

The source with every ANSI CSI escape removed, split into runs by SGR colour (catalog/code.json $comment, ansi); adjacent runs of one colour merge, an incomplete escape stays text. Port of ansiRuns.

code_language_names

The languages, in codeLanguage enum order.

code_text

The text a CodeBlock shows and copies: CRLF → LF and, for ansi, every escape removed. Port of codeText.

language

tokenize_code

The source tokenized per LINE: each line’s tokens concatenate to exactly its text (no \n), adjacent tokens of one kind merged; an empty line is an empty list. An unknown language tokenizes as plain.