# Syntax highlighting
> Part of the edamame contributor deep-dives. Index and project-wide conventions: [`AGENTS.md`](../../AGENTS.md). Sibling docs live in [`docs/dev/`](.).
- **The language comes only from the fence's info string, and the lookup takes only its first token.** No auto-detection, and none planned — guessing wrong recolors a document the author never asked to have recolored. `language_token` splits on `,`, whitespace and `{` so `rust,ignore` and `js {1,3-4}` resolve; the lookup is `find_syntax_by_token`, never `find_syntax_by_extension`, because a fence info string is a language name, not a path. The ` lang ` label row still renders the **whole** info string verbatim.
- **Token ranges are char indices.** syntect reports *byte* offsets, every column map in this crate is char-indexed, so `highlight.rs` converts once at its API boundary and a byte offset never escapes it — one that does puts every token boundary after a non-ASCII character in the wrong column (`ranges_are_char_indices_not_byte_offsets`).
- **`TokenClass` has no `Default` variant, and that omission is load-bearing.** Unclassified text produces no token, so "nothing to say", "unknown language", "over cap" and "setting off" are one path downstream: `code_body_row` with no tokens, reproducing the pre-feature single-span line character for character. That equivalence lets the existing code-block snapshots stand as the regression guard (`an_unknown_language_renders_exactly_like_highlighting_off`); a `Default` class would fork the plain path in two.
- **Classification walks the scope stack innermost-outward, and `storage` is a keyword.** A string's delimiter carries `punctuation.definition.string.begin`, which no rule claims, so the walk must fall outward to the enclosing `string.quoted.double`. All of `storage` maps to `Keyword`: TextMate uses `storage.type` for the keyword that *declares* something (`fn`, `int`, `var`) while an actual type name is `entity.name.type` / `support.type`.
- **Three caps, none redundant.** `MAX_HIGHLIGHT_SOURCE_BYTES` (64 KiB) bounds the *cold* parse; `MAX_HIGHLIGHT_LINE_CHARS` (2 000) bounds the *per-keystroke* cost, which the byte cap cannot, since incremental reuse needs an unchanged prefix and a one-line block has none; `MAX_HIGHLIGHT_GRAMMARS` (24) bounds grammar *compilation*, which is not a function of block size — syntect compiles a grammar's regexes lazily on first use (~9 ms, ~18 ms for Rust), so fifty one-line fences in fifty languages pass the other two caps and still cost ~430 ms of queued worker CPU. All three bound **color, never content**: over a cap the block still renders every byte, reusing the unknown-language path, because mermaid's refuse-to-render posture would be wrong for code a user must read. Tokenizing is the one content-handling path running *synchronously on the render thread*, so it is also wrapped in `catch_unwind`.
- **The grammar cap bounds a burst, not the process lifetime.** A flat lifetime counter locked a long session out permanently, and LRU eviction is backwards: syntect keeps compiled regexes in a `OnceCell` for the process's life, so re-using a grammar is free and evicting its record refuses a free grammar to admit a paid one. `GrammarBudget` splits the two questions: `warm` (already paid for?) only grows; `budget` (affordable *now*?) is a token bucket refilling one slot per `GRAMMAR_BUDGET_REFILL` and saturating at the cap, so a long idle can't bank slots. `refill` advances `last_refill` by the intervals actually consumed, not to `now`, or a document re-rendering faster than the interval never earns a slot. A third state, `pending`, stops the render thread re-queueing the same cold grammar every reparse. The rule is `GrammarBudget::admit`, with clock and cap injected so tests never touch the process-global `GRAMMARS`.
- **A refusal is also an event: `refused_grammar_retry_due` is what makes the burst bound true in practice.** Only a re-render re-asks `admit`, which for highlighting means `warm_generation` moving — and the queued burst compiles well inside one refill interval, so a document over the cap would stay partly plain for the session unless the user typed or resized. `GrammarBudget::refused` gives `tick_syntax_warm` a second reason to reparse, **edge-triggered** (`take_retry` consumes the answer), so a standing refusal costs one reparse per refilled slot. **That reparse only reaches the highlighter because it also bumps `highlight::retry_epoch`, which rides in `RenderSettings` beside `warm_generation`**: a retry is granted precisely when no grammar warmed, so nothing else in the fingerprint changed, every block hits the cache and `admit` is never re-asked — leaving the cap a session limit again with every `GrammarBudget` unit test still passing (`render_cache::tests::both_highlight_counters_clear_the_cache`).
- **Parsing is synchronous and compiling is not, and the asymmetry is the design.** Tokenizing stays on the render thread because highlighting changes only *color*, on text already on screen: any deferral must paint something during the gap, and both options are bad — plain text drops colors out and back on every keystroke, and stale tokens are char ranges, so an insert shifts every color on the line. A debounce was rejected on the same argument plus one of its own: `"`, `` ` ``, `/*` and `#` reclassify the rest of a block in one keystroke, so a heuristic waiting for whitespace holds the stale render exactly when it is most wrong. Compilation is the exception because it happens once per language: `spawn_warm_worker` owns it, a cold grammar returns `Admission::Queue`, the block renders plain, and the worker replays *the block's own lines* — syntect compiles per **match pattern**, so replaying the real block compiles exactly the patterns the render thread is about to need. A pattern the warm parse missed compiles one regex inline; that residual is accepted. A warm parse that panics leaves the grammar `pending` forever — plain, rather than retried into the same panic every frame.
- **`warm_generation` is how a plain block ever becomes colored, and it must ride in `RenderSettings`** — warming changes no `Block` value, so `RenderCache` would serve the plain render for the life of the document. `App::tick_syntax_warm` polls it on the 60 ms loop tick (polled, not pushed: `markdown` sits far below `app` and must not learn about `AppEvent`). `App::syntax_warm_generation` is seeded from the live counter rather than 0 (a second `App` in one process must not see a spurious first-tick change), and that read is taken **before** `spawn_warm_worker` and `configure_new_editor`, not at the struct literal — the constructor's own first render queues this document's grammars, and a later read can capture a landed compile *as* the starting value, leaving the block plain until some unrelated reparse. The renderer reads the counter itself and pins it to 0 when the feature is off, so toggling can't leave a stale generation in the fingerprint.
- **Asynchronous compilation makes highlighting eventually-consistent, which tests must opt out of.** A bare `highlight_block` on a cold grammar answers `[]`, so a color assertion either races the worker or passes because an unrelated test warmed that grammar first. `warm_inline` marks a grammar usable immediately, compiling on the calling thread: every classification test goes through the `hl` helper and both `render_highlighted` helpers call `warm_fence_languages` first. Production code must never call it on the render thread.
- **Incremental reuse is two-sided, and the convergence check is what makes it correct.** The cache holds, per block, the `(ParseState, ScopeStack)` at the *start* of each line plus a tail entry for the state after the last (without the tail, appending at a block's end could not resume). Lines before the first change carry over directly; after it, the parser position is compared against the cached position at the matching line (re-based, since an edit adds or removes lines), and once the two are equal *and* the remaining text is unchanged, the rest carries over. **Both halves of the position are needed** — `ParseState` omits the scope stack, and resuming with a fresh one misclassifies everything below the edit. The invariant tests assert is "the incremental result equals a cold parse"; `reuse_is_actually_happening` guards the other direction, so a cache degrading into a full re-parse can't leave every correctness test passing.
- **`syntax_*` theme styles are patched over `code_block_text`, never used alone.** Each sets a foreground only; the code surface owns the background, and a syntax style with its own `bg` paints a stale one wherever a theme moved that surface (`syntax_styles_carry_no_background_of_their_own`). All seven derive from existing `Palette` slots, so every built-in gets a coherent set with no per-theme authoring; only `monochrome_dark` has hand-written entries, marking just `Keyword` and `Comment` since with no color to spend, attributing everything distinguishes nothing.
- **The seven slots must be *foreground* slots, and the derivation is contrast-checked.** A palette splits into text carriers (`primary`, `text_muted`, `link`, `success`, `warning`, `error`, `code`) and fills/chrome (`secondary`, `accent`, `bg` / `surface`); only the first group has had to be legible as characters. `syntax_type` was `secondary` and `syntax_function` `accent`, measuring 1.99:1 and 1.51:1 against `dark_256`'s code surface versus 6.97:1 for the plain code text they replaced — highlighting made code *less* readable, on the very theme the indexed fallback substitutes into. They are now `code` and `link`, with `error` taking `attribute`; `themes::util::legible_on` is the backstop for the rest, lifting each color toward `text` until it clears `SYNTAX_MIN_CONTRAST` (4.5:1) while keeping its hue.
- **`legible_on` is a no-op for indexed colors, so `dark_256` / `light_256` pin all seven by hand** — the same reason those files already pin the heading ramp, `code_bg`, `selection_muted` and the diff washes. Their picks are the bright- (or dark-) tint siblings of the slots the RGB derivation would use, since the mid shades are chosen to read on `bg`, not on the code surface; `light_256` diverges further (no dark orange in the cube clears the floor, so `keyword` takes the conventional deep red and `attribute` moves to teal). `syntax_contrast_clears_the_floor_for_every_builtin_theme` holds all 27 themes to account, resolving the xterm cube itself and treating an ANSI 0–15 index as a failure rather than a skip. Its floor is `min(SYNTAX_MIN_CONTRAST, plain code text)`: a theme cannot make a token more legible than its own body text, and the promise that matters is that highlighting never makes a block *less* readable than plain.
- **The mermaid source reveal is out of scope, not unfinished** — `make_code_styled_body_line`'s one call site always has the language `"mermaid"`, and no Mermaid grammar exists in either syntax set, so threading tokens through it would be provably inert (`mermaid_has_no_grammar_so_that_surface_stays_out_of_scope`).
- **`syntax_highlighting` must be in `RenderSettings`** — the `Block` value is unchanged by the toggle, so without the fingerprint field a cached hit repaints with the old setting. It reaches `EditorState` through `app::configure_new_editor`, like `big_h1` and `cursor_blink`, so a document opened by following a link gets it too.