Tree-sitter-based syntax highlighting for code blocks.

Why this crate exists

devup-editor is a Notion-style block editor. A code block carries a language prop (e.g. "rust", "ts"). When the block renders, we need to turn its flat text content into a sequence of styled tokens so the view layer can paint each slice with a colour.

Doing this in Rust is deliberate. The WASM boundary for this module is:

(code: &str, language: &str)  →  Vec<Token { start_utf16, end_utf16, kind: &str }>

Both sides of that boundary are primitives, which is exactly the case where porting to Rust pays for itself (see the primitive-boundary rule in AGENTS.md). A full 200-line code block serialised as a token list is a few KB of JSON — the parse cost dominates the JSON round-trip cost by orders of magnitude.

Engine choice: arborium

We use arborium rather than tree-sitter + individual grammar crates for one reason only: tree-sitter-* grammar crates on crates.io as of 2026-01 still fail cargo build --target wasm32-unknown-unknown with stdlib.h not found. The core tree-sitter crate was patched (PR #4820, merged Sep 2025) but the fix has not propagated to grammar crates. arborium ships its own arborium-sysroot and re-generates every grammar against it, and its CI verifies the output .wasm has no unsatisfied imports.

See the librarian research report in the session history for the full picture.

Supported language set

We ship with the following grammars, selected to cover the majority of code blocks a Notion-style editor encounters in the wild:

• rust — project's primary language
• typescript / javascript — React ecosystem consumers
• python — broad user base
• json — config-file ubiquity
• bash — README examples
• html / css — frontend context
• markdown — meta case (code blocks inside code blocks)
• sql — query pastes

The measured WASM bundle cost is ~1.5 MB gzipped across all of these grammars. See .sisyphus/plans/tier-a-benchmark-decisions.md for the budget discussion. Adding another grammar is a one-line change to Cargo.toml features, a one-line normalize_language entry, and a one-arm addition to [compile_grammar]. Do it deliberately — each new grammar adds roughly 50–300 KB gzipped.