Crate devup_editor_syntax_highlight 

Tree-sitter-based syntax highlighting for code blocks.

Why this crate exists

devup-editor is a Notion-style block editor. A code block carries a language prop (e.g. "rust", "ts"). When the block renders, we need to turn its flat text content into a sequence of styled tokens so the view layer can paint each slice with a colour.

Doing this in Rust is deliberate. The WASM boundary for this module is:

(code: &str, language: &str)  →  Vec<Token { start_utf16, end_utf16, kind: &str }>

Both sides of that boundary are primitives, which is exactly the case where porting to Rust pays for itself (see the primitive-boundary rule in AGENTS.md). A full 200-line code block serialised as a token list is a few KB of JSON — the parse cost dominates the JSON round-trip cost by orders of magnitude.

Engine choice: arborium

We use arborium rather than tree-sitter + individual grammar crates for one reason only: tree-sitter-* grammar crates on crates.io as of 2026-01 still fail cargo build --target wasm32-unknown-unknown with stdlib.h not found. The core tree-sitter crate was patched (PR #4820, merged Sep 2025) but the fix has not propagated to grammar crates. arborium ships its own arborium-sysroot and re-generates every grammar against it, and its CI verifies the output .wasm has no unsatisfied imports.

See the librarian research report in the session history for the full picture.

Supported language set

We ship with the following grammars, selected to cover the majority of code blocks a Notion-style editor encounters in the wild:

• rust — project’s primary language
• typescript / javascript — React ecosystem consumers
• python — broad user base
• json — config-file ubiquity
• bash — README examples
• html / css — frontend context
• markdown — meta case (code blocks inside code blocks)
• sql — query pastes

The measured WASM bundle cost is ~1.5 MB gzipped across all of these grammars. See .sisyphus/plans/tier-a-benchmark-decisions.md for the budget discussion. Adding another grammar is a one-line change to Cargo.toml features, a one-line normalize_language entry, and a one-arm addition to [compile_grammar]. Do it deliberately — each new grammar adds roughly 50–300 KB gzipped.

Structs

Token

A highlighted range within the source, ready to cross the WASM boundary. Offsets are UTF-16 code units to match the convention used everywhere else in this codebase (TextSpan, selection, utf16_len) — the JS side does string.slice(token.start, token.end) directly with these numbers.

Enums

TokenKind

A coarse token kind. The view layer maps each kind to a colour via theme tokens. We deliberately flatten arborium’s capture-name taxonomy (which has ~50 distinct strings like "keyword.function", "string.special.symbol", …) into a small enum because:

Functions

highlight

Highlight code as language and return a de-duplicated, left-to-right sorted list of tokens with UTF-16 offsets.

is_supported_language

Names we accept from the editor’s language prop.