Module highlight 

Syntax highlighting — a GUI-free public surface over a private syntect engine. Syntect is named only inside this module; nothing from it crosses the pub line, so the GUI crate never has to pin syntect’s version.

The core is language-agnostic: the grammar is an app-supplied .sublime-syntax, injected as text (SyntaxDef::from_sublime_syntax), and the theme an app-supplied .tmTheme (TokenTheme::from_tm_theme); scrive-core ships neither.

Two entry points

Highlighter::highlight tokenizes the whole document top-to-bottom on each call (state carried line to line) — correct but O(lines), used as the convergence oracle in tests. Production reads go through HighlightCache, the incremental engine: geometric shift on edit (HighlightCache::on_commit), lazy end-state convergence (HighlightCache::tokenize_until), and full invalidation on theme change (HighlightCache::set_theme). Untokenized lines return None and render in the default style — never an error or a stall. Each drive tokenizes at most HIGHLIGHT_MAX_LINES_PER_CALL lines, so no single call can stall a frame however far a cascade wants to run.

Retention is virtualized so RAM does not grow with the idle sweep: spans + per-line states live only in a window around the viewport (HighlightCache::set_window); everywhere else, sparse checkpoints (HIGHLIGHT_CHECKPOINT_STRIDE) keep every row re-derivable. A fully swept document holds O(window + lines/stride), not O(lines) — see HighlightCache for the mechanics.

Structs

HighlightCache

The document-owned incremental, virtualized highlight cache.

HighlightEngine

A cheap-to-clone, Send + Sync handle to a grammar + theme for off-thread bulk tokenization. Nothing syntect crosses its surface; SegmentBoundary and SegmentTokens are opaque. Obtain via HighlightCache::engine (to share the cache’s exact grammar/theme) or HighlightEngine::new, and move clones to worker threads.

HighlightSpan

One styled run within a single buffer line. range is bytes within the line (not document offsets), so it survives line-local repair and drops straight onto a display chunk; byte→cell conversion happens only at render.

Highlighter

A whole-document highlighter over a grammar + theme. Owns both; syntect stays private behind it.

Rgba

A GUI-free color; scrive-iced maps it to iced::Color at render time.

SegmentBoundary

The per-line carry state at a segment edge - opaque, comparable, sendable. Two edges compare equal iff everything downstream is provably identical (the convergence theorem: same state + same text => same tokens).

SegmentTokens

One segment tokenized off-thread - opaque; ingested by crate::Document::absorb_highlight. Carries stride checkpoints, the end boundary, and (for a requested sub-range) dense window spans+states.

SpanStyle

Resolved inline style for one run — the theme is consulted at tokenize time (syntect’s highlight iterator already yields resolved styles), so nothing downstream re-touches syntect.

SyntaxDef

A parsed grammar. The app supplies a validated .sublime-syntax definition; scrive-core ships no grammar of its own and stays language-agnostic.

SyntaxError

Failed to parse an injected .sublime-syntax grammar.

ThemeError

Failed to parse a .tmTheme.

TokenTheme

A parsed highlight theme — opaque over syntect’s Theme. Clone is cheap (a syntect Theme is a small style table), so an integrating widget can retain a theme and re-apply it across a document reload or grammar swap.

Enums

SegmentStart

Where a segment’s tokenization begins.

Constants

HIGHLIGHT_CHECKPOINT_STRIDE

Sparse-checkpoint stride, in rows: outside the retention window the cache keeps one end state per this many tokenized rows (plus each budget-stop resume point), so any row’s start state is re-derivable by tokenizing at most a stride forward from the checkpoint above it. Memory per fully-swept document: lines / stride states instead of lines.

HIGHLIGHT_MAX_LINES_PER_CALL

Per-call tokenize budget, expressed as an op count — deterministic and testable, unlike wall-clock. At ~2–20 µs of syntect per line this is roughly 0.5–5 ms per call, so one drive can never stall a frame however far a state-changing cascade wants to run; the idle sweep resumes where the budget stopped.

HIGHLIGHT_MAX_WINDOW_ROWS

Hard cap on the retention window’s total length, in rows. A collapsed mega-fold makes the reported visible range span the fold’s hidden interior; capping the window keeps retention bounded there instead of re-growing to O(document). Rows past the cap render in the fallback style until scrolled to (which re-aims the window at them); a fold hiding fewer than ~3k rows still fits entirely.

HIGHLIGHT_WINDOW_SLACK

Retention slack, in rows, kept on EACH side of the window handed to HighlightCache::set_window — scrolling within the slack costs nothing; beyond it, evicted rows refill from the nearest checkpoint (budgeted).

Functions

padded_highlight_window

The retention/paint window for a viewport: viewport padded by HIGHLIGHT_WINDOW_SLACK on each side, its length capped at HIGHLIGHT_MAX_WINDOW_ROWS, and clamped to [0, n_lines]. The one owner of this formula: HighlightCache::set_window (what the cache retains) and any parallel-highlight worker pool (what it speculatively paints and tokenizes) must aim at the SAME rows, so both call this instead of re-deriving it — an integrator running the core’s speculative tokenizer on worker threads cannot drift from the core’s retention rule. The cap is load-bearing: a collapsed mega-fold makes the reported visible range span the fold’s hidden interior, and without it retention (or a pool’s synchronous speculate) would re-grow to O(document).

tokenize_segment

Tokenize rows of snapshot off-thread - pure, sync, unbudgeted (the app runs it on a worker over an O(1) Snapshot clone). Records stride checkpoints and, for spans_for intersected with rows, dense spans+states.