Function tokenize_segment 

pub fn tokenize_segment(
    engine: &HighlightEngine,
    snapshot: &Snapshot,
    rows: Range<u32>,
    start: SegmentStart,
    spans_for: Option<Range<u32>>,
    converge_against: Option<&SegmentTokens>,
) -> SegmentTokens

Tokenize rows of snapshot off-thread - pure, sync, unbudgeted (the app runs it on a worker over an O(1) Snapshot clone). Records stride checkpoints and, for spans_for intersected with rows, dense spans+states.

converge_against: when RE-running a segment from a corrected start (a speculative guess proved wrong), pass the old result. At each stride checkpoint the new end state is compared with the old’s; on the first match the tail is spliced verbatim from the old (provably identical downstream) - so repair costs O(distance to the construct’s close), not O(segment). Pass it on a re-run whose rows match the old segment’s; the window is forced to the old segment’s (the passed spans_for is ignored when converge_against is Some), so the caller need not track which window the old segment was tokenized with — the splice stays internally consistent even if the viewport moved since.