Struct SegmentTokens 

pub struct SegmentTokens { /* private fields */ }

One segment tokenized off-thread - opaque; ingested by crate::Document::absorb_highlight. Carries stride checkpoints, the end boundary, and (for a requested sub-range) dense window spans+states.

Implementations

impl SegmentTokens

pub fn rows(&self) -> Range<u32>

The rows this segment covers.

pub fn tokenized_rows(&self) -> u32

How many rows this segment actually tokenized (< rows().len() when a converging re-run spliced a proven-identical tail from the old segment).

pub fn end_boundary(&self) -> &SegmentBoundary

The end-of-segment boundary (the true state after rows.end - 1 once this segment is verified) - what the coordinator chains the next segment’s verification against.

pub fn started_fresh(&self) -> bool

Whether this segment started from the fresh (guessed) state.