Struct StreamingBorderGram 

pub struct StreamingBorderGram { /* private fields */ }

Chunked, out-of-core, bit-reproducible border-Gram accumulator.

Accumulates G = Σ_n x_n x_nᵀ ∈ ℝ^{k×k} over n_rows rows submitted as fixed-size chunks (any submission order), with f64 accumulation throughout and a deterministic pairwise reduction tree whose shape is a pure function of (n_rows, chunk_size). See the module docs for the determinism contract.

Implementations

impl StreamingBorderGram

pub fn new( border_dim: usize, n_rows: usize, chunk_size: usize, ) -> Result<Self, String>

Create an empty accumulator for n_rows total rows of border dimension border_dim, streamed in chunks of chunk_size rows.

pub fn n_chunks(&self) -> usize

Total number of chunks of the pass: ceil(n_rows / chunk_size).

pub fn chunk_rows(&self, chunk_index: usize) -> Range<usize>

Row range covered by chunk chunk_index: [chunk_index·chunk_size, min((chunk_index+1)·chunk_size, n_rows)). A pure function of the partition parameters — the caller slices its shard rows with exactly this range.

pub fn frontier(&self) -> usize

Number of chunks already consumed by the in-order cascade (the chunk cursor). Pending out-of-order chunks are not counted.

pub fn is_complete(&self) -> bool

true once every chunk of the pass has been submitted.

pub fn submit_chunk( &mut self, chunk_index: usize, rows: ArrayView2<'_, f64>, ) -> Result<(), String>

Submit the rows of chunk chunk_index (shape (chunk_rows(chunk_index).len(), border_dim)).

Chunks may arrive in any order; each may be submitted exactly once. The per-chunk Gram contribution is computed immediately (each entry a pairwise_sum over the chunk’s rows, in row order), so the caller’s row buffer can be dropped/remapped right after this returns.

pub fn submit_chunk_gram( &mut self, chunk_index: usize, gram: Vec<f64>, ) -> Result<(), String>

Submit chunk chunk_index as a precomputed per-chunk Gram partial (flattened k·k row-major), produced by chunk_gram_flat over exactly the rows of Self::chunk_rows(chunk_index).

This is the cross-node ingestion seam (crate::cross_node): a worker node computes its chunks’ partials locally and ships the k·k values; the coordinator folds them through the same fixed in-order cascade as row-level submission, so the result is bit-identical to a single process having seen all the rows. The validation here is structural (index range, duplicate, partial length); the content contract — that the partial really is chunk_gram_flat of the chunk’s rows — is the producer’s, enforced by routing both producers through the one free function.

pub fn checkpoint(&self) -> BorderGramCheckpoint

Serialize the full accumulation state — partial Grams + chunk cursor — for checkpointing. StreamingBorderGram::resume reconstructs an accumulator whose future behavior is bit-identical to never having stopped.

pub fn resume(state: BorderGramCheckpoint) -> Result<Self, String>

Reconstruct an accumulator from a checkpoint. Validates the structural invariants so a corrupted checkpoint is rejected loudly instead of silently producing a wrong (but plausible-looking) Gram.

pub fn finish(self) -> Result<Array2<f64>, String>

Finish the pass, returning the k×k border Gram. Errors if any chunk is missing (out-of-order pending chunks the frontier never reached, or chunks never submitted). The result is a pure function of the row content: identical bits for any submission order and for any checkpoint/resume history.