Struct GtAccumulator 

pub struct GtAccumulator { /* private fields */ }

Streaming, single-pass ground-truth builder.

pipeline.rs draws every duplicate copy of a batch exclusively from that same batch’s masters, and writes the copies right after the batch. So at the end of each batch, whether each of its masters is duplicated — and whether at least one copy is byte-identical to it — is already final: no later batch can add a copy of it. Each batch is therefore classified and streamed straight to the final ground-truth file (IPC or Parquet) as soon as its duplicates are known, with only two per-batch scratch bitsets as state. Hard negatives (HN-…) and canaries (CANARY-…) are never duplicated and are classified by kind.

This replaces an earlier two-pass design (draft file of every row + global duplicated-master hash sets + full re-read and reclassification at the end), which an isolated benchmark measured at ~4x the cost (examples/profile_gt_single_pass.rs, byte-identical output).

Implementations

impl GtAccumulator

pub fn new( difficulty: &str, output_format: &str, final_path: &str, metadata: &BTreeMap<String, String>, track_clusters: bool, ) -> Result<Self, String>

pub fn push_entity_batch( &mut self, base: &BaseRows<'_>, dups: Option<&DupRows<'_>>, ) -> Result<(), String>

Classifies and writes one entity batch: its base rows first, then its duplicate copies (dups, None when the batch has none) — the same order they appear in the dataset.

• A base row is unique if no copy of its master exists, otherwise exact_dup if at least one copy is byte-identical to it (a base row has no noise of its own: it reads as an exact duplicate only if the cluster genuinely contains an identical twin), else fuzzy_dup.
• A copy is exact_dup if its noise was a no-op, else fuzzy_dup.

pub fn push_hard_neg_batch( &mut self, record_ids: &ArrayRef, entity_types: &ArrayRef, master_ids: &ArrayRef, ) -> Result<(), String>

Writes a batch of hard-negative rows (HN-… masters): always hard_neg, never part of a duplicate cluster.

pub fn push_canary_batch( &mut self, record_ids: &ArrayRef, entity_types: &ArrayRef, master_ids: &ArrayRef, ) -> Result<(), String>

Writes a batch of canary rows (CANARY-… masters, see canary.rs): always canary, not counted in any statistic.

pub fn finish(self) -> Result<GtResult, String>

Closes the ground-truth file and returns the run’s statistics (plus the duplicate-cluster map, when tracked).