Module batch_ingest 

High-throughput batch ingest pipeline for encoding and inserting large vector corpora (100K+) into a FibSidecarIndex.

The pipeline wraps a FibQuantizer and FibSidecarIndex, encoding vectors in batches via FibQuantizer::encode_batch (which uses Rayon parallelism when the parallel feature is enabled) and inserting the resulting FibCodeV1 artifacts into the sidecar index. Each batch produces an IngestReceipt with timing, byte counts, and any errors.

Throughput

The pipeline is designed for 100K+ vector corpora. The encoding step is dominated by FibQuantizer::encode_batch, which dispatches the per-vector codebook lookup across Rayon worker threads when the batch is large enough (≥ 16 vectors). The index insertion step (FibSidecarIndex::add_batch) is a simple Vec::push loop with no per-entry allocation beyond the (Id, FibCodeV1) tuple.

Example

use fib_quant::{BatchIngestPipeline, FibQuantProfileV1, FibQuantizer};

fn main() -> fib_quant::Result<()> {

let mut profile = FibQuantProfileV1::paper_default(8, 2, 8, 7)?;

profile.training_samples = 128;

profile.lloyd_restarts = 1;

profile.lloyd_iterations = 2;

let quantizer = FibQuantizer::new(profile)?;

let mut pipeline = BatchIngestPipeline::new(quantizer, 32)?;

let items: Vec<(u32, Vec)> = (0..100)

.map(|i| (i as u32, vec![0.1 * i as f32 + 0.01; 8]))

.collect();

let receipts = pipeline.ingest_from_iter(items.into_iter())?;

let total: usize = receipts.iter().map(|r| r.batch_count).sum();

assert_eq!(total, 100);

let index = pipeline.finish();

assert_eq!(index.len(), 100);

Ok(())

}

Structs

BatchIngestPipeline

High-throughput batch ingest pipeline for encoding and inserting large vector corpora into a FibSidecarIndex.

IngestReceipt

Receipt documenting the outcome of a single batch ingest operation.