Struct BatchIngestPipeline 

pub struct BatchIngestPipeline<Id>
where
    Id: Clone + Eq + Debug,
{ /* private fields */ }

High-throughput batch ingest pipeline for encoding and inserting large vector corpora into a FibSidecarIndex.

The pipeline owns a FibQuantizer for encoding and a FibSidecarIndex for storage. Vectors are processed in configurable batch sizes via ingest_batch or ingest_from_iter, with each batch producing an IngestReceipt for progress tracking.

Parallelism

FibQuantizer::encode_batch uses Rayon parallelism internally when the parallel feature is enabled (default) and the batch is large enough to amortize dispatch overhead (≥ 16 vectors). The pipeline itself does not add additional parallelism — it relies on the codec’s internal parallelism for the compute-heavy encoding step.

Generics

Id must be Clone + Eq + Debug, matching the bound on FibSidecarIndex. Common choices are u64, String, or a newtype key.

Implementations

impl<Id> BatchIngestPipeline<Id>

where Id: Clone + Eq + Debug,

pub fn new(quantizer: FibQuantizer, batch_size: usize) -> Result<Self>

Create a new batch ingest pipeline.

Builds a FibScorer from the quantizer (cloning it so the pipeline retains its own copy for encoding) and creates an empty FibSidecarIndex.

batch_size is the default chunk size used by ingest_from_iter. It must be > 0.

Errors

Returns FibQuantError::CorruptPayload if batch_size is 0.

pub fn ingest_batch(&mut self, items: &[(Id, &[f32])]) -> Result<IngestReceipt>

Ingest a batch of (Id, vector) pairs.

Extracts all f32 vector slices, calls FibQuantizer::encode_batch for parallel encoding (Rayon when the parallel feature is enabled and the batch is ≥ 16 vectors), then adds all encoded codes to the sidecar index via FibSidecarIndex::add_batch. Returns an IngestReceipt with the count, bytes, timing, and any errors.

If encoding fails for the entire batch (e.g., dimension mismatch or a zero-norm vector), the receipt will have failed = items.len() and the error will be recorded in error_messages. The index is not modified in this case.

Empty batches

An empty items slice returns a zero-count receipt immediately without touching the quantizer or index.

pub fn ingest_from_iter<I>(&mut self, iter: I) -> Result<Vec<IngestReceipt>>

where I: Iterator<Item = (Id, Vec<f32>)>,

Ingest vectors from an iterator, chunking into batch_size pieces.

Each chunk is passed to ingest_batch, and all receipts are collected and returned. This is the primary entry point for streaming large corpora (100K+ vectors) through the pipeline.

The iterator is consumed lazily — only one chunk is held in memory at a time (plus the receipts vector). This keeps memory usage bounded regardless of the total corpus size.

pub fn finish(self) -> FibSidecarIndex<Id>

Consume the pipeline and return the completed FibSidecarIndex.

The index contains all successfully encoded and inserted entries. After calling finish, the pipeline is consumed and can no longer be used.

pub fn total_encoded(&self) -> usize

Number of vectors successfully encoded and inserted so far.

pub fn total_bytes(&self) -> u64

Total encoded bytes accumulated so far.

pub fn failed(&self) -> usize

Number of vectors that failed to encode or insert so far.

pub fn batch_size(&self) -> usize

Configured batch size.

pub fn len(&self) -> usize

Current number of entries in the underlying index.

pub fn is_empty(&self) -> bool

Whether the index is empty.