Struct MemTableWriter 

pub struct MemTableWriter { /* private fields */ }

In-memory write buffer that batches small inserts before persisting.

Problem: streaming pipelines (Flink, Spark Streaming) emit small RecordBatches every few seconds. Calling write_batch_deferred on each micro-batch creates many tiny Parquet files and triggers repeated HNSW builds — both are expensive.

Solution: buffer rows in RAM, flush to a single Parquet shard only when the buffer reaches the configured size/row/time threshold. The deferred HNSW build runs once per flush, not once per micro-batch.

Usage

let mut mt = MemTableWriter::new(catalog, store, policy, table, MemTableConfig::default());
loop {
    mt.insert(&batch, &embeddings).await.unwrap();
    mt.flush_if_due().await.unwrap();
}
mt.flush().await.unwrap();

Implementations

impl MemTableWriter

pub fn new( catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, policy: VectorStoragePolicy, table: TableIdent, config: MemTableConfig, ) -> Self

pub async fn insert( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Option<SnapshotId>>

Buffer a micro-batch. Flushes automatically if size or row threshold exceeded. Returns Some(snapshot_id) when an automatic flush occurred, None otherwise.

pub async fn flush_if_due(&mut self) -> AilakeResult<Option<SnapshotId>>

Flush if flush_interval has elapsed since the last flush. Returns Some(snapshot_id) when a flush occurred, None otherwise.

pub async fn flush(&mut self) -> AilakeResult<SnapshotId>

Flush all buffered data immediately, even if thresholds are not met. Returns the committed SnapshotId. Calling on an empty buffer is a no-op and returns SnapshotId::default().

pub fn buffered_rows(&self) -> usize

Number of rows currently in the buffer.

pub fn buffered_bytes(&self) -> usize

Estimated byte size of the embedding data currently in the buffer.

pub fn is_full(&self) -> bool

True when a size or row-count threshold has been reached.