Struct TableWriter 

pub struct TableWriter { /* private fields */ }

Implementations

impl TableWriter

pub fn new( catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, policy: VectorStoragePolicy, table: TableIdent, ) -> Self

pub fn with_parent_snapshot(self, id: SnapshotId) -> Self

pub async fn write_batch_deferred( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write batch as Parquet-only immediately, build HNSW in background.

Returns after the Parquet file is persisted (~LanceDB write speed). A tokio task runs concurrently to build the HNSW index, rewrite the file with the AILK section, and update the catalog entry.

During the build window, SearchSession serves this shard via flat scan (brute-force, exact) instead of HNSW. The transition is automatic once the background task commits the updated manifest entry.

pub async fn write_batch( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write a batch to a new AI-Lake file and stage it for commit.

pub async fn write_batch_auto( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write batch, auto-selecting the index based on detected hardware.

Picks IVF-PQ when a CUDA GPU or ≥8 CPU cores are present AND the batch has ≥5 000 vectors. Falls back to HNSW for weaker / local hardware. Uses IvfPqConfig::for_dataset to scale nlist with dataset size.

pub async fn write_batch_ivf_pq( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ivf_config: IvfPqConfig, ) -> AilakeResult<()>

Write batch with IVF-PQ index built synchronously (no background task).

Smaller index than HNSW; better for S3 sequential-scan workloads.

pub async fn write_batch_multi( &mut self, batch: &RecordBatch, columns: &[MultiVectorBatch<'_>], ) -> AilakeResult<()>

Write a batch with multiple vector columns into a single AI-Lake file.

The first entry in columns is treated as the primary column (used for geometric pruning). Additional columns each get their own HNSW section.

pub async fn commit(self) -> AilakeResult<SnapshotId>

Commit all staged files as a new Iceberg snapshot.

pub async fn create_or_open( catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, policy: VectorStoragePolicy, table: TableIdent, ) -> AilakeResult<Self>

Create a table if it doesn’t exist, then return a writer for it.