Struct TableWriter 

pub struct TableWriter { /* private fields */ }

Implementations

impl TableWriter

pub fn new( catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, policy: VectorStoragePolicy, table: TableIdent, ) -> Self

pub fn with_parent_snapshot(self, id: SnapshotId) -> Self

pub async fn write_batch_deferred( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write batch as Parquet-only immediately, build HNSW in background.

Returns after the Parquet file is persisted (~LanceDB write speed). A tokio task runs concurrently to build the HNSW index, rewrite the file with the AILK section, and update the catalog entry.

During the build window, SearchSession serves this shard via flat scan (brute-force, exact) instead of HNSW. The transition is automatic once the background task commits the updated manifest entry.

pub async fn write_batch_ivf_pq_deferred( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ivf_config: IvfPqConfig, ) -> AilakeResult<()>

Write batch as Parquet-only immediately; train IVF-PQ index in background.

The first shard trains the shared codebook (k-means). All subsequent shards reuse it via OnceCell — build is O(n) assign+encode, not O(n×k) k-means. Returns after Parquet is persisted. Index transitions Indexing → Ready async.

pub async fn write_batch_idempotent( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], batch_id: &str, ) -> AilakeResult<()>

Idempotent variant of write_batch.

Before any I/O, checks if batch_id already appears in the current snapshot. If it does, this is a no-op — safe for Airflow/Kestra retries. If not found, writes the batch and tags the DataFileEntry with batch_id so future retries can detect it.

commit() is likewise a no-op when pending_files is empty.

pub async fn write_batch( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write a batch to a new AI-Lake file and stage it for commit.

pub async fn write_batch_auto( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<()>

Write batch, auto-selecting the index based on detected hardware.

Picks IVF-PQ when a CUDA GPU or ≥8 CPU cores are present AND the batch has ≥5 000 vectors. Falls back to HNSW for weaker / local hardware. Uses IvfPqConfig::for_dataset to scale nlist with dataset size.

pub async fn write_batch_ivf_pq( &mut self, batch: &RecordBatch, embeddings: &[Vec<f32>], ivf_config: IvfPqConfig, ) -> AilakeResult<()>

Write batch with IVF-PQ index built synchronously (no background task).

Smaller index than HNSW; better for S3 sequential-scan workloads.

pub async fn write_batch_multi( &mut self, batch: &RecordBatch, columns: &[MultiVectorBatch<'_>], ) -> AilakeResult<()>

Write a batch with multiple vector columns into a single AI-Lake file.

The first entry in columns is treated as the primary column (used for geometric pruning). Additional columns each get their own HNSW section.

pub async fn commit(self) -> AilakeResult<SnapshotId>

Commit all staged files as a new Iceberg snapshot.

No-op when pending_files is empty (e.g., all write_batch_idempotent calls were skipped because their batch_id was already committed). Returns the current snapshot id in that case (or 0 if no snapshot exists yet).

pub async fn create_or_open( catalog: Arc<dyn CatalogProvider>, store: Arc<dyn Store>, policy: VectorStoragePolicy, table: TableIdent, ) -> AilakeResult<Self>

Create a table if it doesn’t exist, then return a writer for it.