Struct AilakeFileWriter 

pub struct AilakeFileWriter { /* private fields */ }

Implementations

impl AilakeFileWriter

pub fn new(policy: VectorStoragePolicy) -> Self

pub fn with_fts(self, config: FtsConfig) -> Self

Attach a Tantivy FTS config. An AILK_FTS section will be built and embedded.

pub fn with_prebuilt_fts_blob(self, blob: Vec<u8>) -> Self

Supply a pre-built FTS blob (e.g., from compaction). Takes priority over with_fts.

pub fn with_shared_ivf_codebook(self, codebook: Arc<IvfPqCodebook>) -> Self

Use a pre-trained IVF-PQ codebook instead of running k-means. Shards built from the same codebook produce comparable ADC distances.

pub fn with_hnsw_config(self, config: HnswConfig) -> Self

pub fn with_ivf_pq(self, config: IvfPqConfig) -> Self

pub fn with_index_type(self, index_type: IndexType) -> Self

pub fn with_auto_index(self) -> Self

Use IndexType::Auto: detect GPU / CPU cores at write time and pick the best index. Equivalent to .with_index_type(IndexType::Auto).

pub fn write_with_prebuilt_hnsw( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], hnsw: &HnswIndex, ) -> AilakeResult<Bytes>

Write batch + embeddings using a pre-built HnswIndex.

Skips the O(N log N) HNSW construction — the caller is responsible for building and populating the index. The index must contain exactly embeddings.len() nodes with RowIds 0..N matching embeddings[0..N] in order.

Used by incremental compaction to reuse the dominant file’s existing index and only insert vectors from smaller files.

pub fn write_parquet_only( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Bytes>

Write RecordBatch + embeddings as plain Parquet, with no AILK section.

Used by TableWriter::write_batch_deferred() to persist data immediately while the HNSW index is built asynchronously in the background. The resulting file is a valid Parquet readable by any standard reader, but AilakeFileReader::is_ailake_file() returns false until the HNSW section is appended by the background indexing task.

pub fn write( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Bytes>

Write RecordBatch + embeddings into a single AI-Lake file.

Layout: [PAR1][row groups][AILK header+centroid+HNSW+trailer][Parquet footer][footer_len][PAR1]

Standard Parquet readers find PAR1 at the end, read the footer, skip directly to row group offsets. The AILK section sits between row groups and footer and is never touched. AI-Lake readers find the AILK section via ailake.footer_offset in the Parquet footer KV.

pub fn write_single_pass( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Bytes>

Single-pass streaming write — no ailake.footer_offset KV injection.

Produces a valid AI-Lake file in a single Parquet write pass without any seek or footer rewrite. Safe for append-only destinations (HDFS strict mode, piped stdout, write-once distributed filesystems).

Readers bootstrap the AILK section from the AilakeTrailer (the 24 bytes immediately preceding the Parquet footer) instead of from the Parquet KV metadata. Both bootstrap paths are supported by AilakeFileReader.

Trade-off vs write():

• One Parquet write instead of two — saves CPU + memory for large batches.
• On S3, the AILK offset is derived from bytes already fetched in the footer range-GET (trailer is the 24 bytes before the footer). No extra GET.

pub fn write_multi_single_pass( &self, batch: &RecordBatch, columns: &[VectorColumnBatch<'_>], ) -> AilakeResult<Bytes>

Multi-column variant of write_single_pass.

pub fn write_multi( &self, batch: &RecordBatch, columns: &[VectorColumnBatch<'_>], ) -> AilakeResult<Bytes>

Write RecordBatch + multiple vector columns into a single AI-Lake file.

Each column gets its own AILK section appended sequentially before the Parquet footer. Offsets are recorded in Parquet KV metadata:

• Primary (first) column: ailake.footer_offset
• Additional columns: ailake.{column_name}.footer_offset

Readers use the column-specific KV key to locate the right AILK section.