Struct AilakeFileWriter 

pub struct AilakeFileWriter { /* private fields */ }

Implementations

impl AilakeFileWriter

pub fn new(policy: VectorStoragePolicy) -> Self

pub fn with_hnsw_config(self, config: HnswConfig) -> Self

pub fn with_ivf_pq(self, config: IvfPqConfig) -> Self

pub fn with_index_type(self, index_type: IndexType) -> Self

pub fn with_auto_index(self) -> Self

Use IndexType::Auto: detect GPU / CPU cores at write time and pick the best index. Equivalent to .with_index_type(IndexType::Auto).

pub fn write_parquet_only( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Bytes>

Write RecordBatch + embeddings as plain Parquet, with no AILK section.

Used by TableWriter::write_batch_deferred() to persist data immediately while the HNSW index is built asynchronously in the background. The resulting file is a valid Parquet readable by any standard reader, but AilakeFileReader::is_ailake_file() returns false until the HNSW section is appended by the background indexing task.

pub fn write( &self, batch: &RecordBatch, embeddings: &[Vec<f32>], ) -> AilakeResult<Bytes>

Write RecordBatch + embeddings into a single AI-Lake file.

Layout: [PAR1][row groups][AILK header+centroid+HNSW+trailer][Parquet footer][footer_len][PAR1]

Standard Parquet readers find PAR1 at the end, read the footer, skip directly to row group offsets. The AILK section sits between row groups and footer and is never touched. AI-Lake readers find the AILK section via ailake.footer_offset in the Parquet footer KV.

pub fn write_multi( &self, batch: &RecordBatch, columns: &[VectorColumnBatch<'_>], ) -> AilakeResult<Bytes>

Write RecordBatch + multiple vector columns into a single AI-Lake file.

Each column gets its own AILK section appended sequentially before the Parquet footer. Offsets are recorded in Parquet KV metadata:

• Primary (first) column: ailake.footer_offset
• Additional columns: ailake.{column_name}.footer_offset

Readers use the column-specific KV key to locate the right AILK section.