Struct Writer 

pub struct Writer { /* private fields */ }

A writer for the Lance v2.0 file grammar.

Implementations

impl Writer

pub fn try_new( object_writer: Box<dyn ObjectWriter>, schema: LanceSchema, options: FileWriterOptions, ) -> Result<Self>

Create a new v2.0 writer with a desired output schema.

pub fn new_lazy( object_writer: Box<dyn ObjectWriter>, options: FileWriterOptions, ) -> Self

Create a new v2.0 writer without a desired output schema.

The output schema will be set based on the first batch of data to arrive. If no data arrives and the writer is finished then the write will fail.

pub fn with_page_metadata_spill( self, object_store: Arc<ObjectStore>, path: Path, ) -> Self

Spill page metadata to a sidecar file instead of accumulating in memory.

This can dramatically reduce memory usage when many writers are open concurrently (e.g. IVF shuffle with thousands of partition writers). The sidecar file is created lazily on the first page write. The caller is responsible for cleaning up path (e.g. by placing it in a temp directory that is removed via RAII).

pub async fn write_batches( &mut self, batches: impl Iterator<Item = &RecordBatch>, ) -> Result<()>

Schedule batches of data to be written to the file

pub async fn write_batch(&mut self, batch: &RecordBatch) -> Result<()>

Schedule a batch of data to be written to the file

Note: the future returned by this method may complete before the data has been fully flushed to the file (some data may be in the data cache or the I/O cache)

pub async fn write_column( &mut self, column_index: usize, array: ArrayRef, ) -> Result<()>

Write a single column, advancing only that column’s row counter.

Unlike write_batch, which advances every column from a single shared row counter, this method advances one column independently. Used across calls it produces a single file whose columns may have different item counts.

column_index refers to a top-level field in the writer’s schema (the same order as the schema’s fields); a nested child cannot be targeted on its own. Because each call writes the whole field from a single array, the children of a struct field always advance together and stay equal-length; only different top-level fields can diverge in length. A column may be written across multiple calls; its values are appended. A field that is never written ends up as a zero-length column. The writer must have been created with an explicit schema (via try_new); a lazy schema cannot be inferred here because individual calls need not cover every field.

// Field 0 gets three values, field 1 gets one — a non-rectangular file.
writer.write_column(0, Arc::new(Int32Array::from(vec![1, 2, 3]))).await?;
writer.write_column(1, Arc::new(Int32Array::from(vec![10]))).await?;

pub fn add_schema_metadata( &mut self, key: impl Into<String>, value: impl Into<String>, )

Add a metadata entry to the schema

This method is useful because sometimes the metadata is not known until after the data has been written. This method allows you to alter the schema metadata. It must be called before finish is called.

pub fn initialize_with_external_metadata( &mut self, schema: Schema, column_metadata: Vec<ColumnMetadata>, rows_written: u64, )

Prepare the writer when column data and metadata were produced externally.

This is useful for flows that copy already-encoded pages (e.g., binary copy during compaction) where the column buffers have been written directly and we only need to write the footer and schema metadata. The provided column_metadata must describe the buffers already persisted by the underlying ObjectWriter, and rows_written should reflect the total number of rows in those buffers.

pub async fn add_global_buffer(&mut self, buffer: Bytes) -> Result<u32>

Adds a global buffer to the file

The global buffer can contain any arbitrary bytes. It will be written to the disk immediately. This method returns the index of the global buffer (this will always start at 1 and increment by 1 each time this method is called)

pub async fn finish(&mut self) -> Result<FileWriteSummary>

Finishes writing the file

This method will wait until all data has been flushed to the file. Then it will write the file metadata and the footer. It will not return until all data has been flushed and the file has been closed.

Returns a summary of the completed file write.

pub async fn abort(&mut self)

pub async fn tell(&mut self) -> Result<u64>

pub async fn write_external_buffer( &mut self, bytes: &[u8], ) -> Result<(u64, u64)>

Append a buffer whose metadata is supplied by the caller.

pub fn field_id_to_column_indices(&self) -> &[(u32, u32)]