Struct StreamingDeletionVectorWriter 

pub struct StreamingDeletionVectorWriter<'a, W: Write> { /* private fields */ }

A streaming writer for deletion vectors.

This writer allows for writing multiple deletion vectors to a single file in a streaming fashion, which is memory-efficient for distributed workloads where deletion vectors are generated on executors.

Format

The writer produces deletion vector files in the Delta Lake format:

• The first byte of the file is a version byte (currently 1)
• Each DV is prefixed with a 4-byte size (big-endian) of the serialized data
• Followed by a 4-byte magic number (0x64485871, little-endian)
• Followed by the serialized 64-bit Roaring Bitmap
• Followed by a 4-byte CRC32 checksum (big-endian) of the serialized data

Examples

use delta_kernel::actions::deletion_vector_writer::{StreamingDeletionVectorWriter, KernelDeletionVector};

let mut buffer = Vec::new();
let mut writer = StreamingDeletionVectorWriter::new(&mut buffer);

let mut dv = KernelDeletionVector::new();
dv.add_deleted_row_indexes([1, 5, 10]);

let descriptor = writer.write_deletion_vector(dv)?;
writer.finalize()?;

Implementations

impl<'a, W: Write> StreamingDeletionVectorWriter<'a, W>

pub fn new(writer: &'a mut W) -> Self

Create a new streaming deletion vector writer.

Arguments

• writer - A mutable reference to any type implementing std::io::Write.

Examples

let mut buffer = Vec::new();
let writer = StreamingDeletionVectorWriter::new(&mut buffer);

pub fn write_deletion_vector( &mut self, deletion_vector: impl DeletionVector, ) -> DeltaResult<DeletionVectorWriteResult>

Write a deletion vector to the underlying writer.

This method can be called multiple times to write multiple deletion vectors to the same writer. The caller is responsible for keeping track of which deletion vector corresponds to which data file.

Arguments

• deletion_vector - The deletion vector to write

Returns

A DeletionVectorWriteResult containing the offset, size, and cardinality of the written deletion vector.

Errors

Returns an error if:

• The writer fails to write data
• The deletion vector cannot be serialized
• The offset or size would overflow an i32

Examples

let mut dv = KernelDeletionVector::new();
dv.add_deleted_row_indexes([1, 5, 10]);

let descriptor = writer.write_deletion_vector(dv)?;
println!("Written DV at offset {} with size {}", descriptor.offset, descriptor.size_in_bytes);

pub fn finalize(self) -> DeltaResult<()>

Finalize all writes and flush the underlying writer.

This method should be called after all deletion vectors have been written. After calling this method, the writer should not be used anymore.

Errors

Returns an error if flushing the writer fails.

Examples

writer.write_deletion_vector(dv1)?;
writer.write_deletion_vector(dv2)?;
writer.finalize()?;