Trait StreamingWriter

Source

pub trait StreamingWriter: Send {
    // Required methods
    fn write_header(
        &mut self,
        metadata: &QueryMetadata,
    ) -> Result<(), OutputError>;
    fn write_chunk(&mut self, rows: &[QueryRow]) -> Result<usize, OutputError>;
    fn finalize(&mut self) -> Result<(), OutputError>;
    fn rows_written(&self) -> u64;

    // Provided method
    fn bytes_written(&self) -> Option<u64> { ... }
}

Expand description

Trait for writers that support streaming/chunked output

Unlike batch writers that receive a complete QueryResult, streaming writers process data incrementally via write_header(), write_chunk(), and finalize(). This enables processing of arbitrarily large result sets within the 128MB memory budget.

§Memory Budget

To stay within the 128MB target:

Chunk sizes should typically be 5,000-10,000 rows
For large blob/text columns, use smaller chunks (1,000-5,000)
Parquet writers buffer rows for row groups; default is 10,000 rows

§Contract

write_header() MUST be called exactly once before any write_chunk() calls
write_chunk() MAY be called zero or more times
finalize() MUST be called exactly once to complete the output
After finalize(), no further calls are allowed
Implementors SHOULD return errors rather than panic on contract violations

§Troubleshooting

If you encounter OOM errors:

Reduce chunk sizes when calling write_chunk() (max: 10,000 rows)
For Parquet, reduce row_group_size (default: 10,000 rows)
Check for large blob/text columns that inflate row size

§Example

let mut writer = StreamingCSVWriter::new(file);
writer.write_header(&metadata)?;

for chunk in result_stream.chunks(10_000) {
    writer.write_chunk(&chunk)?;
}

writer.finalize()?;