Struct StreamDecoder

Source

pub struct StreamDecoder { /* private fields */ }

Expand description

A low-level interface for reading RecordBatch data from a stream of bytes

See StreamReader for a higher-level interface

Implementations§

Source §

impl StreamDecoder

Source

pub fn new() -> Self

Create a new StreamDecoder

Source

pub fn with_require_alignment(self, require_alignment: bool) -> Self

Specifies whether or not array data in input buffers is required to be properly aligned.

If require_alignment is true, this decoder will return an error if any array data in the input buf is not properly aligned. Under the hood it will use arrow_data::ArrayDataBuilder::build to construct arrow_data::ArrayData.

If require_alignment is false (the default), this decoder will automatically allocate a new aligned buffer and copy over the data if any array data in the input buf is not properly aligned. (Properly aligned array data will remain zero-copy.) Under the hood it will use arrow_data::ArrayDataBuilder::build_aligned to construct arrow_data::ArrayData.

Source

pub fn schema(&self) -> Option<SchemaRef>

Return the schema if decoded, else None.

Source

pub unsafe fn with_skip_validation(self, skip_validation: bool) -> Self

Specifies if validation should be skipped when reading data (defaults to false)

§Safety

This flag must only be set to true when you trust the input data and are sure the data you are reading is valid Arrow IPC stream data, otherwise undefined behavior may result.

For example, DataFusion uses this when reading spill files it wrote itself.

Source

pub fn decode( &mut self, buffer: &mut Buffer, ) -> Result<Option<RecordBatch>, ArrowError>

Try to read the next RecordBatch from the provided Buffer

Buffer::advance will be called on buffer for any consumed bytes.

The push-based interface facilitates integration with sources that yield arbitrarily delimited bytes ranges, such as a chunked byte stream received from object storage

fn print_stream<I>(src: impl Iterator<Item = Buffer>) -> Result<(), ArrowError> {
    let mut decoder = StreamDecoder::new();
    for mut x in src {
        while !x.is_empty() {
            if let Some(x) = decoder.decode(&mut x)? {
                println!("{x:?}");
            }
            if let Some(schema) = decoder.schema() {
                println!("Schema: {schema:?}");
            }
        }
    }
    decoder.finish().unwrap();
    Ok(())
}