Struct arrow2::io::parquet::write::FileSink

pub struct FileSink<'a, W: AsyncWrite + Send + Unpin> {
    pub metadata: AHashMap<String, Option<String>>,
    /* private fields */
}

Available on crate feature io_parquet only.

Sink that writes array chunks as a Parquet file.

Any values in the sink’s metadata field will be written to the file’s footer when the sink is closed.

Examples

use futures::SinkExt;
use arrow2::array::{Array, Int32Array};
use arrow2::datatypes::{DataType, Field, Schema};
use arrow2::chunk::Chunk;
use arrow2::io::parquet::write::{Encoding, WriteOptions, CompressionOptions, Version};

let schema = Schema::from(vec![
    Field::new("values", DataType::Int32, true),
]);
let encoding = vec![vec![Encoding::Plain]];
let options = WriteOptions {
    write_statistics: true,
    compression: CompressionOptions::Uncompressed,
    version: Version::V2,
    data_pagesize_limit: None,
};

let mut buffer = vec![];
let mut sink = FileSink::try_new(
    &mut buffer,
    schema,
    encoding,
    options,
)?;

for i in 0..3 {
    let values = Int32Array::from(&[Some(i), None]);
    let chunk = Chunk::new(vec![values.boxed()]);
    sink.feed(chunk).await?;
}
sink.metadata.insert(String::from("key"), Some(String::from("value")));
sink.close().await?;

Fields

metadata: AHashMap<String, Option<String>>

Key-value metadata that will be written to the file on close.

Implementations

impl<'a, W> FileSink<'a, W>where W: AsyncWrite + Send + Unpin + 'a,

pub fn try_new( writer: W, schema: Schema, encodings: Vec<Vec<Encoding>>, options: WriteOptions ) -> Result<Self, Error>

Create a new sink that writes arrays to the provided writer.

Error

Iff

• the Arrow schema can’t be converted to a valid Parquet schema.
• the length of the encodings is different from the number of fields in schema

pub fn schema(&self) -> &Schema

The Arrow Schema for the file.

pub fn parquet_schema(&self) -> &SchemaDescriptor

The Parquet SchemaDescriptor for the file.

pub fn options(&self) -> &WriteOptions

The write options for the file.

Trait Implementations

impl<'a, W> Sink<Chunk<Box<dyn Array, Global>>> for FileSink<'a, W>where W: AsyncWrite + Send + Unpin + 'a,

type Error = Error

The type of value produced by the sink when an error occurs.

fn start_send( self: Pin<&mut Self>, item: Chunk<Box<dyn Array>> ) -> Result<(), Self::Error>

Begin the process of sending a value to the sink. Each call to this function must be preceded by a successful call to poll_ready which returned Poll::Ready(Ok(())).

fn poll_ready( self: Pin<&mut Self>, cx: &mut Context<'_> ) -> Poll<Result<(), Self::Error>>

Attempts to prepare the Sink to receive a value.

fn poll_flush( self: Pin<&mut Self>, cx: &mut Context<'_> ) -> Poll<Result<(), Self::Error>>

Flush any remaining output from this sink.

fn poll_close( self: Pin<&mut Self>, cx: &mut Context<'_> ) -> Poll<Result<(), Self::Error>>

Flush any remaining output and close this sink, if necessary.