Struct ParquetSink 

pub struct ParquetSink { /* private fields */ }

Implements DataSink for writing to a parquet file.

Implementations

impl ParquetSink

pub fn new(config: FileSinkConfig, parquet_options: TableParquetOptions) -> Self

Create from config.

pub fn written(&self) -> HashMap<Path, FileMetaData>

Retrieve the file metadata for the written files, keyed to the path which may be partitioned (in the case of hive style partitioning).

pub fn parquet_options(&self) -> &TableParquetOptions

Parquet options

Trait Implementations

impl DataSink for ParquetSink

fn as_any(&self) -> &dyn Any

Returns the data sink as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> &SchemaRef

Returns the sink schema

fn write_all<'life0, 'life1, 'async_trait>( &'life0 self, data: SendableRecordBatchStream, context: &'life1 Arc<TaskContext>, ) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Writes the data to the sink, returns the number of values written

fn metrics(&self) -> Option<MetricsSet>

Return a snapshot of the MetricsSet for this DataSink.

impl Debug for ParquetSink

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DisplayAs for ParquetSink

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format according to DisplayFormatType, used when verbose representation looks different from the default one

impl FileSink for ParquetSink

fn config(&self) -> &FileSinkConfig

Retrieves the file sink configuration.

fn spawn_writer_tasks_and_join<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 Arc<TaskContext>, demux_task: SpawnedTask<Result<()>>, file_stream_rx: DemuxedStreamReceiver, object_store: Arc<dyn ObjectStore>, ) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Spawns writer tasks and joins them to perform file writing operations. Is a critical part of FileSink trait, since it’s the very last step for write_all.

fn write_all<'life0, 'life1, 'async_trait>( &'life0 self, data: Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>, context: &'life1 Arc<TaskContext>, ) -> Pin<Box<dyn Future<Output = Result<u64, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: Sync + 'async_trait,

File sink implementation of the DataSink::write_all method.