datafusion_datasource::write::orchestration

Function spawn_writer_tasks_and_join

Source 
pub async fn spawn_writer_tasks_and_join(
    context: &Arc<TaskContext>,
    serializer: Arc<dyn BatchSerializer>,
    compression: FileCompressionType,
    object_store: Arc<dyn ObjectStore>,
    demux_task: SpawnedTask<Result<()>>,
    file_stream_rx: DemuxedStreamReceiver,
) -> Result<u64>
Expand description

Orchestrates multipart put of a dynamic number of output files from a single input stream for any statelessly serialized file type. That is, any file type for which each RecordBatch can be serialized independently of all other RecordBatchs.