Module write

Module containing helper methods/traits related to enabling write support for the various file formats

Modules

demux

Module containing helper methods/traits related to enabling dividing input stream into multiple output files at execution time

orchestration

Module containing helper methods/traits related to orchestrating file serialization, streaming to object store, parallelization, and abort handling

Structs

SharedBuffer

A buffer with interior mutability shared by the SerializedFileWriter and ObjectStore writer

Traits

BatchSerializer

A trait that defines the methods required for a RecordBatch serializer.

Functions

create_writer

Returns an AsyncWrite which writes to the given object store location with the specified compression. We drop the AbortableWrite struct and the writer will not try to cleanup on failure. Users can configure automatic cleanup with their cloud provider.

get_writer_schema

Converts table schema to writer schema, which may differ in the case of hive style partitioning where some columns are removed from the underlying files.