Expand description
THE FILE FORMATS: bytes ⇄ RecordBatch for whole files — the format axis of the connector frame
(Format × Connector × Role), separate from where a file lives (a local directory, an object
store — the engine’s part store) and from what role it plays (a sink’s part, a
source’s input). Nothing here is hand-rolled: every format is the arrow-rs codec for it —
arrow::ipc (the Arrow IPC stream), parquet, arrow::csv, arrow::json — wrapped behind one
writer/reader shape so a sink or a source is written once and works for all of them.
A FormatWriter streams batches into a file the caller staged and hands the file back
finished (its footer / end marker written, its buffers flushed — not synced; the file sink’s
committer syncs it as part of the two-phase commit). A reader iterates a file’s batches.
Exactness: Arrow IPC and Parquet round-trip a batch exactly (schema and values). CSV and JSON lines are text formats — a reader infers their schema from the data, so types round-trip only as far as the text carries them (integers, floats, strings, booleans do; a timestamp comes back as text). They are here because a downstream tool wants them, not for our own round trip.
Formats work on whole batches and whole files: nothing here touches the per-row path.
Structs§
- Arrow
Ipc - The Arrow IPC stream format: the engine’s own representation, byte-for-byte; the fastest edge there is (no encode beyond a memcpy) for an Arrow-speaking peer.
- Csv
- CSV with a header row per file, through
arrow::csv. Text: the reader infers types. - Json
Lines - Newline-delimited JSON (one object per row), through
arrow::json. Text: the reader infers types.encoderswrites the columns it knows (an extension type) its own way; arrow-json writes the rest. - Parquet
- Parquet, through
parquet::arrow::ArrowWriter— the writer DataFusion’s own Parquet sink wraps. One row group per batch group the writer decides; the compression is a choice (snappyby default — the fast, universally read one;zstdfor smaller files;none).
Constants§
- NAMES
- Every format’s name, which is also its files’ extension.
Traits§
- File
Format - A file format: how batches become one file and back.
- Format
Writer - A writer for one file: batches in, the finished file out.
Functions§
- by_
extension - The format a file extension names (
parquetfor.parquet), if any. - by_name
- The format named
name, or the list of the ones there are.
Type Aliases§
- Batch
Iter - One batch iterator over a file.