Skip to main content

Crate fv_streams_formats

Crate fv_streams_formats 

Source
Expand description

THE FILE FORMATS: bytes ⇄ RecordBatch for whole files — the format axis of the connector frame (Format × Connector × Role), separate from where a file lives (a local directory, an object store — the engine’s part store) and from what role it plays (a sink’s part, a source’s input). Nothing here is hand-rolled: every format is the arrow-rs codec for it — arrow::ipc (the Arrow IPC stream), parquet, arrow::csv, arrow::json — wrapped behind one writer/reader shape so a sink or a source is written once and works for all of them.

A FormatWriter streams batches into a file the caller staged and hands the file back finished (its footer / end marker written, its buffers flushed — not synced; the file sink’s committer syncs it as part of the two-phase commit). A reader iterates a file’s batches.

Exactness: Arrow IPC and Parquet round-trip a batch exactly (schema and values). CSV and JSON lines are text formats — a reader infers their schema from the data, so types round-trip only as far as the text carries them (integers, floats, strings, booleans do; a timestamp comes back as text). They are here because a downstream tool wants them, not for our own round trip.

Formats work on whole batches and whole files: nothing here touches the per-row path.

Structs§

ArrowIpc
The Arrow IPC stream format: the engine’s own representation, byte-for-byte; the fastest edge there is (no encode beyond a memcpy) for an Arrow-speaking peer.
Csv
CSV with a header row per file, through arrow::csv. Text: the reader infers types.
JsonLines
Newline-delimited JSON (one object per row), through arrow::json. Text: the reader infers types. encoders writes the columns it knows (an extension type) its own way; arrow-json writes the rest.
Parquet
Parquet, through parquet::arrow::ArrowWriter — the writer DataFusion’s own Parquet sink wraps. One row group per batch group the writer decides; the compression is a choice (snappy by default — the fast, universally read one; zstd for smaller files; none).

Constants§

NAMES
Every format’s name, which is also its files’ extension.

Traits§

FileFormat
A file format: how batches become one file and back.
FormatWriter
A writer for one file: batches in, the finished file out.

Functions§

by_extension
The format a file extension names (parquet for .parquet), if any.
by_name
The format named name, or the list of the ones there are.

Type Aliases§

BatchIter
One batch iterator over a file.