Module stream 

Streaming materialize: pull DF RecordBatch streams batch-by-batch, write, drop.

Peak retained memory ≈ in-flight batch + Parquet row-group encoder + unique tracker. Never holds a full Vec<RecordBatch> for the model result.

Structs

MaterializeWriteOptions

Options for stream / collect writers (derived from MaterializeConfig).

StreamWriteStats

Result of a successful stream materialize.

Functions

atomic_publish

Atomically replace dest with partial (same filesystem). Cleans partial on failure.

load_parquet_batches

Load small Parquet file into memory for optional MemTable ref() after stream write.

materialize_stream

Stream a DataFusion result into destination_path for the given format.

partial_path_for

Staging path for atomic publish: dir/.name.ext.rbt-partial.

write_empty_parquet

Empty schema-only Parquet (0 rows) so ref() registration has a file.

write_parquet_batches_atomic

Collect-mode helper: write batches with same Parquet props / atomic publish as stream.

write_parquet_stream

Stream write Parquet with atomic publish + optional streaming assertions.