Module ipc_stream 

Arrow IPC streams: the format Hugging Face datasets writes its cache in.

A stream is the IPC file format without the ARROW1 magic and the footer that says where each record batch is, so Polars cannot scan it. An open converts the stream, or every stream shard of a directory, once into one IPC file in the temp directory and scans that; IPC files among the shards are scanned where they are, not copied (Part). The conversion holds one record batch at a time: 0.2 GiB at its peak for a 3.0 GiB stream. Read eagerly instead, that stream held 3.6 GiB for as long as it was open, and the same rows with ZSTD buffers, 1.3 GiB on disk, the same 3.6 GiB.

Enums

Part

Where one input’s rows are read from once its streams are converted.

Functions

any_stream

Whether any of paths is a stream: asked once a scan of them as IPC files failed.

is_stream_file

Whether the file at path is an Arrow IPC stream, by its contents.

is_stream_head

Whether head, the first bytes of a file, begins an Arrow IPC stream: a schema message, after the continuation marker or, in a stream older than it, without.

starts_with_stream

Whether the Arrow paths are read by converting their streams: when the first is a stream. Only the first is opened. The conversion opens the rest, and leaves the IPC files among them where they are; a stream behind an IPC file is found by any_stream once the scan, which reads every IPC file’s footer, has failed on it.