Module ipc_stream 

Arrow IPC streams, the format Hugging Face datasets caches in: the IPC file format without the ARROW1 magic and footer, so Polars cannot scan it. An open converts the stream (or a directory’s stream shards) once into a temp IPC file and scans that; IPC files among the shards are scanned in place (Part). One record batch is held at a time: 0.2 GiB peak for a 3.0 GiB stream that read eagerly took 3.6 GiB.

Enums

Part

Where one input’s rows are read from once its streams are converted.

Functions

any_stream

Whether any of paths is a stream: asked once a scan of them as IPC files failed.

is_stream_file

Whether the file at path is an Arrow IPC stream, by its contents.

is_stream_head

Whether head begins an Arrow IPC stream: a schema message, with or (older streams) without the continuation marker. A whole message must be a schema message; a cut one counts if marked. Fields are not parsed: Polars panics on unimplemented column types, and this runs on any opened file.

starts_with_stream

Whether Arrow paths are converted: when the first is a stream (only it is opened). A stream behind an IPC file is found by any_stream once the scan fails on it.