Module ipc 

Zero-copy Arrow IPC stream read/write utilities.

Provides helpers for serializing and deserializing RecordBatches as self-delimiting Arrow IPC streams using synchronous Read/Write I/O.

These are designed for embedding IPC streams inside larger binary formats (e.g. a cache entry that contains multiple IPC sections). Each stream is self-delimiting (schema + batches + EOS marker) and can be read back independently.

Zero-copy reads

read_ipc_stream and read_ipc_stream_single take &Bytes and use Bytes::slice to produce each message buffer. Because Bytes::slice increments a reference count rather than copying, the resulting Buffers — and the array data decoded from them by FileDecoder — are all backed by the same allocation as the input.

Functions

read_ipc_stream

Read all RecordBatches from one Arrow IPC stream.

read_ipc_stream_at

Read all RecordBatches from one Arrow IPC stream starting at offset, advancing offset past the stream (including the EOS marker).

read_ipc_stream_single

Read exactly one RecordBatch from one Arrow IPC stream.

read_ipc_stream_single_at

Read exactly one RecordBatch from one Arrow IPC stream starting at offset, advancing offset past the stream (including the EOS marker).

read_len_prefixed_bytes

Read a byte slice written by write_len_prefixed_bytes.

read_len_prefixed_bytes_at

Read a length-prefixed byte slice at offset in data, advancing offset.

write_ipc_stream

Write batch as a single-batch Arrow IPC stream to writer.

write_ipc_stream_batches

Write all batches from iter as a single Arrow IPC stream to writer.

write_len_prefixed_bytes

Write data prefixed by its length as a little-endian u64.