Module ipc 

Zero-copy Arrow IPC stream read/write utilities.

Provides helpers for serializing and deserializing RecordBatches as self-delimiting Arrow IPC streams using synchronous Read/Write I/O.

These are designed for embedding IPC streams inside larger binary formats (e.g. a cache entry that contains multiple IPC sections). Each stream is self-delimiting (schema + batches + EOS marker) and can be read back independently.

Zero-copy reads

read_ipc_stream and read_ipc_stream_single take &Bytes and use Bytes::slice to produce each message buffer. Because Bytes::slice increments a reference count rather than copying, the resulting Buffers — and the array data decoded from them by FileDecoder — are all backed by the same allocation as the input.

Constants

IPC_SECTION_ALIGNMENT

Byte alignment that each IPC section’s stream start is padded to.

Functions

read_ipc_section_at

Read a single RecordBatch from an aligned IPC section at offset.

read_ipc_section_batches_at

Read all RecordBatches from an aligned multi-batch IPC section at offset, advancing offset past the section (padding + stream + EOS).

read_ipc_stream

Read all RecordBatches from one Arrow IPC stream.

read_ipc_stream_at

Read all RecordBatches from one Arrow IPC stream starting at offset, advancing offset past the stream (including the EOS marker).

read_ipc_stream_single

Read exactly one RecordBatch from one Arrow IPC stream.

read_ipc_stream_single_at

Read exactly one RecordBatch from one Arrow IPC stream starting at offset, advancing offset past the stream (including the EOS marker).

read_len_prefixed_bytes

Read a byte slice written by write_len_prefixed_bytes.

read_len_prefixed_bytes_at

Read a length-prefixed byte slice at offset in data, advancing offset.

write_ipc_section

Write batch as a 64-byte-aligned single-batch Arrow IPC section.

write_ipc_section_batches

Write batches as a single 64-byte-aligned multi-batch Arrow IPC section.

write_ipc_stream

Write batch as a single-batch Arrow IPC stream to writer.

write_ipc_stream_batches

Write all batches from iter as a single Arrow IPC stream to writer.

write_len_prefixed_bytes

Write data prefixed by its length as a little-endian u64.