Module ipc_encode 

Hand-rolled, 1-copy Arrow IPC stream encoder.

Arrow’s official StreamWriter always stages the record-batch body in an internal Vec before writing it out, so encoding a message through it copies the payload at least twice. This module provides a fast path that writes the IPC flatbuffer headers and copies each array buffer directly into a caller-provided &mut [u8] — exactly one copy of the payload, straight into the (shared-memory) sample. For types the fast path does not handle it falls back to the official writer.

The fast-path output is a normal Arrow IPC stream and decodes through the official StreamDecoder — including the zero-copy decode_arrow_ipc_zero_copy receive path — because every body buffer is placed at a 64-byte-aligned offset, matching what the official writer produces with its default alignment = 64.

How the fast path stays correct with minimal slice handling

Arrow’s writer contains a lot of per-type code to truncate and rebase buffers for sliced arrays. We avoid most of it with two rules:

• Require offset() == 0 on every node — so logical element i lives at physical position i. Any array (or child) with a non-zero offset routes to the fallback. So do offset-carrying arrays (Binary, Utf8, List and their Large variants) whose first offset is not 0, which would need their offsets rebased.
• Copy each buffer’s prefix, never rewrite it. A buffer is copied from its start up to the bytes the node’s len elements use: len * width for fixed-width data, len + 1 offsets, and values up to the last offset. Slicing an offset-carrying array keeps offset() == 0 but leaves the parent’s whole values buffer (or List child) attached, so without that cut every sliced send would ship the entire parent. The only generated buffer is the all-ones validity bitmap for a node with no nulls, exactly as arrow emits.

Children are sliced before recursion to the length the parent implies: Struct fields to the struct’s len, a FixedSizeList child to len * value_size, and a List/LargeList child to its last offset.

Structs

InputDecoder

Per-input receive state for the schema-once zenoh path: one persistent StreamDecoder primed from the schema published on the output’s @schema subtopic (or in-band, from the schema block of a full self-describing stream on the data topic), then reused to decode the schema-less batch messages that flow on the data topic.

PreparedIpc

A prepared Arrow IPC fast-path encode.

PreparedUint8Ipc

A prepared no-null UInt8 IPC fast-path encode.

Functions

batch_fast_path_len

Exact byte length of the schema-less batch message encode_batch_into would write, or None if array is not fast-path eligible.

batch_slice

Given a full IPC stream, return the schema-less record-batch slice — everything after the schema block, including the trailing 8-byte end-of-stream marker. The marker is what lets the receiver’s persistent decoder flush a 0-row batch (whose body is empty); a non-empty batch never reads it. None if stream is malformed. See [decode_one_batch].

encode_batch_into

Encode array as a schema-less Arrow IPC batch message into dst. dst.len() must equal batch_fast_path_len(array).

encode_ipc_into

Encode array as a complete Arrow IPC stream directly into dst, copying each array buffer exactly once.

encode_ipc_to_vec

Fallback encoder for any array (including non-fast-path types): produce a full Arrow IPC stream Vec via the official writer. The caller copies this into the sample, so this path costs two payload copies.

encode_schema_message

Encode the IPC schema message for data_type as a framed, 64-byte-aligned block (one nullable field named data).

encode_uint8_ipc_header

Write a complete no-null UInt8 IPC stream into dst except the data region, and return the byte offset at which the caller must write the data_len data bytes (the buffer-protocol “construct in place” path).

ipc_fast_path_len

Exact byte length of the Arrow IPC stream encode_ipc_into would write for array, or None if array is not fast-path eligible.

schema_block_and_hash

Schema identity of a full IPC stream: the FNV-1a hash of its leading schema block, plus the block itself. This pairing IS the schema-once wire contract — the producer (publish_schema_once), the node receive path (in-band priming), and the daemon’s dora topic rebuild all derive the hash from exactly these bytes via this function; independent re-derivations could drift and silently break batch↔schema matching.

schema_block_len

Length of the leading schema-message block of a full IPC stream produced by encode_ipc_into (so a receiver can split it into the schema prefix and the record-batch+body), or None if stream is not a framed IPC message.

uint8_ipc_len

Total IPC stream length for a no-null UInt8 array of data_len elements. Lets a caller size the sample before constructing the message in place via encode_uint8_ipc_header.