Module ipc_encode 

Hand-rolled, 1-copy Arrow IPC stream encoder.

Arrow’s official StreamWriter always stages the record-batch body in an internal Vec before writing it out, so encoding a message through it copies the payload at least twice. This module provides a fast path that writes the IPC flatbuffer headers and copies each array buffer directly into a caller-provided &mut [u8] — exactly one copy of the payload, straight into the (shared-memory) sample. For types the fast path does not handle it falls back to the official writer.

The fast-path output is a normal Arrow IPC stream and decodes through the official StreamDecoder — including the zero-copy decode_arrow_ipc_zero_copy receive path — because every body buffer is placed at a 64-byte-aligned offset, matching what the official writer produces with its default alignment = 64.

How the fast path stays correct without slice-truncation logic

Arrow’s writer contains a lot of per-type code to truncate buffers for sliced arrays. We sidestep all of it with two rules:

• Require offset() == 0 on every node — so logical element i lives at physical position i. Any array (or child) with a non-zero offset routes to the fallback.
• Copy each data buffer in full. Arrow tolerates buffers that are larger than strictly required for len elements, so copying the whole buffer (a freshly built array’s buffers are exactly sized anyway) always decodes to a logically-equal array. The only generated buffer is the all-ones validity bitmap for a node with no nulls, exactly as arrow emits.

Two types need their children sliced before recursion because the child’s IPC length is the parent’s rather than the child’s own: Struct (each field to the struct’s len) and FixedSizeList (its child to len * value_size). List/LargeList children are bounded by an offsets buffer and recursed at full length.

Structs

InputDecoder

Per-input receive state for the schema-once zenoh path: one persistent StreamDecoder primed from the schema published on the output’s @schema subtopic (or in-band, from the schema block of a full self-describing stream on the data topic), then reused to decode the schema-less batch messages that flow on the data topic.

Functions

batch_fast_path_len

Exact byte length of the schema-less batch message encode_batch_into would write (record-batch header block + body + 8-byte end-of-stream marker, no schema prefix), or None if array is not fast-path eligible.

batch_slice

Given a full IPC stream, return the schema-less record-batch slice — everything after the schema block, including the trailing 8-byte end-of-stream marker. The marker is what lets the receiver’s persistent decoder flush a 0-row batch (whose body is empty); a non-empty batch never reads it. None if stream is malformed. See [decode_one_batch].

encode_batch_into

Encode array as a schema-less Arrow IPC batch message into dst: the record-batch header block, the body, and a trailing 8-byte end-of-stream marker (no schema prefix). Decoded by a [StreamDecoder] already primed with the matching encode_schema_message. The trailing marker lets the decoder flush a 0-row batch (empty body); a non-empty batch never reads it. dst.len() must equal batch_fast_path_len(array).

encode_ipc_into

Encode array as a complete Arrow IPC stream directly into dst, copying each array buffer exactly once.

encode_ipc_to_vec

Fallback encoder for any array (including non-fast-path types): produce a full Arrow IPC stream Vec via the official writer. The caller copies this into the sample, so this path costs two payload copies.

encode_schema_message

Encode the IPC schema message for data_type as a framed, 64-byte-aligned block (one nullable field named data).

encode_uint8_ipc_header

Write a complete no-null UInt8 IPC stream into dst except the data region, and return the byte offset at which the caller must write the data_len data bytes (the buffer-protocol “construct in place” path).

ipc_fast_path_len

Exact byte length of the IPC stream encode_ipc_into would write, or None if array is not fast-path eligible (use encode_ipc_to_vec).

schema_block_and_hash

Schema identity of a full IPC stream: the FNV-1a hash of its leading schema block, plus the block itself. This pairing IS the schema-once wire contract — the producer (publish_schema_once), the node receive path (in-band priming), and the daemon’s dora topic rebuild all derive the hash from exactly these bytes via this function; independent re-derivations could drift and silently break batch↔schema matching.

schema_block_len

Length of the leading schema-message block of a full IPC stream produced by encode_ipc_into (so a receiver can split it into the schema prefix and the record-batch+body), or None if stream is not a framed IPC message.

uint8_ipc_len

Total IPC stream length for a no-null UInt8 array of data_len elements. Lets a caller size the sample before constructing the message in place via encode_uint8_ipc_header.