Skip to main content

decode_arrow_ipc_zero_copy

Function decode_arrow_ipc_zero_copy 

Source
pub fn decode_arrow_ipc_zero_copy(payload: IpcPayload) -> Result<DoraArray>
Expand description

Decode an Arrow IPC stream from an Arrow [Buffer] without copying the payload buffers when they are properly aligned.

Unlike decode_arrow_ipc, which reads from a byte slice through StreamReader (and therefore allocates a fresh buffer and copies every array buffer out of the stream), this uses arrow::ipc::reader::StreamDecoder, which slices the array buffers directly out of the provided [Buffer]. When the input buffer is suitably aligned — as Dora’s shared-memory payloads always are (128-byte AVec / page-aligned Zenoh SHM) — the decoded array aliases the input and no payload copy happens.

The decoder runs with the default require_alignment = false, so an under-aligned input (e.g. an arbitrary heap Vec) is handled gracefully by copying just the misaligned buffers rather than erroring. This keeps the receive path robust while preserving zero-copy for the common SHM case.

§Errors

Like decode_arrow_ipc, returns an error for an empty, truncated, or otherwise malformed stream, for a batch whose column count is not exactly one, and for any payload larger than 256 MB — it never panics on bad input.

§Example

use dora_node_api::IntoArrow;
use dora_node_api::arrow_utils::{IpcPayload, decode_arrow_ipc_zero_copy, encode_arrow_ipc};

let ipc = encode_arrow_ipc(&vec![1u64, 2, 3].into_arrow())?;
let decoded = decode_arrow_ipc_zero_copy(IpcPayload::from_vec(ipc))?;
let values: Vec<u64> = (&decoded).try_into()?;
assert_eq!(values, vec![1, 2, 3]);

// A malformed stream is rejected with an error rather than panicking.
assert!(decode_arrow_ipc_zero_copy(IpcPayload::from_vec(vec![0u8; 8])).is_err());