Skip to main content

Module batch

Module batch 

Source
Available on crate features transport-grpc and transport only.
Expand description

§Native batch transport wire mapper (Task 0.6)

Serde-less rustlib<->rustlib (DFE<->DFE) transfer of a whole WorkBatch over the existing gRPC mesh. One WorkBatch maps to one proto Batch and travels in a single RouteBatch RPC – batch-at-a-time, NOT record-by-record streaming.

§Payloads are OPAQUE in transit

Each Record payload is mapped straight onto the proto Record.payload bytes field and back. prost is configured (in build.rs, .bytes(".")) to decode that field ZERO-COPY into bytes::Bytes, so on the receive side the payload is a refcounted view of the decode buffer, never copied. The JSON / MsgPack codec (crate::transport::codec) is NEVER invoked here – the bytes pass through untouched, so a payload that is not valid JSON or MsgPack survives a round-trip byte-identical.

§Swappable wire

The wire is protobuf BY DEFAULT but the mapping is isolated to this module: a future hand-rolled frame could replace records_to_proto / proto_batch_to_records without touching the WorkBatch types or the transport seam.

§What does NOT cross the wire

commit_tokens and dlq_entries are deliberately left off the proto Batch. Commit tokens are the SENDER’s source acks – fired locally after the block is sent (at-least-once), they have no meaning on the receiver. Inline-DLQ entries are a local no-silent-drop concern. So the mapper takes / returns the records (&[Record] / Vec<Record>), not the whole WorkBatch<T>; this also avoids dragging the CommitToken generic onto the wire path.

Functions§

proto_batch_to_records
Map a proto Batch back onto a Vec<Record>.
records_to_proto
Map a slice of rustlib Records onto a proto Batch.