Expand description
THE INTERNAL EXCHANGE (Phase 4): Arrow-IPC-carrying frames over kernel TCP on tokio, the transport a shuffle edge takes when its destination lives in another process. Decision D3 built to the letter (design/streams-v2-plan.md §1; investigation §7.1–7.3; review-followups §3):
- kernel TCP on tokio,
TCP_NODELAY— Flight, rkyv, io_uring and QUIC each rejected with evidence; the transport sits behind a trait so aSEND_ZCexperiment can swap in later (D6). - N flows per ordered worker pair, each with its own source port so the NIC’s RSS hash spreads them across queues and CPUs — the one thing that steps past the single-flow ceiling Flink, Arroyo and timely all sit under (§3.3).
- one owner task per socket direction — compute tasks never touch a socket; they push frames
into bounded per-channel queues, so the only real
sk_lockcontention (more than one writer per socket) never arises. - credit-based flow control in bytes per channel, granted by the receiver from its own consumption of that channel alone — never conditioned on another channel draining, which is what keeps a join diamond (Nexmark q8) deadlock-free.
The transport moves opaque Frames tagged with a ChannelId and a FrameKind; it routes on
the channel, acts on Credit itself, and delivers Data/Signal/Eos to the engine, which maps
them to and from its runtime Msg. That boundary keeps this crate free of the engine’s types.
Structs§
- Closed
- The far end went away.
- Frame
- One frame: a channel, a kind, and an opaque body.
- Grant
- A received frame’s credit, granted back to its sender when this is dropped (see
LinkReceiver::recv_held). Dropping never blocks, and works from any thread: the grant goes to the link’s own task, which sends it on. Credit is never lost, even for a frame dropped unread. - Link
- One worker-pair link — a
LinkSenderand aLinkReceiver, joined for convenience. Split it withLink::into_splitto send and receive from different tasks, which the sync-runtime bridge does (send is&self, recv is&mut self, so they cannot share one handle). - Link
Receiver - The receive half of a link: owns the delivered-frame stream, and grants credit back as frames are handed on. It holds the socket-owner task handles, so the flows live as long as frames are read.
- Link
Sender - The send half of a link: cloneable, so many producers send on their channels concurrently. The runtime’s outbound bridge holds one.
Enums§
- Frame
Error - A frame header the reader cannot accept: a wire-format version other than its own (a peer built
from another
fv-streamsversion), or akindbyte it does not know. - Frame
Kind - The transport-level category of a frame.
Data/Signal/Eosare delivered to the engine (and count against the channel’s credit);Creditis the transport’s own flow-control replenishment.
Constants§
- HEADER
- The 12-byte frame header:
body_len(4) · channel(4) · kind(1) · wire_version(1) · reserved(2). - WIRE_
VERSION - The wire-format version every frame carries (header byte 9). It names the whole frame contract —
this header, the Arrow IPC stream framing a
Databody uses, the postcardSignallayout, the one-byte reason anEosbody carries — so a peer on another version is refused at its first frame, never misread mid-stream. Bump it when any of that changes on purpose (the engine’s pinned golden-bytes test is what says it changed). A cluster’s handshake checks this same number, so it also covers the control messages between the leader and its workers. v2: anEossays why the stream ended. v3: the leader asks each worker whether its work is done.
Traits§
- Exchange
- The transport seam. Today the only implementor is the TCP
Link; the point of the trait is that the engine holdsimpl Exchangeand never learns which — so aSEND_ZC/ io_uring transport on a new kernel (D6, deferred behind exactly this trait) or an in-memory loopback for a test can take its place without an engine change. Deliberately notdyn-safe: the engine is generic over its transport, not boxing a heterogeneous set, so the seam stays zero-cost.
Functions§
- read_
loop - The read owner for one socket direction: decode frames from the wire and forward them to
inbox, in order. Ends cleanly on EOF (with no partial frame left over), or when theinboxreceiver drops. A junkkindbyte or a frame truncated by EOF is an error. - write_
loop - The write owner for one socket direction: drain
outboxand write each frame, in order. Ends (and shuts the write half, signalling EOF to the far reader) when every sender ofoutboxdrops.
Type Aliases§
- Channel
Id - A logical stream between two operator tasks (the engine’s
EdgeId). Many channels are multiplexed over the flows of one worker-pair link.