Expand description
THE DATAFLOW RUNTIME of FusionVault Streams: a pipeline as tasks on OS threads connected by
bounded in-memory edges that carry Arrow batches and the two control signals — watermarks and
checkpoint barriers — in band; the coordinator’s epochs; the cross-worker exchange (bridges over
fv-streams-exchange links, Arrow IPC frames); placement of a graph over a member set; the
cluster handshake and the leader’s control channel. Sources, sinks and operators are the
contract crate’s traits: the runtime runs whatever it is handed and never knows a connector,
an operator’s algebra or a hash function (the shuffle’s dataflow::Hasher is the planner’s).
Modules§
- cluster
- THE MEMBERSHIP HANDSHAKE (D9 — zero control plane, the leader is a worker): a run’s workers rendezvous through the leader and open the exchange mesh. No scheduler, no etcd:
- dataflow
- THE DATAFLOW RUNTIME (Phase 2): a pipeline as tasks on OS threads connected by bounded in-memory edges that carry Arrow batches and the two control signals — watermarks and checkpoint barriers — in band (design/2026-09-15-phase2-dataflow-scoping.md).
- placement
- DETERMINISTIC PLACEMENT (D8 / D9): every worker computes the SAME task→worker map from the topology and the sorted worker set, so no worker is told its placement and there is no control plane that owns it (D9 — “zero control plane; the leader is a worker”). Two properties make that sound: