Skip to main content

astrs_node_api/session/
inputs.rs

1//! The slow-start route handshake, **consumer** side (blueprint §6.2, §6.3).
2//!
3//! [`super::routes`] is the producer's half: a `RouteUpgrade` arrives, a
4//! [`astrs_shm::Producer`] is opened, and every later publish writes into a
5//! ring slot. This module is the other half, and it is the one that makes the
6//! plane actually engage: a route is only upgraded once *"all static same-host
7//! consumers attached to the ring"* (§6.3), so until a consumer attaches by
8//! itself, nothing ever moves off the daemon path.
9//!
10//! ```text
11//!   InputRouteUpgrade{input, source, segment}
12//!            │
13//!            ▼
14//!   SegmentClient::attach ──► astrs_shm::Consumer ──► reader thread
15//!            │                                             │
16//!            │  doorbell registered with the broker        │ Payload::zero_copy
17//!            ▼                                             ▼
18//!   the daemon observes the pid in the consumer table   the node's input queue
19//!            │
20//!            ▼
21//!   RouteUpgrade to the producer — §6.3 completes
22//! ```
23//!
24//! # Why attaching early is safe
25//!
26//! The daemon sends `InputRouteUpgrade` *before* the producer is offered its
27//! own upgrade, because the producer's offer is gated on observing this
28//! consumer in the segment's consumer table. So a consumer always attaches
29//! while the producer is still publishing through the daemon — and the ring is
30//! therefore empty. [`astrs_shm::StartPosition::Latest`] puts the cursor at the
31//! *next* write, so the transition can neither duplicate a message the daemon
32//! already delivered nor skip the first one the ring carries.
33//!
34//! # What still arrives on the daemon path
35//!
36//! An upgraded route is not a ring *only*. §6.2 keeps the threshold rule — *"a
37//! heap payload ≥ threshold is copied once into a slot; below threshold it
38//! rides the UDS control channel"* — so a producer on the shared-memory plane
39//! still publishes its small messages through the daemon, and so does one
40//! whose ring was momentarily full (§6.2: *never sleep-retry*). Both arrive
41//! here as ordinary [`astrs_wire::NodeEvent::Input`] frames on the same input
42//! queue. A node sees one stream; [`crate::Payload::is_zero_copy`] is the only
43//! thing that differs.
44//!
45//! The consequence worth stating plainly: on an output that publishes payloads
46//! on *both* sides of the threshold, the two carriers have different latencies
47//! and the relative order of a large and a small message is not guaranteed.
48//! An output whose payloads are uniformly sized — the normal case, since a
49//! port carries one type — stays on one carrier and stays in order.
50//!
51//! # Falling back is a detach
52//!
53//! Every fatal condition ends the same way: the [`astrs_shm::Consumer`] is
54//! dropped, which releases its consumer-table entry, which the daemon's next
55//! plane pass observes as a detachment — and a detached consumer downgrades
56//! the producer, which resumes publishing through the daemon (§6.3). The node
57//! keeps reading the same event stream throughout. Nothing has to be
58//! negotiated for the fallback to be correct; it falls out of the handshake.
59//!
60//! # Holding a sample holds a slot, and that is fine
61//!
62//! A queued [`crate::Payload`] on the zero-copy plane pins the ring slot it
63//! points at, so a node that reads slowly holds slots its producer cannot
64//! refill. That sounds like a way to stall a producer, and it is worth being
65//! explicit about why it is not.
66//!
67//! Two bounds close it. The consumer's queue policy (§11.2) caps how many
68//! events can be outstanding at once — a `queue_size: 4` input pins at most
69//! four slots, whatever the node does — and the producer's reaction to a ring
70//! with no free slot is [`astrs_shm::Producer::try_allocate`] failing, which
71//! the node API turns into §6.2's documented fallback: publish inline, count
72//! `shm_fallback_total`, carry on. Nothing waits, so nothing stalls.
73//!
74//! An earlier version of this module copied payloads out of their slots once
75//! an input's queue passed a threshold, to "protect" the producer. It bought
76//! nothing the two bounds above do not already give, and it cost the thing the
77//! plane exists for: a consumer that fell briefly behind started copying every
78//! frame, which made it slower, which kept it behind. The copy is still
79//! available — [`crate::Payload::detach`] — to the node, which knows whether
80//! it is about to stash the payload and is the right place for the decision.
81
82use std::collections::HashMap;
83use std::path::Path;
84use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
85use std::sync::{Arc, Mutex, MutexGuard, Weak};
86use std::thread::JoinHandle;
87use std::time::Duration;
88
89use astrs_shm::{
90    AttachOptions, Consumer, RecvError, Sample, Segment, SegmentClient, SegmentKey, StartPosition,
91};
92use astrs_wire::{DataId, DataflowId, Metadata, PortRef, ShmSegmentSpec, WireDecode};
93
94use crate::error::{NodeError, Result};
95use crate::events::{Event, QueuedEvent};
96use crate::payload::Payload;
97use crate::session::SessionShared;
98use crate::session::routes::RoutePlane;
99
100/// How long one blocking receive parks on the doorbell before the reader
101/// looks at its stop flag again.
102///
103/// It bounds how long a detach takes to take effect, not how long a message
104/// waits: the doorbell wakes the reader the moment the producer commits.
105pub const RECV_SLICE: Duration = Duration::from_millis(25);
106
107/// How many consecutive [`RecvError::Lagged`] reports end the attachment.
108///
109/// A lag means the producer overwrote messages this consumer had not read —
110/// possible only under [`astrs_shm::OverflowPolicy::Overwrite`], since the
111/// default blocks instead. One is a hiccup and the cursor recovers by itself.
112/// This many in a row is a consumer that cannot keep up with the ring at all,
113/// and the honest answer is to give the route back to the daemon, whose queue
114/// policy (§11.2) is the node's *declared* answer to being behind.
115pub const MAX_CONSECUTIVE_LAGS: u32 = 8;
116
117/// Counters describing what the consumer-side plane has done.
118#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
119#[non_exhaustive]
120pub struct InputPlaneStats {
121    /// Inputs successfully attached to a ring.
122    pub attaches: u64,
123    /// Attachments that ended, for any reason.
124    pub detaches: u64,
125    /// Upgrades that could not be taken up, so the route stayed on the daemon
126    /// path (§6.2: never sleep-retry).
127    pub refusals: u64,
128    /// Samples delivered from a ring.
129    pub samples: u64,
130    /// Messages the producer overwrote before this consumer read them.
131    pub lagged: u64,
132}
133
134/// One input's place on the plane.
135#[derive(Debug)]
136struct InputRoute {
137    /// Which plane it reads from right now.
138    plane: RoutePlane,
139    /// The segment it is attached to, when it is.
140    segment: Option<ShmSegmentSpec>,
141    /// The live reader, if any.
142    reader: Option<ReaderHandle>,
143    /// Readers that were asked to stop and have not been joined yet.
144    ///
145    /// Joining is deferred rather than skipped: a reader must be *gone* before
146    /// another attaches to the same segment, or two cursors would deliver the
147    /// same message twice. Doing it here rather than in [`InputRouteTable::detach`]
148    /// keeps the session's frame loop off a thread join.
149    retiring: Vec<ReaderHandle>,
150    /// Which incarnation of this input's reader is current.
151    ///
152    /// A reader that exits stamps the table only if it is still the current
153    /// one, so a slow exit cannot clobber a re-attachment that overtook it.
154    epoch: u64,
155}
156
157impl InputRoute {
158    /// A route on the daemon path with no reader.
159    const fn daemon() -> Self {
160        Self {
161            plane: RoutePlane::Daemon,
162            segment: None,
163            reader: None,
164            retiring: Vec::new(),
165            epoch: 0,
166        }
167    }
168}
169
170/// What happened to a closure offered to a reader.
171///
172/// Three outcomes, not two, and the third is the one that matters: a daemon
173/// closes a source from more than one place — the producer's `OutputDone` and
174/// its session ending — so the same `InputClosed` can arrive twice. Treating
175/// the duplicate like an ordinary "the reader would not take it" and queueing
176/// it directly puts a closure *in front of* the frames the first one is still
177/// waiting behind, which is precisely the loss this hand-off exists to
178/// prevent. A duplicate carries no information, so it is discarded.
179#[derive(Debug)]
180enum ClosureOutcome {
181    /// The reader has it and will push it once it has drained.
182    Taken,
183    /// A closure for this input is already queued for delivery; this one is a
184    /// duplicate and is dropped.
185    AlreadyPending,
186    /// The reader has already exited, so its drain is complete: the caller
187    /// should queue the closure itself.
188    Finished(QueuedEvent),
189}
190
191/// The hand-off of an input's closure to the reader that must deliver it last.
192///
193/// One mutex covers both fields on purpose. The reader marks itself finished
194/// and takes any pending closure in the same critical section, so a caller
195/// either wins the race and hands the event over, or loses it and is told to
196/// deliver the event itself — never a window where both believe the other
197/// will, which would lose a closure and hang a node forever.
198#[derive(Debug, Default)]
199struct ClosureSlot {
200    /// The event to push once the drain finishes.
201    pending: Option<QueuedEvent>,
202    /// Set by the reader as it exits, under this same lock.
203    finished: bool,
204}
205
206/// A reader thread and the two ways to end it.
207///
208/// The distinction is the last message of a stream. A producer that finishes
209/// commits its final frame into the ring and *then* tells the daemon, which
210/// tells this consumer its input is closed — over the control connection,
211/// which can easily overtake a sample still travelling on the ring. A reader
212/// that stopped on the closure would drop that frame.
213#[derive(Debug)]
214struct ReaderHandle {
215    /// Set to end the reader without reading anything further.
216    stop: Arc<AtomicBool>,
217    /// Set to make the reader drain what is resident and *then* end.
218    drain: Arc<AtomicBool>,
219    /// The input's closure, delivered behind the messages it closes rather
220    /// than in front of them.
221    closing: Arc<Mutex<ClosureSlot>>,
222    /// The thread itself.
223    join: JoinHandle<()>,
224}
225
226impl ReaderHandle {
227    /// Asks the reader to stop at once, without waiting for it.
228    ///
229    /// For a ring that is gone or replaced: there is nothing left to drain,
230    /// and reading a segment the daemon has reclaimed is the one thing §6.2's
231    /// generation stamp exists to prevent.
232    fn abort(&self) {
233        self.stop.store(true, Ordering::Release);
234    }
235
236    /// Asks the reader to drain what is resident and then stop.
237    fn signal_drain(&self) {
238        self.drain.store(true, Ordering::Release);
239    }
240
241    /// Hands the reader the event to push once it has drained.
242    fn take_closure(&self, event: QueuedEvent) -> ClosureOutcome {
243        let mut slot = lock(&self.closing);
244        if slot.finished {
245            return ClosureOutcome::Finished(event);
246        }
247        if slot.pending.is_some() {
248            return ClosureOutcome::AlreadyPending;
249        }
250        slot.pending = Some(event);
251        self.signal_drain();
252        ClosureOutcome::Taken
253    }
254
255    /// Waits for the reader to exit, which is also when its
256    /// [`astrs_shm::Consumer`] is dropped and its consumer-table entry freed.
257    fn join(self) {
258        self.abort();
259        // A reader that panicked has still released its entry, because the
260        // `Consumer` is dropped by the unwind. There is nothing to report and
261        // nothing to retry.
262        let _ = self.join.join();
263    }
264}
265
266/// Every input's plane state, and the reader threads behind the attached ones.
267#[derive(Debug, Default)]
268pub struct InputRouteTable {
269    /// One entry per input the daemon has said anything about.
270    routes: Mutex<HashMap<DataId, InputRoute>>,
271    /// Counters.
272    attaches: AtomicU64,
273    detaches: AtomicU64,
274    refusals: AtomicU64,
275    samples: AtomicU64,
276    lagged: AtomicU64,
277}
278
279impl InputRouteTable {
280    /// An empty table: every input on the daemon path.
281    #[must_use]
282    pub fn new() -> Self {
283        Self {
284            routes: Mutex::new(HashMap::new()),
285            attaches: AtomicU64::new(0),
286            detaches: AtomicU64::new(0),
287            refusals: AtomicU64::new(0),
288            samples: AtomicU64::new(0),
289            lagged: AtomicU64::new(0),
290        }
291    }
292
293    /// The plane one input reads from.
294    #[must_use]
295    pub fn plane(&self, input: &DataId) -> RoutePlane {
296        lock(&self.routes)
297            .get(input)
298            .map_or(RoutePlane::Daemon, |route| route.plane)
299    }
300
301    /// The plane every input the daemon has spoken about reads from, sorted.
302    #[must_use]
303    pub fn planes(&self) -> Vec<(DataId, RoutePlane)> {
304        let mut planes: Vec<(DataId, RoutePlane)> = lock(&self.routes)
305            .iter()
306            .map(|(id, route)| (id.clone(), route.plane))
307            .collect();
308        planes.sort_by(|left, right| left.0.cmp(&right.0));
309        planes
310    }
311
312    /// The segment one input is attached to.
313    #[must_use]
314    pub fn segment(&self, input: &DataId) -> Option<ShmSegmentSpec> {
315        lock(&self.routes)
316            .get(input)
317            .and_then(|route| route.segment.clone())
318    }
319
320    /// How many inputs read from a ring right now.
321    #[must_use]
322    pub fn attached_count(&self) -> usize {
323        lock(&self.routes)
324            .values()
325            .filter(|route| route.plane.is_zero_copy())
326            .count()
327    }
328
329    /// A snapshot of the counters.
330    #[must_use]
331    pub fn stats(&self) -> InputPlaneStats {
332        InputPlaneStats {
333            attaches: self.attaches.load(Ordering::Relaxed),
334            detaches: self.detaches.load(Ordering::Relaxed),
335            refusals: self.refusals.load(Ordering::Relaxed),
336            samples: self.samples.load(Ordering::Relaxed),
337            lagged: self.lagged.load(Ordering::Relaxed),
338        }
339    }
340
341    /// Records one upgrade this node could not take up.
342    pub fn count_refusal(&self) {
343        self.refusals.fetch_add(1, Ordering::Relaxed);
344    }
345
346    /// Attaches `input` to `segment` and starts reading it (§6.3).
347    ///
348    /// Idempotent for the generation already attached: a repeated offer for
349    /// the same incarnation is answered by leaving the working reader alone. A
350    /// *different* generation replaces it, which is the producer-restart case
351    /// — the old ring is gone and its reader has nothing left to read.
352    ///
353    /// # Errors
354    ///
355    /// [`NodeError::Shm`] when the segment cannot be opened, verified or read,
356    /// and [`NodeError::Pattern`] when the daemon's segment name does not
357    /// describe the producer the event names. Either leaves the input on the
358    /// daemon path, which is the honest outcome (§6.2: never sleep-retry).
359    pub fn attach(
360        &self,
361        shared: &Arc<SessionShared>,
362        input: &DataId,
363        source: &PortRef,
364        segment: &ShmSegmentSpec,
365    ) -> Result<()> {
366        // A repeat of what is already running costs nothing and changes
367        // nothing. Checked before the segment is opened, because opening is
368        // the expensive half.
369        let retiring = {
370            let mut routes = lock(&self.routes);
371            let route = routes
372                .entry(input.clone())
373                .or_insert_with(InputRoute::daemon);
374            if route.plane.is_zero_copy()
375                && route.segment.as_ref().is_some_and(|open| {
376                    open.generation == segment.generation && open.name == segment.name
377                })
378            {
379                return Ok(());
380            }
381            // Anything already running for this input is superseded, and its
382            // ring is a previous incarnation's: aborted, not drained.
383            if let Some(reader) = route.reader.take() {
384                reader.abort();
385                route.retiring.push(reader);
386            }
387            route.plane = RoutePlane::Daemon;
388            route.segment = None;
389            core::mem::take(&mut route.retiring)
390        };
391        // Joined outside the lock: an exiting reader takes it to stamp itself
392        // detached, and joining while holding it would deadlock.
393        for reader in retiring {
394            reader.join();
395        }
396
397        let (mapped, consumer) = open_consumer(
398            shared.dataflow,
399            source,
400            segment,
401            shared.shm_broker.as_deref(),
402        )?;
403        let stop = Arc::new(AtomicBool::new(false));
404        let drain = Arc::new(AtomicBool::new(false));
405        let closing: Arc<Mutex<ClosureSlot>> = Arc::new(Mutex::new(ClosureSlot::default()));
406
407        let epoch = {
408            let mut routes = lock(&self.routes);
409            let route = routes
410                .entry(input.clone())
411                .or_insert_with(InputRoute::daemon);
412            route.epoch = route.epoch.wrapping_add(1);
413            route.epoch
414        };
415
416        let reader = ReaderContext {
417            input: input.clone(),
418            source: source.clone(),
419            session: Arc::downgrade(shared),
420            stop: Arc::clone(&stop),
421            drain: Arc::clone(&drain),
422            closing: Arc::clone(&closing),
423            epoch,
424        };
425        let join = std::thread::Builder::new()
426            .name(format!("astrs-shm-{input}"))
427            .spawn(move || reader.run(consumer))
428            .map_err(|error| {
429                NodeError::Pattern(format!(
430                    "could not start the ring reader for `{input}`: {error}"
431                ))
432            })?;
433
434        let mut routes = lock(&self.routes);
435        let route = routes
436            .entry(input.clone())
437            .or_insert_with(InputRoute::daemon);
438        route.plane = RoutePlane::Shm;
439        route.segment = Some(segment.clone());
440        route.reader = Some(ReaderHandle {
441            stop,
442            drain,
443            closing,
444            join,
445        });
446        // The mapping itself is kept alive by the `Consumer` the reader owns;
447        // this handle has done its job.
448        drop(mapped);
449        self.attaches.fetch_add(1, Ordering::Relaxed);
450        Ok(())
451    }
452
453    /// Takes `input` off the ring, if it was on one.
454    ///
455    /// Returns whether anything was attached. The reader **drains what is
456    /// resident** before it exits — §6.2's rule for a producer that goes away
457    /// is that the daemon *"marks `closed`, lets consumers drain, and unlinks"*,
458    /// and a consumer that stopped reading the instant it was told to detach
459    /// would throw away messages the producer had already committed.
460    ///
461    /// The input is on the daemon path from this instant, so anything the
462    /// daemon sends next is delivered inline and nothing arrives twice: the
463    /// ring holds only what was written before the detach.
464    pub fn detach(&self, input: &DataId) -> bool {
465        let mut routes = lock(&self.routes);
466        let Some(route) = routes.get_mut(input) else {
467            return false;
468        };
469        let was_attached = route.plane.is_zero_copy();
470        route.plane = RoutePlane::Daemon;
471        route.segment = None;
472        route.epoch = route.epoch.wrapping_add(1);
473        if let Some(reader) = route.reader.take() {
474            reader.signal_drain();
475            route.retiring.push(reader);
476        }
477        if was_attached {
478            self.detaches.fetch_add(1, Ordering::Relaxed);
479        }
480        was_attached
481    }
482
483    /// Delivers `input`'s closure *behind* whatever its ring still holds.
484    ///
485    /// Returns [`None`] when a reader took the event, and gives it back —
486    /// never drops it — when the input is not on a ring and the caller should
487    /// queue it itself.
488    ///
489    /// # Why the closure cannot simply be queued
490    ///
491    /// A producer that finishes commits its last frame into the ring and only
492    /// then tells the daemon, which tells this consumer. That message travels
493    /// on the control connection, which routinely overtakes a sample still
494    /// being read from the ring — so queueing the closure directly would put
495    /// "this input is finished" in front of the message that finished it, and
496    /// a node that stops reading on the closure (every node) would lose the
497    /// last frame of every stream.
498    ///
499    /// Handing it to the reader instead makes the ordering structural: the
500    /// reader drains, pushes the samples, pushes the closure, and exits.
501    pub fn close_input(&self, input: &DataId, event: QueuedEvent) -> Option<QueuedEvent> {
502        let mut routes = lock(&self.routes);
503        let Some(route) = routes.get_mut(input) else {
504            return Some(event);
505        };
506        if !route.plane.is_zero_copy() {
507            // Already off the ring — but perhaps only just, with a reader
508            // still draining what the producer committed before it went.
509            return Self::hand_to_retiring(route, event);
510        }
511        let Some(reader) = route.reader.take() else {
512            return Some(event);
513        };
514        let outcome = reader.take_closure(event);
515        route.plane = RoutePlane::Daemon;
516        route.segment = None;
517        route.epoch = route.epoch.wrapping_add(1);
518        route.retiring.push(reader);
519        self.detaches.fetch_add(1, Ordering::Relaxed);
520        match outcome {
521            ClosureOutcome::Taken | ClosureOutcome::AlreadyPending => None,
522            ClosureOutcome::Finished(event) => Some(event),
523        }
524    }
525
526    /// Hands `input`'s closure to a reader that is still draining, if there is
527    /// one.
528    ///
529    /// The ordering this exists for: a producer that dies makes the daemon
530    /// send *both* an `InputRouteDowngrade` (the ring is going away) and an
531    /// `InputClosed` (the stream ended), and the downgrade arrives first — so
532    /// by the time the closure gets here the input is already off the ring and
533    /// its reader is in [`ReaderHandle`]'s retiring list, draining the last
534    /// frames the producer committed. Queueing the closure directly would put
535    /// it in front of them.
536    ///
537    /// Returns [`None`] when a draining reader took it.
538    fn hand_to_retiring(route: &mut InputRoute, event: QueuedEvent) -> Option<QueuedEvent> {
539        let mut event = event;
540        // Most recent first: an older reader is draining an older ring, and
541        // this closure belongs behind the newest thing that will be delivered.
542        for reader in route.retiring.iter().rev() {
543            match reader.take_closure(event) {
544                ClosureOutcome::Taken | ClosureOutcome::AlreadyPending => return None,
545                ClosureOutcome::Finished(returned) => event = returned,
546            }
547        }
548        Some(event)
549    }
550
551    /// Finishes every ring reader, so nothing is still in flight.
552    ///
553    /// The [`InputRouteTable::close_input`] hand-off orders one input's
554    /// closure behind its own ring. [`astrs_wire::NodeEvent::AllInputsClosed`]
555    /// has no input to hang off: the daemon sends it straight to the session,
556    /// so it arrives on the node's *control* lane, which is served before any
557    /// data — and it is terminal
558    /// ([`astrs_wire::NodeEvent::is_terminal`]), so a node that winds down on
559    /// it (`astrs-record-node` and `examples/rust-pipeline`'s recorder both do)
560    /// would drop whatever a ring was still draining.
561    ///
562    /// So the session drains first and pushes it second. This *does* wait —
563    /// bounded by [`RECV_SLICE`] per reader — which is acceptable here and
564    /// nowhere else: every input has just been declared finished, so there is
565    /// nothing left for the session to be responsive to.
566    ///
567    /// Returns how many readers were waited for.
568    pub fn drain_all(&self) -> usize {
569        let readers: Vec<ReaderHandle> = {
570            let mut routes = lock(&self.routes);
571            let mut readers = Vec::new();
572            for route in routes.values_mut() {
573                if route.plane.is_zero_copy() {
574                    self.detaches.fetch_add(1, Ordering::Relaxed);
575                }
576                route.plane = RoutePlane::Daemon;
577                route.segment = None;
578                route.epoch = route.epoch.wrapping_add(1);
579                if let Some(reader) = route.reader.take() {
580                    readers.push(reader);
581                }
582                readers.append(&mut route.retiring);
583            }
584            for reader in &readers {
585                reader.signal_drain();
586            }
587            readers
588        };
589        let waited = readers.len();
590        // Joined outside the lock: an exiting reader takes it to stamp itself
591        // detached, and joining while holding it would deadlock. `join` also
592        // sets the abort flag, which a reader that has finished draining has
593        // already passed — it ends the wait for one that is parked with an
594        // empty ring rather than cutting a drain short.
595        for reader in readers {
596            reader.join();
597        }
598        waited
599    }
600
601    /// Detaches every input and waits for every reader to exit.
602    ///
603    /// Called when the session ends. Waiting matters: a reader still holding a
604    /// [`astrs_shm::Sample`] holds a ring slot, and a process that exits
605    /// without releasing its consumer-table entry leaves the daemon to reclaim
606    /// it on a timeout instead of immediately.
607    pub fn close(&self) {
608        let readers: Vec<ReaderHandle> = {
609            let mut routes = lock(&self.routes);
610            let mut readers = Vec::new();
611            for route in routes.values_mut() {
612                if route.plane.is_zero_copy() {
613                    self.detaches.fetch_add(1, Ordering::Relaxed);
614                }
615                route.plane = RoutePlane::Daemon;
616                route.segment = None;
617                route.epoch = route.epoch.wrapping_add(1);
618                if let Some(reader) = route.reader.take() {
619                    readers.push(reader);
620                }
621                readers.append(&mut route.retiring);
622            }
623            for reader in &readers {
624                reader.abort();
625            }
626            readers
627        };
628        for reader in readers {
629            reader.join();
630        }
631    }
632
633    /// Marks one input back on the daemon path from its own reader thread.
634    ///
635    /// Ignored when `epoch` is not the current one: a reader that took a while
636    /// to notice its stop flag must not undo the attachment that replaced it.
637    fn mark_detached(&self, input: &DataId, epoch: u64) {
638        let mut routes = lock(&self.routes);
639        let Some(route) = routes.get_mut(input) else {
640            return;
641        };
642        if route.epoch != epoch {
643            return;
644        }
645        if route.plane.is_zero_copy() {
646            self.detaches.fetch_add(1, Ordering::Relaxed);
647        }
648        route.plane = RoutePlane::Daemon;
649        route.segment = None;
650    }
651
652    /// Records one sample delivered from a ring.
653    fn count_sample(&self) {
654        self.samples.fetch_add(1, Ordering::Relaxed);
655    }
656
657    /// Records messages the producer overwrote before they were read.
658    fn count_lagged(&self, missed: u64) {
659        self.lagged.fetch_add(missed, Ordering::Relaxed);
660    }
661}
662
663/// Everything one reader thread needs, so the thread body is a method and not
664/// a seven-argument function.
665struct ReaderContext {
666    /// The input being read.
667    input: DataId,
668    /// The producer port behind it, which every delivered event names.
669    source: PortRef,
670    /// The session to deliver into.
671    ///
672    /// Weak because the table this reader is registered in lives *inside* the
673    /// session: a strong reference would keep it alive forever.
674    session: Weak<SessionShared>,
675    /// Set when the reader should end at once, reading nothing further.
676    stop: Arc<AtomicBool>,
677    /// Set when the reader should drain what is resident and then end.
678    drain: Arc<AtomicBool>,
679    /// The input's closure, to push once the drain finishes.
680    closing: Arc<Mutex<ClosureSlot>>,
681    /// Which incarnation of this input's reader this is.
682    epoch: u64,
683}
684
685impl ReaderContext {
686    /// Reads until told to stop, the session ends, or the ring does.
687    fn run(self, mut consumer: Consumer) {
688        let mut consecutive_lags = 0u32;
689        while !self.stop.load(Ordering::Acquire) {
690            let Some(session) = self.session.upgrade() else {
691                break;
692            };
693            if session.is_closed() {
694                break;
695            }
696            if self.drain.load(Ordering::Acquire) {
697                self.finish(&session, &mut consumer);
698                break;
699            }
700            match consumer.next_blocking(RECV_SLICE) {
701                Ok(sample) => {
702                    consecutive_lags = 0;
703                    self.deliver(&session, sample);
704                }
705                Err(RecvError::Empty) => {}
706                Err(RecvError::Lagged(missed)) => {
707                    session.inputs.count_lagged(missed);
708                    consecutive_lags += 1;
709                    if consecutive_lags >= MAX_CONSECUTIVE_LAGS {
710                        session.source.push_control(Event::Error(format!(
711                            "input `{}` fell behind its ring {MAX_CONSECUTIVE_LAGS} times in a \
712                             row; reading it through the daemon instead",
713                            self.input
714                        )));
715                        break;
716                    }
717                }
718                // The producer finished, or its segment was reclaimed: both
719                // are the daemon's business to report, and it does — as
720                // `InputClosed` or `InputRouteDowngrade` on the control
721                // connection. Ending quietly here avoids saying it twice.
722                Err(RecvError::Closed) => break,
723                // `RecvError::Shm` and — since the enum is `#[non_exhaustive]`
724                // — any outcome a future `astrs-shm` adds: a condition this
725                // build cannot classify is treated as fatal for the
726                // attachment, which costs the plane and keeps the data.
727                Err(error) => {
728                    session.source.push_control(Event::Error(format!(
729                        "input `{}` left the shared-memory plane: {error}",
730                        self.input
731                    )));
732                    break;
733                }
734            }
735        }
736
737        // Dropping the consumer releases its consumer-table entry, which is
738        // what the daemon observes as a detachment — and a detached consumer
739        // downgrades the producer back onto the daemon path (§6.3). The
740        // fallback needs no message of its own.
741        drop(consumer);
742        // Marked finished and drained of any pending closure in one critical
743        // section, so a `close_input` racing this exit is told to deliver the
744        // closure itself rather than handing it to a thread that is gone.
745        let closure = {
746            let mut slot = lock(&self.closing);
747            slot.finished = true;
748            slot.pending.take()
749        };
750        if let Some(session) = self.session.upgrade() {
751            if let Some(closure) = closure {
752                session.source.push_input(&self.input, closure);
753            }
754            session.inputs.mark_detached(&self.input, self.epoch);
755        }
756    }
757
758    /// Delivers everything still resident in the ring, then the closure.
759    ///
760    /// Bounded by construction: a ring holds at most `slot_count` messages and
761    /// nothing is being written into it any more, so the loop ends at the
762    /// first `Empty`. A lag report is not a reason to stop draining — the
763    /// messages behind the gap are still there and still wanted.
764    ///
765    /// The closure itself is pushed by [`ReaderContext::run`]'s exit path,
766    /// which is where the hand-off is settled against a concurrent
767    /// [`InputRouteTable::close_input`].
768    fn finish(&self, session: &Arc<SessionShared>, consumer: &mut Consumer) {
769        loop {
770            match consumer.try_next() {
771                Ok(sample) => self.deliver(session, sample),
772                Err(RecvError::Lagged(missed)) => session.inputs.count_lagged(missed),
773                // `Empty`, `Closed`, or a fault: in every case the ring has
774                // nothing more to give this consumer.
775                Err(_) => break,
776            }
777        }
778    }
779
780    /// Turns one sample into a queued input event.
781    fn deliver(&self, session: &Arc<SessionShared>, sample: Sample) {
782        let metadata = slot_metadata(sample.metadata(), session);
783        // Fold the producer's clock reading into ours, exactly as the daemon
784        // path does, so causal order survives the hop (§4.3).
785        let _accepted = session.observe_remote(metadata.timestamp);
786
787        // Queued as it lies: the slot stays pinned until the node drops the
788        // payload, which is what makes reading it free. See the module
789        // documentation for why that cannot stall the producer.
790        let payload = Payload::zero_copy(sample);
791        session.source.push_input(
792            &self.input,
793            QueuedEvent::Input {
794                source: self.source.clone(),
795                metadata,
796                payload,
797            },
798        );
799        session.inputs.count_sample();
800    }
801}
802
803/// The metadata a slot carried, or a fresh stamp when it carried none.
804///
805/// The producer commits its [`Metadata`] beside the payload (§6.1, §6.2) and it
806/// is not replaceable: `request_id`, `goal_id`, `seq` and `_schema_hash` are
807/// what make a service exchange correlate (§9.4) and a schema drift detectable.
808/// Re-stamping would deliver the bytes and lose the message — so it happens
809/// only when there is nothing to lose.
810fn slot_metadata(bytes: &[u8], session: &Arc<SessionShared>) -> Metadata {
811    if bytes.is_empty() {
812        return Metadata::new(session.hlc_now());
813    }
814    Metadata::decode_exact(bytes).unwrap_or_else(|_| Metadata::new(session.hlc_now()))
815}
816
817/// Opens the segment an
818/// [`InputRouteUpgrade`](astrs_wire::NodeEvent::InputRouteUpgrade) names and
819/// attaches a reader to it.
820///
821/// The mirror of [`crate::session::routes::open_producer`], and it checks the
822/// same two things: the name the daemon sent must be the one the *producer's*
823/// `(dataflow, node, output, generation)` describes, and the mapped header's
824/// digest and generation must match that key. A stale ring from a previous
825/// incarnation therefore cannot be read, which is §6.2's crash-safety property
826/// from the consumer's side.
827///
828/// # The doorbell
829///
830/// A consumer in a different process from its producer cannot be woken by the
831/// producer directly — the two share a mapping, not a descriptor table. The
832/// broker is the go-between: [`astrs_shm::SegmentClient::register_doorbell`]
833/// passes this consumer's ringer over the same Unix socket that carried the
834/// segment, and the producer picks it up from the shared registry on its next
835/// commit. Without it a receive still works, but only by waiting out
836/// `astrs-shm`'s growing fallback slice — correct, and needlessly slow.
837///
838/// # Errors
839///
840/// [`NodeError::Shm`] when the segment cannot be opened, verified or attached
841/// to, and [`NodeError::Pattern`] when the daemon's name does not describe the
842/// producer port the event names.
843pub fn open_consumer(
844    dataflow: DataflowId,
845    source: &PortRef,
846    segment: &ShmSegmentSpec,
847    broker: Option<&Path>,
848) -> Result<(Arc<Segment>, Consumer)> {
849    let key = SegmentKey::new(
850        dataflow,
851        source.node().clone(),
852        source.port().clone(),
853        segment.generation,
854    );
855    let canonical = key.canonical();
856    if canonical != segment.name {
857        return Err(NodeError::Pattern(format!(
858            "input route upgrade names segment `{}`, but `{source}` at generation {} is \
859             `{canonical}`",
860            segment.name, segment.generation,
861        )));
862    }
863
864    let mut client = match broker {
865        Some(path) => Some(SegmentClient::connect(path)?),
866        None => None,
867    };
868    let mapped = match client.as_mut() {
869        Some(client) => client.attach(&key)?,
870        // No broker: the in-process harness and any embedder that disabled the
871        // plane socket. Only a named backing can be opened this way, which is
872        // exactly what those two use.
873        None => Segment::open_named(&key.os_name(), Some(&key))?,
874    }
875    .shared();
876
877    let consumer = Consumer::attach(
878        Arc::clone(&mapped),
879        AttachOptions::new()
880            // The next write, not the oldest resident one: everything already
881            // in the ring — if anything is — was published before this route
882            // moved, and the daemon delivered it inline.
883            .with_start(StartPosition::Latest)
884            .with_doorbell(true)
885            .expecting(key.clone()),
886    )?;
887
888    if let Some(client) = client.as_mut()
889        && let Some(doorbell) = consumer.doorbell()
890    {
891        let ringer = doorbell.ringer()?;
892        // A broker that will not take the ringer costs latency, never
893        // correctness: `next_blocking` falls back to a bounded wait slice.
894        if let Err(error) =
895            client.register_doorbell(&key, consumer.index(), consumer.token(), ringer)
896        {
897            tracing::debug!(%error, segment = %canonical, "the broker refused a doorbell");
898        }
899    }
900
901    Ok((mapped, consumer))
902}
903
904/// Locks a mutex, recovering from a poisoning panic elsewhere.
905fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
906    mutex
907        .lock()
908        .unwrap_or_else(std::sync::PoisonError::into_inner)
909}
910
911#[cfg(test)]
912mod tests {
913    #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
914
915    use super::*;
916    use astrs_wire::{DataflowId, NodeId};
917
918    fn input() -> DataId {
919        DataId::new("frames").unwrap()
920    }
921
922    fn source() -> PortRef {
923        PortRef::from_parts("camera", "image").unwrap()
924    }
925
926    #[test]
927    fn a_fresh_input_is_on_the_daemon_path() {
928        let table = InputRouteTable::new();
929        assert_eq!(table.plane(&input()), RoutePlane::Daemon);
930        assert_eq!(table.segment(&input()), None);
931        assert_eq!(table.attached_count(), 0);
932        assert!(table.planes().is_empty());
933        assert_eq!(table.stats(), InputPlaneStats::default());
934    }
935
936    #[test]
937    fn detaching_an_unknown_input_is_a_no_op() {
938        let table = InputRouteTable::new();
939        assert!(!table.detach(&input()));
940        table.close();
941        assert_eq!(table.stats().detaches, 0);
942    }
943
944    #[test]
945    fn refusals_are_counted() {
946        let table = InputRouteTable::new();
947        table.count_refusal();
948        assert_eq!(table.stats().refusals, 1);
949    }
950
951    #[test]
952    fn a_segment_name_that_does_not_match_the_producer_is_refused() {
953        let spec = ShmSegmentSpec::new("astrs-not-this-segment", 3, 8, 4096);
954        let error = open_consumer(DataflowId::from_u128(1), &source(), &spec, None).unwrap_err();
955        assert!(matches!(error, NodeError::Pattern(_)), "{error}");
956        assert!(error.to_string().contains("camera/image"));
957    }
958
959    #[test]
960    fn the_canonical_producer_name_is_what_the_daemon_sends() {
961        let dataflow = DataflowId::from_u128(0x9001);
962        let key = SegmentKey::new(
963            dataflow,
964            NodeId::new("camera").unwrap(),
965            DataId::new("image").unwrap(),
966            5,
967        );
968        assert!(key.canonical().ends_with("/camera/image/5"));
969        // The hashed `shm_open` form never travels, so it is refused.
970        let hashed = ShmSegmentSpec::new(key.os_name().as_str(), 5, 8, 4096);
971        assert!(open_consumer(dataflow, &source(), &hashed, None).is_err());
972    }
973
974    #[cfg(unix)]
975    #[test]
976    fn a_real_segment_is_opened_verified_and_read_from() {
977        use astrs_shm::{Backing, Producer, SegmentConfig};
978
979        let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9002);
980        let key = SegmentKey::new(
981            dataflow,
982            NodeId::new("camera").unwrap(),
983            DataId::new("image").unwrap(),
984            1,
985        );
986        let config = SegmentConfig::new(4, 4096)
987            .unwrap()
988            .with_backing(Backing::Named);
989        let Ok(segment) = Segment::create(key.clone(), config) else {
990            // A sandbox without a usable shared-memory backing is not a test
991            // failure; the plane is an optimisation (§6.2).
992            return;
993        };
994        let segment = segment.shared();
995        let mut producer = Producer::new(Arc::clone(&segment)).unwrap();
996
997        let spec = ShmSegmentSpec::new(key.canonical(), 1, 4, 4096);
998        let (mapped, mut consumer) =
999            open_consumer(dataflow, &source(), &spec, None).expect("the segment opens");
1000        assert_eq!(mapped.layout().slot_count(), 4);
1001
1002        // `Latest` means the next write, so a message published *after* the
1003        // attach is the first one seen.
1004        producer.send(b"after", b"").unwrap();
1005        let sample = consumer.next_blocking(Duration::from_secs(2)).unwrap();
1006        assert_eq!(sample.payload(), b"after");
1007
1008        // A generation this node does not believe in is refused before any
1009        // mapping happens.
1010        let stale = ShmSegmentSpec::new(key.canonical(), 2, 4, 4096);
1011        assert!(open_consumer(dataflow, &source(), &stale, None).is_err());
1012
1013        drop(sample);
1014        drop(consumer);
1015        let _ = segment.unlink();
1016    }
1017
1018    /// A session with one input, `frames`, fed by `camera/image`.
1019    #[cfg(unix)]
1020    fn session(
1021        dataflow: DataflowId,
1022    ) -> (
1023        Arc<SessionShared>,
1024        tokio::sync::mpsc::Receiver<crate::session::Outgoing>,
1025    ) {
1026        use astrs_wire::{FrameLimits, InputSpec, NodeSource, NodeSpawnSpec, SessionId};
1027
1028        let spec = Arc::new(
1029            NodeSpawnSpec::new(
1030                dataflow,
1031                NodeId::new("detect").unwrap(),
1032                0,
1033                NodeSource::Dynamic,
1034            )
1035            .with_input(InputSpec::new(input(), source())),
1036        );
1037        let events = Arc::new(crate::events::EventSource::new());
1038        for declared in &spec.inputs {
1039            events.register_input(declared).unwrap();
1040        }
1041        let (sender, receiver) = tokio::sync::mpsc::channel(crate::session::OUTGOING_CAPACITY);
1042        let shared = Arc::new(SessionShared::new(
1043            spec,
1044            SessionId::from_u128(11),
1045            sender,
1046            events,
1047            crate::runtime::NodeRuntime::owned().unwrap(),
1048            4096,
1049            FrameLimits::uds(),
1050        ));
1051        (shared, receiver)
1052    }
1053
1054    /// A named segment for `camera/image` at `generation`, plus its producer.
1055    #[cfg(unix)]
1056    fn ring(
1057        dataflow: DataflowId,
1058        generation: u64,
1059    ) -> Option<(
1060        SegmentKey,
1061        Arc<Segment>,
1062        astrs_shm::Producer,
1063        ShmSegmentSpec,
1064    )> {
1065        use astrs_shm::{Backing, Producer, SegmentConfig};
1066
1067        let key = SegmentKey::new(
1068            dataflow,
1069            NodeId::new("camera").unwrap(),
1070            DataId::new("image").unwrap(),
1071            generation,
1072        );
1073        let config = SegmentConfig::new(4, 4096)
1074            .unwrap()
1075            .with_backing(Backing::Named);
1076        let segment = Segment::create(key.clone(), config).ok()?.shared();
1077        let producer = Producer::new(Arc::clone(&segment)).ok()?;
1078        let spec = ShmSegmentSpec::new(key.canonical(), generation, 4, 4096);
1079        Some((key, segment, producer, spec))
1080    }
1081
1082    #[cfg(unix)]
1083    #[test]
1084    fn a_new_generation_replaces_an_attachment_rather_than_being_ignored() {
1085        // The producer-restart case, which an "already attached, ignore" early
1086        // return would wedge permanently: the node would keep reading a ring
1087        // whose producer has gone and never see the one that replaced it.
1088        let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9004);
1089        let Some((_key_one, first, _producer_one, spec_one)) = ring(dataflow, 1) else {
1090            return;
1091        };
1092        let Some((_key_two, second, mut producer_two, spec_two)) = ring(dataflow, 2) else {
1093            let _ = first.unlink();
1094            return;
1095        };
1096        let (shared, _outgoing) = session(dataflow);
1097
1098        shared
1099            .inputs
1100            .attach(&shared, &input(), &source(), &spec_one)
1101            .expect("the first ring attaches");
1102        assert_eq!(shared.inputs.plane(&input()), RoutePlane::Shm);
1103        assert_eq!(
1104            shared.inputs.segment(&input()).map(|s| s.generation),
1105            Some(1)
1106        );
1107        assert_eq!(shared.inputs.stats().attaches, 1);
1108
1109        // The same offer again is silence, not churn: re-attaching would drop
1110        // the reader mid-message for nothing.
1111        shared
1112            .inputs
1113            .attach(&shared, &input(), &source(), &spec_one)
1114            .expect("a repeated offer is accepted");
1115        assert_eq!(shared.inputs.stats().attaches, 1, "no second attachment");
1116
1117        // A different generation is a genuinely different ring.
1118        shared
1119            .inputs
1120            .attach(&shared, &input(), &source(), &spec_two)
1121            .expect("the second ring attaches");
1122        assert_eq!(shared.inputs.stats().attaches, 2);
1123        assert_eq!(
1124            shared.inputs.segment(&input()).map(|s| s.generation),
1125            Some(2)
1126        );
1127        assert_eq!(
1128            shared.inputs.planes(),
1129            vec![(input(), RoutePlane::Shm)],
1130            "the plane is reported per input"
1131        );
1132
1133        // And it is the *new* ring being read.
1134        producer_two.send(b"second-incarnation", b"").unwrap();
1135        let deadline = std::time::Instant::now() + Duration::from_secs(5);
1136        let mut delivered = None;
1137        while std::time::Instant::now() < deadline && delivered.is_none() {
1138            if let Some(crate::events::Event::Input { id, data, .. }) = shared.source.try_next() {
1139                assert_eq!(id, input());
1140                delivered = Some(data);
1141            } else {
1142                std::thread::sleep(Duration::from_millis(5));
1143            }
1144        }
1145        let payload = delivered.expect("a sample from the second ring");
1146        assert!(payload.is_zero_copy());
1147        assert_eq!(payload.bytes(), b"second-incarnation");
1148        let slot = payload.slot().expect("a ring slot");
1149        assert!(
1150            slot.is_inside_mapping() && slot.is_at_layout_offset(),
1151            "{slot:?}"
1152        );
1153        drop(payload);
1154
1155        assert!(shared.inputs.detach(&input()));
1156        assert_eq!(shared.inputs.plane(&input()), RoutePlane::Daemon);
1157        assert!(!shared.inputs.detach(&input()), "a detach happens once");
1158
1159        shared.inputs.close();
1160        let _ = first.unlink();
1161        let _ = second.unlink();
1162    }
1163
1164    #[cfg(unix)]
1165    #[test]
1166    fn latest_skips_what_the_daemon_already_delivered() {
1167        use astrs_shm::{Backing, Producer, SegmentConfig};
1168
1169        let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9003);
1170        let key = SegmentKey::new(
1171            dataflow,
1172            NodeId::new("camera").unwrap(),
1173            DataId::new("image").unwrap(),
1174            1,
1175        );
1176        let config = SegmentConfig::new(4, 4096)
1177            .unwrap()
1178            .with_backing(Backing::Named);
1179        let Ok(segment) = Segment::create(key.clone(), config) else {
1180            return;
1181        };
1182        let segment = segment.shared();
1183        let mut producer = Producer::new(Arc::clone(&segment)).unwrap();
1184        producer.send(b"before", b"").unwrap();
1185
1186        let spec = ShmSegmentSpec::new(key.canonical(), 1, 4, 4096);
1187        let (_mapped, mut consumer) = open_consumer(dataflow, &source(), &spec, None).unwrap();
1188        assert!(
1189            matches!(consumer.try_next(), Err(RecvError::Empty)),
1190            "a message published before the attach is not replayed"
1191        );
1192        producer.send(b"after", b"").unwrap();
1193        let sample = consumer.next_blocking(Duration::from_secs(2)).unwrap();
1194        assert_eq!(sample.payload(), b"after");
1195
1196        drop(sample);
1197        drop(consumer);
1198        let _ = segment.unlink();
1199    }
1200}