astrs_node_api/session/inputs.rs
1//! The slow-start route handshake, **consumer** side (blueprint §6.2, §6.3).
2//!
3//! [`super::routes`] is the producer's half: a `RouteUpgrade` arrives, a
4//! [`astrs_shm::Producer`] is opened, and every later publish writes into a
5//! ring slot. This module is the other half, and it is the one that makes the
6//! plane actually engage: a route is only upgraded once *"all static same-host
7//! consumers attached to the ring"* (§6.3), so until a consumer attaches by
8//! itself, nothing ever moves off the daemon path.
9//!
10//! ```text
11//! InputRouteUpgrade{input, source, segment}
12//! │
13//! ▼
14//! SegmentClient::attach ──► astrs_shm::Consumer ──► reader thread
15//! │ │
16//! │ doorbell registered with the broker │ Payload::zero_copy
17//! ▼ ▼
18//! the daemon observes the pid in the consumer table the node's input queue
19//! │
20//! ▼
21//! RouteUpgrade to the producer — §6.3 completes
22//! ```
23//!
24//! # Why attaching early is safe
25//!
26//! The daemon sends `InputRouteUpgrade` *before* the producer is offered its
27//! own upgrade, because the producer's offer is gated on observing this
28//! consumer in the segment's consumer table. So a consumer always attaches
29//! while the producer is still publishing through the daemon — and the ring is
30//! therefore empty. [`astrs_shm::StartPosition::Latest`] puts the cursor at the
31//! *next* write, so the transition can neither duplicate a message the daemon
32//! already delivered nor skip the first one the ring carries.
33//!
34//! # What still arrives on the daemon path
35//!
36//! An upgraded route is not a ring *only*. §6.2 keeps the threshold rule — *"a
37//! heap payload ≥ threshold is copied once into a slot; below threshold it
38//! rides the UDS control channel"* — so a producer on the shared-memory plane
39//! still publishes its small messages through the daemon, and so does one
40//! whose ring was momentarily full (§6.2: *never sleep-retry*). Both arrive
41//! here as ordinary [`astrs_wire::NodeEvent::Input`] frames on the same input
42//! queue. A node sees one stream; [`crate::Payload::is_zero_copy`] is the only
43//! thing that differs.
44//!
45//! The consequence worth stating plainly: on an output that publishes payloads
46//! on *both* sides of the threshold, the two carriers have different latencies
47//! and the relative order of a large and a small message is not guaranteed.
48//! An output whose payloads are uniformly sized — the normal case, since a
49//! port carries one type — stays on one carrier and stays in order.
50//!
51//! # Falling back is a detach
52//!
53//! Every fatal condition ends the same way: the [`astrs_shm::Consumer`] is
54//! dropped, which releases its consumer-table entry, which the daemon's next
55//! plane pass observes as a detachment — and a detached consumer downgrades
56//! the producer, which resumes publishing through the daemon (§6.3). The node
57//! keeps reading the same event stream throughout. Nothing has to be
58//! negotiated for the fallback to be correct; it falls out of the handshake.
59//!
60//! # Holding a sample holds a slot, and that is fine
61//!
62//! A queued [`crate::Payload`] on the zero-copy plane pins the ring slot it
63//! points at, so a node that reads slowly holds slots its producer cannot
64//! refill. That sounds like a way to stall a producer, and it is worth being
65//! explicit about why it is not.
66//!
67//! Two bounds close it. The consumer's queue policy (§11.2) caps how many
68//! events can be outstanding at once — a `queue_size: 4` input pins at most
69//! four slots, whatever the node does — and the producer's reaction to a ring
70//! with no free slot is [`astrs_shm::Producer::try_allocate`] failing, which
71//! the node API turns into §6.2's documented fallback: publish inline, count
72//! `shm_fallback_total`, carry on. Nothing waits, so nothing stalls.
73//!
74//! An earlier version of this module copied payloads out of their slots once
75//! an input's queue passed a threshold, to "protect" the producer. It bought
76//! nothing the two bounds above do not already give, and it cost the thing the
77//! plane exists for: a consumer that fell briefly behind started copying every
78//! frame, which made it slower, which kept it behind. The copy is still
79//! available — [`crate::Payload::detach`] — to the node, which knows whether
80//! it is about to stash the payload and is the right place for the decision.
81
82use std::collections::HashMap;
83use std::path::Path;
84use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
85use std::sync::{Arc, Mutex, MutexGuard, Weak};
86use std::thread::JoinHandle;
87use std::time::Duration;
88
89use astrs_shm::{
90 AttachOptions, Consumer, RecvError, Sample, Segment, SegmentClient, SegmentKey, StartPosition,
91};
92use astrs_wire::{DataId, DataflowId, Metadata, PortRef, ShmSegmentSpec, WireDecode};
93
94use crate::error::{NodeError, Result};
95use crate::events::{Event, QueuedEvent};
96use crate::payload::Payload;
97use crate::session::SessionShared;
98use crate::session::routes::RoutePlane;
99
100/// How long one blocking receive parks on the doorbell before the reader
101/// looks at its stop flag again.
102///
103/// It bounds how long a detach takes to take effect, not how long a message
104/// waits: the doorbell wakes the reader the moment the producer commits.
105pub const RECV_SLICE: Duration = Duration::from_millis(25);
106
107/// How many consecutive [`RecvError::Lagged`] reports end the attachment.
108///
109/// A lag means the producer overwrote messages this consumer had not read —
110/// possible only under [`astrs_shm::OverflowPolicy::Overwrite`], since the
111/// default blocks instead. One is a hiccup and the cursor recovers by itself.
112/// This many in a row is a consumer that cannot keep up with the ring at all,
113/// and the honest answer is to give the route back to the daemon, whose queue
114/// policy (§11.2) is the node's *declared* answer to being behind.
115pub const MAX_CONSECUTIVE_LAGS: u32 = 8;
116
117/// Counters describing what the consumer-side plane has done.
118#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
119#[non_exhaustive]
120pub struct InputPlaneStats {
121 /// Inputs successfully attached to a ring.
122 pub attaches: u64,
123 /// Attachments that ended, for any reason.
124 pub detaches: u64,
125 /// Upgrades that could not be taken up, so the route stayed on the daemon
126 /// path (§6.2: never sleep-retry).
127 pub refusals: u64,
128 /// Samples delivered from a ring.
129 pub samples: u64,
130 /// Messages the producer overwrote before this consumer read them.
131 pub lagged: u64,
132}
133
134/// One input's place on the plane.
135#[derive(Debug)]
136struct InputRoute {
137 /// Which plane it reads from right now.
138 plane: RoutePlane,
139 /// The segment it is attached to, when it is.
140 segment: Option<ShmSegmentSpec>,
141 /// The live reader, if any.
142 reader: Option<ReaderHandle>,
143 /// Readers that were asked to stop and have not been joined yet.
144 ///
145 /// Joining is deferred rather than skipped: a reader must be *gone* before
146 /// another attaches to the same segment, or two cursors would deliver the
147 /// same message twice. Doing it here rather than in [`InputRouteTable::detach`]
148 /// keeps the session's frame loop off a thread join.
149 retiring: Vec<ReaderHandle>,
150 /// Which incarnation of this input's reader is current.
151 ///
152 /// A reader that exits stamps the table only if it is still the current
153 /// one, so a slow exit cannot clobber a re-attachment that overtook it.
154 epoch: u64,
155}
156
157impl InputRoute {
158 /// A route on the daemon path with no reader.
159 const fn daemon() -> Self {
160 Self {
161 plane: RoutePlane::Daemon,
162 segment: None,
163 reader: None,
164 retiring: Vec::new(),
165 epoch: 0,
166 }
167 }
168}
169
170/// What happened to a closure offered to a reader.
171///
172/// Three outcomes, not two, and the third is the one that matters: a daemon
173/// closes a source from more than one place — the producer's `OutputDone` and
174/// its session ending — so the same `InputClosed` can arrive twice. Treating
175/// the duplicate like an ordinary "the reader would not take it" and queueing
176/// it directly puts a closure *in front of* the frames the first one is still
177/// waiting behind, which is precisely the loss this hand-off exists to
178/// prevent. A duplicate carries no information, so it is discarded.
179#[derive(Debug)]
180enum ClosureOutcome {
181 /// The reader has it and will push it once it has drained.
182 Taken,
183 /// A closure for this input is already queued for delivery; this one is a
184 /// duplicate and is dropped.
185 AlreadyPending,
186 /// The reader has already exited, so its drain is complete: the caller
187 /// should queue the closure itself.
188 Finished(QueuedEvent),
189}
190
191/// The hand-off of an input's closure to the reader that must deliver it last.
192///
193/// One mutex covers both fields on purpose. The reader marks itself finished
194/// and takes any pending closure in the same critical section, so a caller
195/// either wins the race and hands the event over, or loses it and is told to
196/// deliver the event itself — never a window where both believe the other
197/// will, which would lose a closure and hang a node forever.
198#[derive(Debug, Default)]
199struct ClosureSlot {
200 /// The event to push once the drain finishes.
201 pending: Option<QueuedEvent>,
202 /// Set by the reader as it exits, under this same lock.
203 finished: bool,
204}
205
206/// A reader thread and the two ways to end it.
207///
208/// The distinction is the last message of a stream. A producer that finishes
209/// commits its final frame into the ring and *then* tells the daemon, which
210/// tells this consumer its input is closed — over the control connection,
211/// which can easily overtake a sample still travelling on the ring. A reader
212/// that stopped on the closure would drop that frame.
213#[derive(Debug)]
214struct ReaderHandle {
215 /// Set to end the reader without reading anything further.
216 stop: Arc<AtomicBool>,
217 /// Set to make the reader drain what is resident and *then* end.
218 drain: Arc<AtomicBool>,
219 /// The input's closure, delivered behind the messages it closes rather
220 /// than in front of them.
221 closing: Arc<Mutex<ClosureSlot>>,
222 /// The thread itself.
223 join: JoinHandle<()>,
224}
225
226impl ReaderHandle {
227 /// Asks the reader to stop at once, without waiting for it.
228 ///
229 /// For a ring that is gone or replaced: there is nothing left to drain,
230 /// and reading a segment the daemon has reclaimed is the one thing §6.2's
231 /// generation stamp exists to prevent.
232 fn abort(&self) {
233 self.stop.store(true, Ordering::Release);
234 }
235
236 /// Asks the reader to drain what is resident and then stop.
237 fn signal_drain(&self) {
238 self.drain.store(true, Ordering::Release);
239 }
240
241 /// Hands the reader the event to push once it has drained.
242 fn take_closure(&self, event: QueuedEvent) -> ClosureOutcome {
243 let mut slot = lock(&self.closing);
244 if slot.finished {
245 return ClosureOutcome::Finished(event);
246 }
247 if slot.pending.is_some() {
248 return ClosureOutcome::AlreadyPending;
249 }
250 slot.pending = Some(event);
251 self.signal_drain();
252 ClosureOutcome::Taken
253 }
254
255 /// Waits for the reader to exit, which is also when its
256 /// [`astrs_shm::Consumer`] is dropped and its consumer-table entry freed.
257 fn join(self) {
258 self.abort();
259 // A reader that panicked has still released its entry, because the
260 // `Consumer` is dropped by the unwind. There is nothing to report and
261 // nothing to retry.
262 let _ = self.join.join();
263 }
264}
265
266/// Every input's plane state, and the reader threads behind the attached ones.
267#[derive(Debug, Default)]
268pub struct InputRouteTable {
269 /// One entry per input the daemon has said anything about.
270 routes: Mutex<HashMap<DataId, InputRoute>>,
271 /// Counters.
272 attaches: AtomicU64,
273 detaches: AtomicU64,
274 refusals: AtomicU64,
275 samples: AtomicU64,
276 lagged: AtomicU64,
277}
278
279impl InputRouteTable {
280 /// An empty table: every input on the daemon path.
281 #[must_use]
282 pub fn new() -> Self {
283 Self {
284 routes: Mutex::new(HashMap::new()),
285 attaches: AtomicU64::new(0),
286 detaches: AtomicU64::new(0),
287 refusals: AtomicU64::new(0),
288 samples: AtomicU64::new(0),
289 lagged: AtomicU64::new(0),
290 }
291 }
292
293 /// The plane one input reads from.
294 #[must_use]
295 pub fn plane(&self, input: &DataId) -> RoutePlane {
296 lock(&self.routes)
297 .get(input)
298 .map_or(RoutePlane::Daemon, |route| route.plane)
299 }
300
301 /// The plane every input the daemon has spoken about reads from, sorted.
302 #[must_use]
303 pub fn planes(&self) -> Vec<(DataId, RoutePlane)> {
304 let mut planes: Vec<(DataId, RoutePlane)> = lock(&self.routes)
305 .iter()
306 .map(|(id, route)| (id.clone(), route.plane))
307 .collect();
308 planes.sort_by(|left, right| left.0.cmp(&right.0));
309 planes
310 }
311
312 /// The segment one input is attached to.
313 #[must_use]
314 pub fn segment(&self, input: &DataId) -> Option<ShmSegmentSpec> {
315 lock(&self.routes)
316 .get(input)
317 .and_then(|route| route.segment.clone())
318 }
319
320 /// How many inputs read from a ring right now.
321 #[must_use]
322 pub fn attached_count(&self) -> usize {
323 lock(&self.routes)
324 .values()
325 .filter(|route| route.plane.is_zero_copy())
326 .count()
327 }
328
329 /// A snapshot of the counters.
330 #[must_use]
331 pub fn stats(&self) -> InputPlaneStats {
332 InputPlaneStats {
333 attaches: self.attaches.load(Ordering::Relaxed),
334 detaches: self.detaches.load(Ordering::Relaxed),
335 refusals: self.refusals.load(Ordering::Relaxed),
336 samples: self.samples.load(Ordering::Relaxed),
337 lagged: self.lagged.load(Ordering::Relaxed),
338 }
339 }
340
341 /// Records one upgrade this node could not take up.
342 pub fn count_refusal(&self) {
343 self.refusals.fetch_add(1, Ordering::Relaxed);
344 }
345
346 /// Attaches `input` to `segment` and starts reading it (§6.3).
347 ///
348 /// Idempotent for the generation already attached: a repeated offer for
349 /// the same incarnation is answered by leaving the working reader alone. A
350 /// *different* generation replaces it, which is the producer-restart case
351 /// — the old ring is gone and its reader has nothing left to read.
352 ///
353 /// # Errors
354 ///
355 /// [`NodeError::Shm`] when the segment cannot be opened, verified or read,
356 /// and [`NodeError::Pattern`] when the daemon's segment name does not
357 /// describe the producer the event names. Either leaves the input on the
358 /// daemon path, which is the honest outcome (§6.2: never sleep-retry).
359 pub fn attach(
360 &self,
361 shared: &Arc<SessionShared>,
362 input: &DataId,
363 source: &PortRef,
364 segment: &ShmSegmentSpec,
365 ) -> Result<()> {
366 // A repeat of what is already running costs nothing and changes
367 // nothing. Checked before the segment is opened, because opening is
368 // the expensive half.
369 let retiring = {
370 let mut routes = lock(&self.routes);
371 let route = routes
372 .entry(input.clone())
373 .or_insert_with(InputRoute::daemon);
374 if route.plane.is_zero_copy()
375 && route.segment.as_ref().is_some_and(|open| {
376 open.generation == segment.generation && open.name == segment.name
377 })
378 {
379 return Ok(());
380 }
381 // Anything already running for this input is superseded, and its
382 // ring is a previous incarnation's: aborted, not drained.
383 if let Some(reader) = route.reader.take() {
384 reader.abort();
385 route.retiring.push(reader);
386 }
387 route.plane = RoutePlane::Daemon;
388 route.segment = None;
389 core::mem::take(&mut route.retiring)
390 };
391 // Joined outside the lock: an exiting reader takes it to stamp itself
392 // detached, and joining while holding it would deadlock.
393 for reader in retiring {
394 reader.join();
395 }
396
397 let (mapped, consumer) = open_consumer(
398 shared.dataflow,
399 source,
400 segment,
401 shared.shm_broker.as_deref(),
402 )?;
403 let stop = Arc::new(AtomicBool::new(false));
404 let drain = Arc::new(AtomicBool::new(false));
405 let closing: Arc<Mutex<ClosureSlot>> = Arc::new(Mutex::new(ClosureSlot::default()));
406
407 let epoch = {
408 let mut routes = lock(&self.routes);
409 let route = routes
410 .entry(input.clone())
411 .or_insert_with(InputRoute::daemon);
412 route.epoch = route.epoch.wrapping_add(1);
413 route.epoch
414 };
415
416 let reader = ReaderContext {
417 input: input.clone(),
418 source: source.clone(),
419 session: Arc::downgrade(shared),
420 stop: Arc::clone(&stop),
421 drain: Arc::clone(&drain),
422 closing: Arc::clone(&closing),
423 epoch,
424 };
425 let join = std::thread::Builder::new()
426 .name(format!("astrs-shm-{input}"))
427 .spawn(move || reader.run(consumer))
428 .map_err(|error| {
429 NodeError::Pattern(format!(
430 "could not start the ring reader for `{input}`: {error}"
431 ))
432 })?;
433
434 let mut routes = lock(&self.routes);
435 let route = routes
436 .entry(input.clone())
437 .or_insert_with(InputRoute::daemon);
438 route.plane = RoutePlane::Shm;
439 route.segment = Some(segment.clone());
440 route.reader = Some(ReaderHandle {
441 stop,
442 drain,
443 closing,
444 join,
445 });
446 // The mapping itself is kept alive by the `Consumer` the reader owns;
447 // this handle has done its job.
448 drop(mapped);
449 self.attaches.fetch_add(1, Ordering::Relaxed);
450 Ok(())
451 }
452
453 /// Takes `input` off the ring, if it was on one.
454 ///
455 /// Returns whether anything was attached. The reader **drains what is
456 /// resident** before it exits — §6.2's rule for a producer that goes away
457 /// is that the daemon *"marks `closed`, lets consumers drain, and unlinks"*,
458 /// and a consumer that stopped reading the instant it was told to detach
459 /// would throw away messages the producer had already committed.
460 ///
461 /// The input is on the daemon path from this instant, so anything the
462 /// daemon sends next is delivered inline and nothing arrives twice: the
463 /// ring holds only what was written before the detach.
464 pub fn detach(&self, input: &DataId) -> bool {
465 let mut routes = lock(&self.routes);
466 let Some(route) = routes.get_mut(input) else {
467 return false;
468 };
469 let was_attached = route.plane.is_zero_copy();
470 route.plane = RoutePlane::Daemon;
471 route.segment = None;
472 route.epoch = route.epoch.wrapping_add(1);
473 if let Some(reader) = route.reader.take() {
474 reader.signal_drain();
475 route.retiring.push(reader);
476 }
477 if was_attached {
478 self.detaches.fetch_add(1, Ordering::Relaxed);
479 }
480 was_attached
481 }
482
483 /// Delivers `input`'s closure *behind* whatever its ring still holds.
484 ///
485 /// Returns [`None`] when a reader took the event, and gives it back —
486 /// never drops it — when the input is not on a ring and the caller should
487 /// queue it itself.
488 ///
489 /// # Why the closure cannot simply be queued
490 ///
491 /// A producer that finishes commits its last frame into the ring and only
492 /// then tells the daemon, which tells this consumer. That message travels
493 /// on the control connection, which routinely overtakes a sample still
494 /// being read from the ring — so queueing the closure directly would put
495 /// "this input is finished" in front of the message that finished it, and
496 /// a node that stops reading on the closure (every node) would lose the
497 /// last frame of every stream.
498 ///
499 /// Handing it to the reader instead makes the ordering structural: the
500 /// reader drains, pushes the samples, pushes the closure, and exits.
501 pub fn close_input(&self, input: &DataId, event: QueuedEvent) -> Option<QueuedEvent> {
502 let mut routes = lock(&self.routes);
503 let Some(route) = routes.get_mut(input) else {
504 return Some(event);
505 };
506 if !route.plane.is_zero_copy() {
507 // Already off the ring — but perhaps only just, with a reader
508 // still draining what the producer committed before it went.
509 return Self::hand_to_retiring(route, event);
510 }
511 let Some(reader) = route.reader.take() else {
512 return Some(event);
513 };
514 let outcome = reader.take_closure(event);
515 route.plane = RoutePlane::Daemon;
516 route.segment = None;
517 route.epoch = route.epoch.wrapping_add(1);
518 route.retiring.push(reader);
519 self.detaches.fetch_add(1, Ordering::Relaxed);
520 match outcome {
521 ClosureOutcome::Taken | ClosureOutcome::AlreadyPending => None,
522 ClosureOutcome::Finished(event) => Some(event),
523 }
524 }
525
526 /// Hands `input`'s closure to a reader that is still draining, if there is
527 /// one.
528 ///
529 /// The ordering this exists for: a producer that dies makes the daemon
530 /// send *both* an `InputRouteDowngrade` (the ring is going away) and an
531 /// `InputClosed` (the stream ended), and the downgrade arrives first — so
532 /// by the time the closure gets here the input is already off the ring and
533 /// its reader is in [`ReaderHandle`]'s retiring list, draining the last
534 /// frames the producer committed. Queueing the closure directly would put
535 /// it in front of them.
536 ///
537 /// Returns [`None`] when a draining reader took it.
538 fn hand_to_retiring(route: &mut InputRoute, event: QueuedEvent) -> Option<QueuedEvent> {
539 let mut event = event;
540 // Most recent first: an older reader is draining an older ring, and
541 // this closure belongs behind the newest thing that will be delivered.
542 for reader in route.retiring.iter().rev() {
543 match reader.take_closure(event) {
544 ClosureOutcome::Taken | ClosureOutcome::AlreadyPending => return None,
545 ClosureOutcome::Finished(returned) => event = returned,
546 }
547 }
548 Some(event)
549 }
550
551 /// Finishes every ring reader, so nothing is still in flight.
552 ///
553 /// The [`InputRouteTable::close_input`] hand-off orders one input's
554 /// closure behind its own ring. [`astrs_wire::NodeEvent::AllInputsClosed`]
555 /// has no input to hang off: the daemon sends it straight to the session,
556 /// so it arrives on the node's *control* lane, which is served before any
557 /// data — and it is terminal
558 /// ([`astrs_wire::NodeEvent::is_terminal`]), so a node that winds down on
559 /// it (`astrs-record-node` and `examples/rust-pipeline`'s recorder both do)
560 /// would drop whatever a ring was still draining.
561 ///
562 /// So the session drains first and pushes it second. This *does* wait —
563 /// bounded by [`RECV_SLICE`] per reader — which is acceptable here and
564 /// nowhere else: every input has just been declared finished, so there is
565 /// nothing left for the session to be responsive to.
566 ///
567 /// Returns how many readers were waited for.
568 pub fn drain_all(&self) -> usize {
569 let readers: Vec<ReaderHandle> = {
570 let mut routes = lock(&self.routes);
571 let mut readers = Vec::new();
572 for route in routes.values_mut() {
573 if route.plane.is_zero_copy() {
574 self.detaches.fetch_add(1, Ordering::Relaxed);
575 }
576 route.plane = RoutePlane::Daemon;
577 route.segment = None;
578 route.epoch = route.epoch.wrapping_add(1);
579 if let Some(reader) = route.reader.take() {
580 readers.push(reader);
581 }
582 readers.append(&mut route.retiring);
583 }
584 for reader in &readers {
585 reader.signal_drain();
586 }
587 readers
588 };
589 let waited = readers.len();
590 // Joined outside the lock: an exiting reader takes it to stamp itself
591 // detached, and joining while holding it would deadlock. `join` also
592 // sets the abort flag, which a reader that has finished draining has
593 // already passed — it ends the wait for one that is parked with an
594 // empty ring rather than cutting a drain short.
595 for reader in readers {
596 reader.join();
597 }
598 waited
599 }
600
601 /// Detaches every input and waits for every reader to exit.
602 ///
603 /// Called when the session ends. Waiting matters: a reader still holding a
604 /// [`astrs_shm::Sample`] holds a ring slot, and a process that exits
605 /// without releasing its consumer-table entry leaves the daemon to reclaim
606 /// it on a timeout instead of immediately.
607 pub fn close(&self) {
608 let readers: Vec<ReaderHandle> = {
609 let mut routes = lock(&self.routes);
610 let mut readers = Vec::new();
611 for route in routes.values_mut() {
612 if route.plane.is_zero_copy() {
613 self.detaches.fetch_add(1, Ordering::Relaxed);
614 }
615 route.plane = RoutePlane::Daemon;
616 route.segment = None;
617 route.epoch = route.epoch.wrapping_add(1);
618 if let Some(reader) = route.reader.take() {
619 readers.push(reader);
620 }
621 readers.append(&mut route.retiring);
622 }
623 for reader in &readers {
624 reader.abort();
625 }
626 readers
627 };
628 for reader in readers {
629 reader.join();
630 }
631 }
632
633 /// Marks one input back on the daemon path from its own reader thread.
634 ///
635 /// Ignored when `epoch` is not the current one: a reader that took a while
636 /// to notice its stop flag must not undo the attachment that replaced it.
637 fn mark_detached(&self, input: &DataId, epoch: u64) {
638 let mut routes = lock(&self.routes);
639 let Some(route) = routes.get_mut(input) else {
640 return;
641 };
642 if route.epoch != epoch {
643 return;
644 }
645 if route.plane.is_zero_copy() {
646 self.detaches.fetch_add(1, Ordering::Relaxed);
647 }
648 route.plane = RoutePlane::Daemon;
649 route.segment = None;
650 }
651
652 /// Records one sample delivered from a ring.
653 fn count_sample(&self) {
654 self.samples.fetch_add(1, Ordering::Relaxed);
655 }
656
657 /// Records messages the producer overwrote before they were read.
658 fn count_lagged(&self, missed: u64) {
659 self.lagged.fetch_add(missed, Ordering::Relaxed);
660 }
661}
662
663/// Everything one reader thread needs, so the thread body is a method and not
664/// a seven-argument function.
665struct ReaderContext {
666 /// The input being read.
667 input: DataId,
668 /// The producer port behind it, which every delivered event names.
669 source: PortRef,
670 /// The session to deliver into.
671 ///
672 /// Weak because the table this reader is registered in lives *inside* the
673 /// session: a strong reference would keep it alive forever.
674 session: Weak<SessionShared>,
675 /// Set when the reader should end at once, reading nothing further.
676 stop: Arc<AtomicBool>,
677 /// Set when the reader should drain what is resident and then end.
678 drain: Arc<AtomicBool>,
679 /// The input's closure, to push once the drain finishes.
680 closing: Arc<Mutex<ClosureSlot>>,
681 /// Which incarnation of this input's reader this is.
682 epoch: u64,
683}
684
685impl ReaderContext {
686 /// Reads until told to stop, the session ends, or the ring does.
687 fn run(self, mut consumer: Consumer) {
688 let mut consecutive_lags = 0u32;
689 while !self.stop.load(Ordering::Acquire) {
690 let Some(session) = self.session.upgrade() else {
691 break;
692 };
693 if session.is_closed() {
694 break;
695 }
696 if self.drain.load(Ordering::Acquire) {
697 self.finish(&session, &mut consumer);
698 break;
699 }
700 match consumer.next_blocking(RECV_SLICE) {
701 Ok(sample) => {
702 consecutive_lags = 0;
703 self.deliver(&session, sample);
704 }
705 Err(RecvError::Empty) => {}
706 Err(RecvError::Lagged(missed)) => {
707 session.inputs.count_lagged(missed);
708 consecutive_lags += 1;
709 if consecutive_lags >= MAX_CONSECUTIVE_LAGS {
710 session.source.push_control(Event::Error(format!(
711 "input `{}` fell behind its ring {MAX_CONSECUTIVE_LAGS} times in a \
712 row; reading it through the daemon instead",
713 self.input
714 )));
715 break;
716 }
717 }
718 // The producer finished, or its segment was reclaimed: both
719 // are the daemon's business to report, and it does — as
720 // `InputClosed` or `InputRouteDowngrade` on the control
721 // connection. Ending quietly here avoids saying it twice.
722 Err(RecvError::Closed) => break,
723 // `RecvError::Shm` and — since the enum is `#[non_exhaustive]`
724 // — any outcome a future `astrs-shm` adds: a condition this
725 // build cannot classify is treated as fatal for the
726 // attachment, which costs the plane and keeps the data.
727 Err(error) => {
728 session.source.push_control(Event::Error(format!(
729 "input `{}` left the shared-memory plane: {error}",
730 self.input
731 )));
732 break;
733 }
734 }
735 }
736
737 // Dropping the consumer releases its consumer-table entry, which is
738 // what the daemon observes as a detachment — and a detached consumer
739 // downgrades the producer back onto the daemon path (§6.3). The
740 // fallback needs no message of its own.
741 drop(consumer);
742 // Marked finished and drained of any pending closure in one critical
743 // section, so a `close_input` racing this exit is told to deliver the
744 // closure itself rather than handing it to a thread that is gone.
745 let closure = {
746 let mut slot = lock(&self.closing);
747 slot.finished = true;
748 slot.pending.take()
749 };
750 if let Some(session) = self.session.upgrade() {
751 if let Some(closure) = closure {
752 session.source.push_input(&self.input, closure);
753 }
754 session.inputs.mark_detached(&self.input, self.epoch);
755 }
756 }
757
758 /// Delivers everything still resident in the ring, then the closure.
759 ///
760 /// Bounded by construction: a ring holds at most `slot_count` messages and
761 /// nothing is being written into it any more, so the loop ends at the
762 /// first `Empty`. A lag report is not a reason to stop draining — the
763 /// messages behind the gap are still there and still wanted.
764 ///
765 /// The closure itself is pushed by [`ReaderContext::run`]'s exit path,
766 /// which is where the hand-off is settled against a concurrent
767 /// [`InputRouteTable::close_input`].
768 fn finish(&self, session: &Arc<SessionShared>, consumer: &mut Consumer) {
769 loop {
770 match consumer.try_next() {
771 Ok(sample) => self.deliver(session, sample),
772 Err(RecvError::Lagged(missed)) => session.inputs.count_lagged(missed),
773 // `Empty`, `Closed`, or a fault: in every case the ring has
774 // nothing more to give this consumer.
775 Err(_) => break,
776 }
777 }
778 }
779
780 /// Turns one sample into a queued input event.
781 fn deliver(&self, session: &Arc<SessionShared>, sample: Sample) {
782 let metadata = slot_metadata(sample.metadata(), session);
783 // Fold the producer's clock reading into ours, exactly as the daemon
784 // path does, so causal order survives the hop (§4.3).
785 let _accepted = session.observe_remote(metadata.timestamp);
786
787 // Queued as it lies: the slot stays pinned until the node drops the
788 // payload, which is what makes reading it free. See the module
789 // documentation for why that cannot stall the producer.
790 let payload = Payload::zero_copy(sample);
791 session.source.push_input(
792 &self.input,
793 QueuedEvent::Input {
794 source: self.source.clone(),
795 metadata,
796 payload,
797 },
798 );
799 session.inputs.count_sample();
800 }
801}
802
803/// The metadata a slot carried, or a fresh stamp when it carried none.
804///
805/// The producer commits its [`Metadata`] beside the payload (§6.1, §6.2) and it
806/// is not replaceable: `request_id`, `goal_id`, `seq` and `_schema_hash` are
807/// what make a service exchange correlate (§9.4) and a schema drift detectable.
808/// Re-stamping would deliver the bytes and lose the message — so it happens
809/// only when there is nothing to lose.
810fn slot_metadata(bytes: &[u8], session: &Arc<SessionShared>) -> Metadata {
811 if bytes.is_empty() {
812 return Metadata::new(session.hlc_now());
813 }
814 Metadata::decode_exact(bytes).unwrap_or_else(|_| Metadata::new(session.hlc_now()))
815}
816
817/// Opens the segment an
818/// [`InputRouteUpgrade`](astrs_wire::NodeEvent::InputRouteUpgrade) names and
819/// attaches a reader to it.
820///
821/// The mirror of [`crate::session::routes::open_producer`], and it checks the
822/// same two things: the name the daemon sent must be the one the *producer's*
823/// `(dataflow, node, output, generation)` describes, and the mapped header's
824/// digest and generation must match that key. A stale ring from a previous
825/// incarnation therefore cannot be read, which is §6.2's crash-safety property
826/// from the consumer's side.
827///
828/// # The doorbell
829///
830/// A consumer in a different process from its producer cannot be woken by the
831/// producer directly — the two share a mapping, not a descriptor table. The
832/// broker is the go-between: [`astrs_shm::SegmentClient::register_doorbell`]
833/// passes this consumer's ringer over the same Unix socket that carried the
834/// segment, and the producer picks it up from the shared registry on its next
835/// commit. Without it a receive still works, but only by waiting out
836/// `astrs-shm`'s growing fallback slice — correct, and needlessly slow.
837///
838/// # Errors
839///
840/// [`NodeError::Shm`] when the segment cannot be opened, verified or attached
841/// to, and [`NodeError::Pattern`] when the daemon's name does not describe the
842/// producer port the event names.
843pub fn open_consumer(
844 dataflow: DataflowId,
845 source: &PortRef,
846 segment: &ShmSegmentSpec,
847 broker: Option<&Path>,
848) -> Result<(Arc<Segment>, Consumer)> {
849 let key = SegmentKey::new(
850 dataflow,
851 source.node().clone(),
852 source.port().clone(),
853 segment.generation,
854 );
855 let canonical = key.canonical();
856 if canonical != segment.name {
857 return Err(NodeError::Pattern(format!(
858 "input route upgrade names segment `{}`, but `{source}` at generation {} is \
859 `{canonical}`",
860 segment.name, segment.generation,
861 )));
862 }
863
864 let mut client = match broker {
865 Some(path) => Some(SegmentClient::connect(path)?),
866 None => None,
867 };
868 let mapped = match client.as_mut() {
869 Some(client) => client.attach(&key)?,
870 // No broker: the in-process harness and any embedder that disabled the
871 // plane socket. Only a named backing can be opened this way, which is
872 // exactly what those two use.
873 None => Segment::open_named(&key.os_name(), Some(&key))?,
874 }
875 .shared();
876
877 let consumer = Consumer::attach(
878 Arc::clone(&mapped),
879 AttachOptions::new()
880 // The next write, not the oldest resident one: everything already
881 // in the ring — if anything is — was published before this route
882 // moved, and the daemon delivered it inline.
883 .with_start(StartPosition::Latest)
884 .with_doorbell(true)
885 .expecting(key.clone()),
886 )?;
887
888 if let Some(client) = client.as_mut()
889 && let Some(doorbell) = consumer.doorbell()
890 {
891 let ringer = doorbell.ringer()?;
892 // A broker that will not take the ringer costs latency, never
893 // correctness: `next_blocking` falls back to a bounded wait slice.
894 if let Err(error) =
895 client.register_doorbell(&key, consumer.index(), consumer.token(), ringer)
896 {
897 tracing::debug!(%error, segment = %canonical, "the broker refused a doorbell");
898 }
899 }
900
901 Ok((mapped, consumer))
902}
903
904/// Locks a mutex, recovering from a poisoning panic elsewhere.
905fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
906 mutex
907 .lock()
908 .unwrap_or_else(std::sync::PoisonError::into_inner)
909}
910
911#[cfg(test)]
912mod tests {
913 #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
914
915 use super::*;
916 use astrs_wire::{DataflowId, NodeId};
917
918 fn input() -> DataId {
919 DataId::new("frames").unwrap()
920 }
921
922 fn source() -> PortRef {
923 PortRef::from_parts("camera", "image").unwrap()
924 }
925
926 #[test]
927 fn a_fresh_input_is_on_the_daemon_path() {
928 let table = InputRouteTable::new();
929 assert_eq!(table.plane(&input()), RoutePlane::Daemon);
930 assert_eq!(table.segment(&input()), None);
931 assert_eq!(table.attached_count(), 0);
932 assert!(table.planes().is_empty());
933 assert_eq!(table.stats(), InputPlaneStats::default());
934 }
935
936 #[test]
937 fn detaching_an_unknown_input_is_a_no_op() {
938 let table = InputRouteTable::new();
939 assert!(!table.detach(&input()));
940 table.close();
941 assert_eq!(table.stats().detaches, 0);
942 }
943
944 #[test]
945 fn refusals_are_counted() {
946 let table = InputRouteTable::new();
947 table.count_refusal();
948 assert_eq!(table.stats().refusals, 1);
949 }
950
951 #[test]
952 fn a_segment_name_that_does_not_match_the_producer_is_refused() {
953 let spec = ShmSegmentSpec::new("astrs-not-this-segment", 3, 8, 4096);
954 let error = open_consumer(DataflowId::from_u128(1), &source(), &spec, None).unwrap_err();
955 assert!(matches!(error, NodeError::Pattern(_)), "{error}");
956 assert!(error.to_string().contains("camera/image"));
957 }
958
959 #[test]
960 fn the_canonical_producer_name_is_what_the_daemon_sends() {
961 let dataflow = DataflowId::from_u128(0x9001);
962 let key = SegmentKey::new(
963 dataflow,
964 NodeId::new("camera").unwrap(),
965 DataId::new("image").unwrap(),
966 5,
967 );
968 assert!(key.canonical().ends_with("/camera/image/5"));
969 // The hashed `shm_open` form never travels, so it is refused.
970 let hashed = ShmSegmentSpec::new(key.os_name().as_str(), 5, 8, 4096);
971 assert!(open_consumer(dataflow, &source(), &hashed, None).is_err());
972 }
973
974 #[cfg(unix)]
975 #[test]
976 fn a_real_segment_is_opened_verified_and_read_from() {
977 use astrs_shm::{Backing, Producer, SegmentConfig};
978
979 let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9002);
980 let key = SegmentKey::new(
981 dataflow,
982 NodeId::new("camera").unwrap(),
983 DataId::new("image").unwrap(),
984 1,
985 );
986 let config = SegmentConfig::new(4, 4096)
987 .unwrap()
988 .with_backing(Backing::Named);
989 let Ok(segment) = Segment::create(key.clone(), config) else {
990 // A sandbox without a usable shared-memory backing is not a test
991 // failure; the plane is an optimisation (§6.2).
992 return;
993 };
994 let segment = segment.shared();
995 let mut producer = Producer::new(Arc::clone(&segment)).unwrap();
996
997 let spec = ShmSegmentSpec::new(key.canonical(), 1, 4, 4096);
998 let (mapped, mut consumer) =
999 open_consumer(dataflow, &source(), &spec, None).expect("the segment opens");
1000 assert_eq!(mapped.layout().slot_count(), 4);
1001
1002 // `Latest` means the next write, so a message published *after* the
1003 // attach is the first one seen.
1004 producer.send(b"after", b"").unwrap();
1005 let sample = consumer.next_blocking(Duration::from_secs(2)).unwrap();
1006 assert_eq!(sample.payload(), b"after");
1007
1008 // A generation this node does not believe in is refused before any
1009 // mapping happens.
1010 let stale = ShmSegmentSpec::new(key.canonical(), 2, 4, 4096);
1011 assert!(open_consumer(dataflow, &source(), &stale, None).is_err());
1012
1013 drop(sample);
1014 drop(consumer);
1015 let _ = segment.unlink();
1016 }
1017
1018 /// A session with one input, `frames`, fed by `camera/image`.
1019 #[cfg(unix)]
1020 fn session(
1021 dataflow: DataflowId,
1022 ) -> (
1023 Arc<SessionShared>,
1024 tokio::sync::mpsc::Receiver<crate::session::Outgoing>,
1025 ) {
1026 use astrs_wire::{FrameLimits, InputSpec, NodeSource, NodeSpawnSpec, SessionId};
1027
1028 let spec = Arc::new(
1029 NodeSpawnSpec::new(
1030 dataflow,
1031 NodeId::new("detect").unwrap(),
1032 0,
1033 NodeSource::Dynamic,
1034 )
1035 .with_input(InputSpec::new(input(), source())),
1036 );
1037 let events = Arc::new(crate::events::EventSource::new());
1038 for declared in &spec.inputs {
1039 events.register_input(declared).unwrap();
1040 }
1041 let (sender, receiver) = tokio::sync::mpsc::channel(crate::session::OUTGOING_CAPACITY);
1042 let shared = Arc::new(SessionShared::new(
1043 spec,
1044 SessionId::from_u128(11),
1045 sender,
1046 events,
1047 crate::runtime::NodeRuntime::owned().unwrap(),
1048 4096,
1049 FrameLimits::uds(),
1050 ));
1051 (shared, receiver)
1052 }
1053
1054 /// A named segment for `camera/image` at `generation`, plus its producer.
1055 #[cfg(unix)]
1056 fn ring(
1057 dataflow: DataflowId,
1058 generation: u64,
1059 ) -> Option<(
1060 SegmentKey,
1061 Arc<Segment>,
1062 astrs_shm::Producer,
1063 ShmSegmentSpec,
1064 )> {
1065 use astrs_shm::{Backing, Producer, SegmentConfig};
1066
1067 let key = SegmentKey::new(
1068 dataflow,
1069 NodeId::new("camera").unwrap(),
1070 DataId::new("image").unwrap(),
1071 generation,
1072 );
1073 let config = SegmentConfig::new(4, 4096)
1074 .unwrap()
1075 .with_backing(Backing::Named);
1076 let segment = Segment::create(key.clone(), config).ok()?.shared();
1077 let producer = Producer::new(Arc::clone(&segment)).ok()?;
1078 let spec = ShmSegmentSpec::new(key.canonical(), generation, 4, 4096);
1079 Some((key, segment, producer, spec))
1080 }
1081
1082 #[cfg(unix)]
1083 #[test]
1084 fn a_new_generation_replaces_an_attachment_rather_than_being_ignored() {
1085 // The producer-restart case, which an "already attached, ignore" early
1086 // return would wedge permanently: the node would keep reading a ring
1087 // whose producer has gone and never see the one that replaced it.
1088 let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9004);
1089 let Some((_key_one, first, _producer_one, spec_one)) = ring(dataflow, 1) else {
1090 return;
1091 };
1092 let Some((_key_two, second, mut producer_two, spec_two)) = ring(dataflow, 2) else {
1093 let _ = first.unlink();
1094 return;
1095 };
1096 let (shared, _outgoing) = session(dataflow);
1097
1098 shared
1099 .inputs
1100 .attach(&shared, &input(), &source(), &spec_one)
1101 .expect("the first ring attaches");
1102 assert_eq!(shared.inputs.plane(&input()), RoutePlane::Shm);
1103 assert_eq!(
1104 shared.inputs.segment(&input()).map(|s| s.generation),
1105 Some(1)
1106 );
1107 assert_eq!(shared.inputs.stats().attaches, 1);
1108
1109 // The same offer again is silence, not churn: re-attaching would drop
1110 // the reader mid-message for nothing.
1111 shared
1112 .inputs
1113 .attach(&shared, &input(), &source(), &spec_one)
1114 .expect("a repeated offer is accepted");
1115 assert_eq!(shared.inputs.stats().attaches, 1, "no second attachment");
1116
1117 // A different generation is a genuinely different ring.
1118 shared
1119 .inputs
1120 .attach(&shared, &input(), &source(), &spec_two)
1121 .expect("the second ring attaches");
1122 assert_eq!(shared.inputs.stats().attaches, 2);
1123 assert_eq!(
1124 shared.inputs.segment(&input()).map(|s| s.generation),
1125 Some(2)
1126 );
1127 assert_eq!(
1128 shared.inputs.planes(),
1129 vec![(input(), RoutePlane::Shm)],
1130 "the plane is reported per input"
1131 );
1132
1133 // And it is the *new* ring being read.
1134 producer_two.send(b"second-incarnation", b"").unwrap();
1135 let deadline = std::time::Instant::now() + Duration::from_secs(5);
1136 let mut delivered = None;
1137 while std::time::Instant::now() < deadline && delivered.is_none() {
1138 if let Some(crate::events::Event::Input { id, data, .. }) = shared.source.try_next() {
1139 assert_eq!(id, input());
1140 delivered = Some(data);
1141 } else {
1142 std::thread::sleep(Duration::from_millis(5));
1143 }
1144 }
1145 let payload = delivered.expect("a sample from the second ring");
1146 assert!(payload.is_zero_copy());
1147 assert_eq!(payload.bytes(), b"second-incarnation");
1148 let slot = payload.slot().expect("a ring slot");
1149 assert!(
1150 slot.is_inside_mapping() && slot.is_at_layout_offset(),
1151 "{slot:?}"
1152 );
1153 drop(payload);
1154
1155 assert!(shared.inputs.detach(&input()));
1156 assert_eq!(shared.inputs.plane(&input()), RoutePlane::Daemon);
1157 assert!(!shared.inputs.detach(&input()), "a detach happens once");
1158
1159 shared.inputs.close();
1160 let _ = first.unlink();
1161 let _ = second.unlink();
1162 }
1163
1164 #[cfg(unix)]
1165 #[test]
1166 fn latest_skips_what_the_daemon_already_delivered() {
1167 use astrs_shm::{Backing, Producer, SegmentConfig};
1168
1169 let dataflow = DataflowId::from_u128((u128::from(std::process::id()) << 32) | 0x9003);
1170 let key = SegmentKey::new(
1171 dataflow,
1172 NodeId::new("camera").unwrap(),
1173 DataId::new("image").unwrap(),
1174 1,
1175 );
1176 let config = SegmentConfig::new(4, 4096)
1177 .unwrap()
1178 .with_backing(Backing::Named);
1179 let Ok(segment) = Segment::create(key.clone(), config) else {
1180 return;
1181 };
1182 let segment = segment.shared();
1183 let mut producer = Producer::new(Arc::clone(&segment)).unwrap();
1184 producer.send(b"before", b"").unwrap();
1185
1186 let spec = ShmSegmentSpec::new(key.canonical(), 1, 4, 4096);
1187 let (_mapped, mut consumer) = open_consumer(dataflow, &source(), &spec, None).unwrap();
1188 assert!(
1189 matches!(consumer.try_next(), Err(RecvError::Empty)),
1190 "a message published before the attach is not replayed"
1191 );
1192 producer.send(b"after", b"").unwrap();
1193 let sample = consumer.next_blocking(Duration::from_secs(2)).unwrap();
1194 assert_eq!(sample.payload(), b"after");
1195
1196 drop(sample);
1197 drop(consumer);
1198 let _ = segment.unlink();
1199 }
1200}