ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
//! What a consumer SAYS happened, and the open seam that observes it.

use std::{
  any::Any,
  sync::{Arc, Mutex},
};

use jiff::SignedDuration;

use crate::Skip;

use super::{
  NodeName,
  fate::Terminal,
  policy::{Concern, Occupancy, OverflowDecision, ShedKey},
  poll::PollUpstream,
};

/// Where one item got to, as its consumer states it.
///
/// # The stations are CLOSED and the info is OPEN
///
/// The four variants are the whole table, because the watermark condition MATCHES on them: a
/// framework that could not enumerate the stations could not decide whether an item is done. The
/// payload is the opposite — `I` is the consumer's own detail, and a batch ack is where it earns
/// its keep, since one answer covers many items and per-item detail has nowhere else to go.
///
/// # No default parameter
///
/// `Lifecycle<I>` and never `Lifecycle<I = ()>`. A default would let a consumer write
/// `Lifecycle::Processed(())` without noticing it had discarded the one slot the design gave it,
/// and the omission would be invisible at the call site. A consumer with nothing to say writes the
/// unit type on purpose.
///
/// # Zero bounds
///
/// `I` is bounded nowhere in this type, nowhere in [`Ack`](super::Ack), and nowhere in the
/// transport. The single bound is at [`AckSender::send`](super::AckSender::send), the erased
/// boundary, and it is `Any + Send + Sync` — what crossing an object-safe tap costs and nothing
/// more. A bound written here would be paid by every consumer whether or not its info ever reached
/// a tap.
/// # Why it CLONES
///
/// One outcome can be owed to SEVERAL senders. A keyed group's delivery answers the token of every
/// member that joined it — see [`Custody`](super::Custody) — and those answers state the same
/// station and the same detail, differing only in which keys each releases. So the station travels
/// by value to each of them, which is what `Clone` is here for. `I: Clone` is the caller's to
/// satisfy and the driver's own info type ([`Tally`](super::Tally)) is `Copy`; a one-token delivery
/// clones nothing.
#[derive(Debug, Clone, derive_more::IsVariant, derive_more::Unwrap, derive_more::TryUnwrap)]
#[unwrap(ref, ref_mut)]
#[try_unwrap(ref, ref_mut)]
pub enum Lifecycle<I> {
  /// The item's effect is durable in this consumer's own store. The station a landing consumer
  /// reaches, and the one that makes a channel ack safe to release: the oneshot fires AFTER the
  /// write, never before it.
  Persisted(I),
  /// The item was processed and this consumer lands nothing — a pure operator's station.
  Processed(I),
  /// The item was intentionally refused, as the operator states it. The refusal travels whole,
  /// because [`Skip`] already carries both halves an app needs: WHY, and — where the operator could
  /// say so — WHICH content.
  Skipped(Skip),
  /// The item could not be processed. It holds the window: a failed item is named in no
  /// [`Ack::released`](super::Ack::released) set, so the sender re-delivers it.
  ///
  /// The failure crosses ERASED for the reason `I` does — one tap observes every node in the
  /// arrangement and each carries its own `Operator::Error`.
  Failed(Failure),
}

impl<I> Lifecycle<I> {
  /// Which station this is, without the payload.
  #[inline]
  pub const fn station(&self) -> Station {
    match self {
      Self::Persisted(_) => Station::Persisted,
      Self::Processed(_) => Station::Processed,
      Self::Skipped(_) => Station::Skipped,
      Self::Failed(_) => Station::Failed,
    }
  }

  /// Cross the tap boundary: keep the station and the framework's own payloads, and hold the
  /// consumer's info as `Any` for a tap that knows what to ask for.
  ///
  /// This is where `I` acquires its only bound — see the type note.
  #[inline]
  #[must_use]
  pub fn erase(self) -> Observation
  where
    I: Any + Send + Sync,
  {
    Observation::Arrived(self.arrive())
  }

  /// The [`Arrival`] alone, without the observation wrapper — what the ack path carries and what a
  /// journal row is written from.
  #[inline]
  #[must_use]
  pub fn arrive(self) -> Arrival
  where
    I: Any + Send + Sync,
  {
    let station = self.station();
    let (info, skip, failure) = match self {
      Self::Persisted(info) | Self::Processed(info) => (
        Some(Box::new(info) as Box<dyn Any + Send + Sync>),
        None,
        None,
      ),
      Self::Skipped(skip) => (None, Some(Box::new(skip)), None),
      Self::Failed(failure) => (None, None, Some(failure)),
    };
    Arrival {
      station,
      info,
      skip,
      failure,
    }
  }
}

/// The station table on its own — what a watermark condition matches and what a tap counts.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, derive_more::Display, derive_more::IsVariant)]
#[display("{}", self.as_str())]
pub enum Station {
  /// [`Lifecycle::Persisted`].
  Persisted,
  /// [`Lifecycle::Processed`].
  Processed,
  /// [`Lifecycle::Skipped`].
  Skipped,
  /// [`Lifecycle::Failed`].
  Failed,
}

impl Station {
  /// A stable name for a tap, a journal and logs.
  #[inline]
  #[must_use]
  pub const fn as_str(&self) -> &'static str {
    match self {
      Self::Persisted => "persisted",
      Self::Processed => "processed",
      Self::Skipped => "skipped",
      Self::Failed => "failed",
    }
  }

  /// Whether an item at this station is DONE — the watermark condition, written once.
  ///
  /// A skip is done: an intentional refusal is a recorded gap and never a retry. A failure is not,
  /// which is what makes [`Failed`](Self::Failed) hold the window.
  #[inline]
  #[must_use]
  pub const fn is_settled(&self) -> bool {
    match self {
      Self::Persisted | Self::Processed | Self::Skipped => true,
      Self::Failed => false,
    }
  }
}

/// A consumer's failure, erased.
///
/// `Operator::Error` is the app's type and a tap observes every node at once, so the failure
/// crosses as the erased trait object the driver already bounds it to. The `source` is preserved
/// rather than rendered, so a tap that knows the concrete error can still walk the chain.
///
/// # `Arc` and not `Box`
///
/// A failure is OBSERVATIONAL and one failure can be owed to several senders — a keyed group's
/// delivery answers every member's token with the same station, see [`Custody`](super::Custody) —
/// so the erased error is shared rather than duplicated. Nothing else changes: the pointer is
/// still one indirection, [`error`](Self::error) still hands back the same `&dyn Error`, and no
/// caller can mutate what it borrows.
#[derive(Debug, Clone, derive_more::Display)]
#[display("{_0}")]
pub struct Failure(Arc<dyn core::error::Error + Send + Sync>);

impl Failure {
  /// Erase `error`.
  #[inline]
  #[must_use]
  pub fn new<E>(error: E) -> Self
  where
    E: core::error::Error + Send + Sync + 'static,
  {
    Self(Arc::new(error))
  }

  /// The erased error, for a tap that renders or walks it.
  #[inline]
  #[must_use]
  pub fn error(&self) -> &(dyn core::error::Error + Send + Sync + 'static) {
    &*self.0
  }
}

impl core::error::Error for Failure {
  #[inline]
  fn source(&self) -> Option<&(dyn core::error::Error + 'static)> {
    Some(&*self.0)
  }
}

/// One [`Lifecycle`] as a tap sees it: the station, the framework's own payloads, and the
/// consumer's info held for whoever knows its type.
///
/// [`info`](Self::info) is the round trip [`SkipReason::decode`](crate::SkipReason::decode) is for
/// reasons: the erasure is what lets ONE tap observe nodes carrying different info types, and the
/// typing belongs at the ends. A tap that does not recognise the type gets `None` and still has the
/// station, which is what the framework's own journal is written against.
///
/// It is ONE arm of [`Observation`] and the only one an ack can carry: a consumer answers about an
/// ITEM, and the framework's own events — a retry, a shed, a mad advisor — are about no item that
/// any consumer has an answer for.
#[derive(Debug)]
pub struct Arrival {
  station: Station,
  info: Option<Box<dyn Any + Send + Sync>>,
  // BOXED, and measured rather than stylistic: a `Skip` carries two `Utf8Bytes` — the reason's own
  // label and the content it named — which is ~136 bytes inline, and an `Observation` travels inside
  // every ack. Boxing it puts the cost on the refusal path, which is the uncommon one, and keeps the
  // ack's own enum small enough that `clippy::large_enum_variant` has nothing to report.
  skip: Option<Box<Skip>>,
  failure: Option<Failure>,
}

impl Arrival {
  /// Which station.
  #[inline]
  pub const fn station(&self) -> Station {
    self.station
  }

  /// The consumer's own info, as `I` — `None` when this station carries none, or when `I` is not
  /// the type the consumer sent.
  #[inline]
  pub fn info<I: Any>(&self) -> Option<&I> {
    self.info.as_ref().and_then(|info| info.downcast_ref())
  }

  /// The refusal, at [`Station::Skipped`].
  #[inline]
  pub fn skip(&self) -> Option<&Skip> {
    self.skip.as_deref()
  }

  /// The failure, at [`Station::Failed`].
  #[inline]
  pub const fn failure(&self) -> Option<&Failure> {
    self.failure.as_ref()
  }
}

/// EVERYTHING A TAP IS SHOWN: an item's arrival at a station, and the framework's own events beside
/// it.
///
/// # One enum, and no second tap kind
///
/// The policy seam produces five things worth watching that are not an item's station — a retry
/// scheduled, a retry given up on, an item shed at a full edge, a deadline elapsed, an advisor that
/// failed — and every one of them arrives HERE rather than through a tap trait of its own. One
/// family, one composition ([`Taps`]), one registration surface: a deployment that wired a tap is
/// watching its policies without having wired a second thing.
///
/// # Three of those arms are TERMINALS, and the same three an ack can answer
///
/// [`Shed`](Self::Shed), [`GivenUp`](Self::GivenUp) and [`Expired`](Self::Expired) are the endings
/// a delivery can reach without a consumer ever stating a station, and the vocabulary is
/// [`Fate`](super::Fate)'s — one wording, spoken on the delivery's own oneshot and here. That is
/// what makes an account balance: every delivery a sender put out either arrived somewhere, ended
/// at one of these three, or is still in flight, and a tap can see all of it.
///
/// The three also share ONE SHAPE, which is [`Fate::erase`](super::Fate::erase)'s whole rule: a
/// terminal arm is its fate arm's own payload MINUS the extra message — same arm name, same field
/// names, same field types, and `message` left behind because that half of a fate is owed to the
/// ack rather than to a tap. An arm holding a whole `Fate` instead would hold a value whose variant
/// the arm has already named, which is the ending written down twice and therefore a place it can
/// be written down wrong: every matcher that reached the payload would carry a branch for a pairing
/// nothing can mint.
///
/// `#[non_exhaustive]`, so the arms a later concern brings are a widening: a tap outside this crate
/// carries a wildcard and keeps compiling through it.
///
/// # WHOSE event it is takes no field
///
/// The node's name reaches a tap on [`LifecycleTap::observe`]'s own parameter, which every
/// observation already travels under. A `node` field beside it would be a second name for one fact,
/// and the two could disagree.
#[derive(Debug)]
#[non_exhaustive]
pub enum Observation {
  /// ONE ITEM reached a station, as its consumer states it — the erasure of a [`Lifecycle<I>`].
  Arrived(Arrival),
  /// A [`RetryPolicy`](super::RetryPolicy) asked for another attempt, and the framework will make
  /// it after `after`.
  ///
  /// `after` is the policy's own backoff and `None` is *at once*: the decision is the hook's and
  /// the sleeping is the framework's, which is what keeps a policy from being able to park the loop
  /// that consults it.
  Retrying {
    /// Which attempt has just failed — the first is `1`.
    attempt: u32,
    /// How long the framework will wait before the next one.
    after: Option<SignedDuration>,
  },
  /// An item was DROPPED at a full edge, because an [`OverflowPolicy`](super::OverflowPolicy) said
  /// to.
  ///
  /// The obituary carries the item's key and never the item: taps OBSERVE and never act, so
  /// carrying the value would open a resurrection door through a tap, copy bulk data at exactly the
  /// moment the machine is under pressure, and add nothing to the three things a shed record is
  /// read for — count it, alarm on it, re-derive it by key.
  Shed {
    /// The COMPLETE item key, and never the root id alone — see [`ShedKey`].
    key: ShedKey,
    /// Which shed this was.
    decision: OverflowDecision,
    /// How full the edge was when the policy answered.
    occupancy: Occupancy,
  },
  /// A policy's own hook returned `Err`, and the framework fell back to the shipped default.
  ///
  /// The record is the ONLY surface a sick policy has: falling back keeps the data path healthy,
  /// which is exactly what makes the sickness invisible everywhere else. The error keeps its type
  /// rather than a rendering, because that is what tells a configuration mistake from a bug. How
  /// loud a flood of these is allowed to be is the tap CONSUMER's business; this contract stays
  /// dumb.
  PolicyFailed {
    /// Which of the four concerns advised.
    concern: Concern,
    /// What its hook said.
    error: crate::BoxedError,
  },
  /// A DEADLINE elapsed before the answer arrived — [`Fate::Expired`](super::Fate::Expired) without
  /// its message.
  ///
  /// The caller of the door that timed out is told
  /// [`QueryError::TimedOut`](super::QueryError::TimedOut) at the same moment. Two views of one
  /// truth: the asker learns its own question went unanswered, and the arrangement learns that one
  /// of its nodes ran out of budget — and neither of those is derivable from the other, since a
  /// caller cannot name the node and the node cannot name the caller.
  Expired {
    /// How long the wait was allowed to be.
    budget: SignedDuration,
  },
  /// A [`RetryPolicy`](super::RetryPolicy) stopped trying — [`Fate::GivenUp`](super::Fate::GivenUp)
  /// without its message.
  ///
  /// [`Retrying`](Self::Retrying)'s missing half. A scheduled retry has been announced since the
  /// policy seam landed and an exhausted one was announced nowhere, which made *the framework gave
  /// up on this item* the one decision a deployment could not watch.
  GivenUp {
    /// Which attempt was the last — the first is `1`.
    attempt: u32,
    /// What it failed with, where the failure travelled.
    ///
    /// `None` where the give-up was decided from a bare error rather than from a [`Failure`], for
    /// the reason [`Fate::GivenUp`](super::Fate::GivenUp) states.
    failure: Option<Failure>,
  },
  /// A POLL TICK WAS SKIPPED because the previous tick of that clock is still running.
  ///
  /// The at-most-one-in-flight default of the sync-detached modes (`spawn`, `rayon`), spoken so it
  /// is watchable. It is INFO and not a warning: the deployment asked for exactly this by declining
  /// the `concurrent` word, and a lap is the configured behaviour working — the same judgement that
  /// puts a decided ending at info rather than at alarm. What a STREAM of them means is the
  /// consumer's to decide, and it is worth deciding: a clock lapping every interval is a cadence
  /// faster than its own work.
  ///
  /// It carries its own payload rather than borrowing another arm's, because nothing else here is
  /// about a clock: the poll's POSITION is what the node's `<Node>Due` translates back into the word
  /// the declaration wrote.
  Lapped {
    /// Which of the node's clocks lapped — the position its ticks arrive under.
    poll: PollUpstream,
  },
}

impl Observation {
  /// The arrival this observation is, or `None` where it is one of the framework's own events.
  #[inline]
  #[must_use]
  pub const fn arrival(&self) -> Option<&Arrival> {
    match self {
      Self::Arrived(arrival) => Some(arrival),
      _ => None,
    }
  }

  /// [`Arrival::station`], where this observation is one — `None` where it is a framework event,
  /// which reaches no station.
  #[inline]
  #[must_use]
  pub const fn station(&self) -> Option<Station> {
    match self {
      Self::Arrived(arrival) => Some(arrival.station()),
      _ => None,
    }
  }

  /// Which TERMINAL this observation is, or `None` where it is not an ending at all.
  ///
  /// The counterpart of [`station`](Self::station), and the pair is what an account is counted
  /// with: a station means an item arrived somewhere, a terminal means a delivery ended nowhere,
  /// and an observation that is neither — a scheduled retry, a mad advisor — moved nothing either
  /// way.
  #[inline]
  #[must_use]
  pub const fn terminal(&self) -> Option<Terminal> {
    match self {
      Self::Shed { .. } => Some(Terminal::Shed),
      Self::GivenUp { .. } => Some(Terminal::GivenUp),
      Self::Expired { .. } => Some(Terminal::Expired),
      _ => None,
    }
  }

  /// [`Arrival::info`], where this observation is one.
  #[inline]
  pub fn info<I: Any>(&self) -> Option<&I> {
    self.arrival().and_then(Arrival::info)
  }

  /// [`Arrival::skip`], where this observation is one.
  #[inline]
  pub fn skip(&self) -> Option<&Skip> {
    self.arrival().and_then(Arrival::skip)
  }

  /// [`Arrival::failure`], where this observation is one.
  #[inline]
  pub fn failure(&self) -> Option<&Failure> {
    self.arrival().and_then(Arrival::failure)
  }
}

/// The OPEN observation seam: everything that happens to an item, offered to whoever asked.
///
/// The framework ships taps of its own and an app adds its own at launch; they compose, because
/// multi-destination is the ruling rather than a fallback — a journal in the database, a tracing
/// span and an app's metric are three destinations for one event and none of them is the others'
/// transport.
///
/// Object-safe on purpose. A tap generic over the info type could observe exactly one node's
/// vocabulary, which is strictly less than what an arrangement needs — hence [`Observation`] and
/// its downcast rather than a parameter here.
///
/// `Send + Sync` because a tap is shared by every resident, and residents are threads and tasks at
/// once. Nothing in the seam is `async`: an observation must never be able to backpressure the loop
/// that produced it, so a tap that writes to a store queues the write and returns.
pub trait LifecycleTap: Send + Sync {
  /// Observe `observation`, produced by the node named `node`.
  fn observe(&self, node: &NodeName, observation: &Observation);
}

/// Several taps as one — the composition the multi-destination ruling needs.
///
/// It is itself a [`LifecycleTap`], so a composite nests and a driver holds exactly one seam
/// whether an app registered none, one or five.
#[derive(Clone, Default)]
pub struct Taps {
  taps: Vec<Arc<dyn LifecycleTap>>,
}

impl Taps {
  /// No taps. Observing through it is a walk over an empty slice.
  #[inline]
  #[must_use]
  pub const fn new() -> Self {
    Self { taps: Vec::new() }
  }

  /// Add `tap`, keeping registration order — which is the order observations reach them.
  #[inline]
  #[must_use]
  pub fn with(mut self, tap: Arc<dyn LifecycleTap>) -> Self {
    self.taps.push(tap);
    self
  }

  /// How many taps are registered.
  #[inline]
  #[must_use]
  pub fn len(&self) -> usize {
    self.taps.len()
  }

  /// Whether none is.
  #[inline]
  #[must_use]
  pub fn is_empty(&self) -> bool {
    self.taps.is_empty()
  }
}

impl core::fmt::Debug for Taps {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    // A tap is a trait object with no `Debug` supertrait — asking for one would bound every app's
    // tap for the benefit of this line — so the composite reports its arity, which is the fact a
    // reader of a driver's debug output is actually after.
    f.debug_struct("Taps")
      .field("len", &self.taps.len())
      .finish()
  }
}

impl LifecycleTap for Taps {
  #[inline]
  fn observe(&self, node: &NodeName, observation: &Observation) {
    for tap in &self.taps {
      tap.observe(node, observation);
    }
  }
}

/// A tap that keeps what it saw, for a test or a probe.
///
/// It ships because every walk in this layer needs one and a test-local copy in each would be the
/// same twenty lines five times. What it records is the pair a journal is keyed by — the node
/// and the station — plus the rendering of whatever the observation named, which is what a human
/// reads when an assertion fails.
#[derive(Debug, Default)]
pub struct RecordingTap {
  seen: Mutex<Vec<(String, Station)>>,
}

impl RecordingTap {
  /// An empty recorder.
  #[inline]
  #[must_use]
  pub const fn new() -> Self {
    Self {
      seen: Mutex::new(Vec::new()),
    }
  }

  /// Everything observed so far, in order.
  #[inline]
  #[must_use]
  pub fn seen(&self) -> Vec<(String, Station)> {
    self.seen.lock().expect("the recording tap's lock").clone()
  }

  /// How many observations reached this tap.
  #[inline]
  #[must_use]
  pub fn len(&self) -> usize {
    self.seen.lock().expect("the recording tap's lock").len()
  }

  /// Whether none did.
  #[inline]
  #[must_use]
  pub fn is_empty(&self) -> bool {
    self.len() == 0
  }
}

impl LifecycleTap for RecordingTap {
  /// ARRIVALS only, and the wildcard is the contract rather than an omission: what this recorder is
  /// for is the pair a journal is keyed by, and a framework event reaches no station to be keyed
  /// by. A walk watching for a retry, a shed or a mad advisor writes a tap that names the arm it is
  /// after — which is what the wildcard leaves room for.
  fn observe(&self, node: &NodeName, observation: &Observation) {
    let Observation::Arrived(arrival) = observation else {
      return;
    };
    self
      .seen
      .lock()
      .expect("the recording tap's lock")
      .push((node.as_str().to_owned(), arrival.station()));
  }
}

#[cfg(test)]
mod tests;