ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
//! WHAT THE FRAMEWORK SAYS ABOUT ITSELF: the level a declaration chose, the four spans, and the
//! shipped tap that turns every observation into an event.
//!
//! # Events ride the tap seam, and nothing here is a second emission point
//!
//! [`TracingTap`] is a [`LifecycleTap`](super::LifecycleTap) like any other. Every event it says was
//! already being said to whoever registered a tap — this one just says it in `tracing`'s vocabulary
//! — so the runtime has exactly one place where an observation is produced and observation has
//! exactly one place where it is consumed. A deployment that wants a journal row, a metric and a log
//! line registers three taps, and the runtime is unchanged by all three.
//!
//! That is also why the tap's `observe` carries no wildcard arm. Inside this crate
//! `#[non_exhaustive]` does not apply, so the match is EXHAUSTIVE on purpose: a later arm of
//! [`Observation`](super::Observation) stops this file from compiling, which is the only mechanism
//! that keeps *the shipped tap speaks the whole vocabulary* true rather than aspirational.
//!
//! # Spans are FOUR, and the list is closed
//!
//! A unit's batch, a query door's round trip, a land commit, and a poll tick. They are here because
//! they are the four scopes a tap CANNOT express: a tap is told that a thing happened, and a span is
//! the interval a thing was happening IN. Every other fact this layer has is an event, and an event
//! already has a seam.
//!
//! A fifth is not a small addition. Nesting is what a trace is read for, so each span here is a
//! level in every trace of every deployment; the list is closed so the shape of that tree is a
//! decision taken once rather than the sum of everywhere somebody thought a span would help.
//!
//! # Both engines span the SAME call
//!
//! Each of the four wraps exactly one call, and the thread engine's spelling and the async engine's
//! spelling wrap the same one. That is deliberate and it is the cheap half of a rule this layer
//! pays for elsewhere: two engines whose spans covered different work would make a trace mean
//! something different depending on which driver a node was declared with, and nothing in a
//! deployment's dashboard would say which.
//!
//! # Why a [`Scope`] and not a `tracing::Span`
//!
//! Every constructor below has ONE signature in both builds, and so do [`Scope::entered`] and
//! [`Scope::over`]. A build without the feature gets a zero-sized value whose two methods are the
//! identity, so the call sites in the engines' own loops are written once with no `#[cfg]` on them —
//! which matters for the reason above: a predicate there is a place the two engines could come to
//! disagree.
//!
//! The two methods are also what keeps a guard off the async path. A synchronous body ENTERS a scope
//! and a future is WRAPPED IN one; entering across an `.await` attributes whatever else the executor
//! runs on that thread to this node, and the type is what makes that spelling unavailable rather
//! than merely discouraged.

/// The LEVEL a node's spans open at — the closed vocabulary `tracing(level = …)` writes.
///
/// # A closed unit enum
///
/// The five are `tracing`'s own levels and they are the whole of what a level can be: this is a
/// framework vocabulary rather than a seam, so a sixth word would be a level `tracing` does not have
/// and could not filter on. That is the test every closed word in this framework is measured by, and
/// it is why this is an enum where a backend or an engine is a trait.
///
/// # It exists in every build
///
/// The type is ungated, because the WORD is: `tracing(level = debug)` parses whether or not this
/// build compiles any tracing at all, and a model has to be able to carry what it parsed.
/// [`Declared::TRACING`](super::Declared::TRACING) names this type in every build for the same
/// reason. What rides the feature is the EMISSION — the const an expansion writes, and the spans
/// this level opens — and never the vocabulary.
///
/// # The default is `Info`
///
/// A declaration that writes no `tracing(…)` word gets [`Info`](Self::Info), the level a reader who
/// installed a subscriber and set no filter sees. The framework's own spans are structural — one per
/// batch, one per commit — so the default is the level at which *this node is working* is worth
/// saying, and a node whose batches are a thousand a second is exactly the node whose declaration
/// writes `tracing(level = debug)`.
#[derive(
  Debug,
  Default,
  Clone,
  Copy,
  PartialEq,
  Eq,
  PartialOrd,
  Ord,
  Hash,
  derive_more::Display,
  derive_more::IsVariant,
)]
#[display("{}", self.as_str())]
pub enum TracingLevel {
  /// `tracing(level = trace)` — the finest, and off in every deployment that has not asked for it.
  Trace,
  /// `tracing(level = debug)` — what a node whose batches are frequent chooses.
  Debug,
  /// `tracing(level = info)`, and the DEFAULT.
  #[default]
  Info,
  /// `tracing(level = warn)`.
  Warn,
  /// `tracing(level = error)`.
  Error,
}

impl TracingLevel {
  /// Every level, in `tracing`'s own order — finest first.
  ///
  /// What a refusal's sentence is built from, so the vocabulary a reader is offered is the enum
  /// rather than a second copy of it that can fall behind while still compiling.
  pub const ALL: [Self; 5] = [
    Self::Trace,
    Self::Debug,
    Self::Info,
    Self::Warn,
    Self::Error,
  ];

  /// The word a declaration writes, and the word a refusal names.
  #[inline]
  #[must_use]
  pub const fn as_str(self) -> &'static str {
    match self {
      Self::Trace => "trace",
      Self::Debug => "debug",
      Self::Info => "info",
      Self::Warn => "warn",
      Self::Error => "error",
    }
  }

  /// The level this word names, or `None` where the word is not one of them.
  #[inline]
  #[must_use]
  pub fn named(word: &str) -> Option<Self> {
    Self::ALL.into_iter().find(|level| level.as_str() == word)
  }
}

#[cfg(feature = "tracing")]
impl From<TracingLevel> for tracing::Level {
  #[inline]
  fn from(level: TracingLevel) -> Self {
    match level {
      TracingLevel::Trace => Self::TRACE,
      TracingLevel::Debug => Self::DEBUG,
      TracingLevel::Info => Self::INFO,
      TracingLevel::Warn => Self::WARN,
      TracingLevel::Error => Self::ERROR,
    }
  }
}

/// One of the four spans, or nothing at all where this build compiles no observation.
///
/// It is a VALUE rather than a guard, so the two ways a scope is spent stay distinct — see the
/// module note for what that separation is load-bearing for.
#[derive(Debug)]
pub(super) struct Scope {
  #[cfg(feature = "tracing")]
  span: tracing::Span,
}

impl Scope {
  /// NO span at all — what a scope is where the thing it would have timed did not happen.
  ///
  /// `tracing::Span::none()` is a disabled span: entering it is a no-op, it reaches no subscriber
  /// and it appears in no trace. That is what keeps a call site in an engine loop unconditional
  /// while the DECISION about whether a scope is interesting stays with the constructor that knows
  /// — see [`commit`], the one that spends it.
  #[inline]
  #[must_use]
  fn none() -> Self {
    Self {
      #[cfg(feature = "tracing")]
      span: tracing::Span::none(),
    }
  }

  /// Enter it for the returned guard's life — the SYNCHRONOUS half, which the thread engine spends
  /// around its own blocking call.
  #[inline]
  #[must_use]
  pub(super) fn entered(self) -> Entered {
    Entered {
      #[cfg(feature = "tracing")]
      _span: self.span.entered(),
    }
  }

  /// Run `future` inside it — the ASYNC half, which follows the future across every yield instead of
  /// staying on the thread that polled it.
  #[cfg(feature = "tracing")]
  #[inline]
  pub(super) fn over<F: core::future::Future>(
    self,
    future: F,
  ) -> impl core::future::Future<Output = F::Output> {
    use tracing::Instrument as _;

    future.instrument(self.span)
  }

  /// The same door with nothing behind it: this build compiles no observation, so the future comes
  /// back as it arrived.
  #[cfg(not(feature = "tracing"))]
  #[inline]
  pub(super) fn over<F: core::future::Future>(
    self,
    future: F,
  ) -> impl core::future::Future<Output = F::Output> {
    future
  }
}

/// A [`Scope`] that is currently entered — the span exits where this is dropped.
#[derive(Debug)]
pub(super) struct Entered {
  #[cfg(feature = "tracing")]
  _span: tracing::span::EnteredSpan,
}

/// Open a span at a level that is not known until run time.
///
/// `tracing::span!` builds its metadata in a `static`, so the level has to be a constant expression
/// — which a declaration's word is not, at this crate's compile time. Five callsites and a match is
/// the shape that answer takes; one of them is reached per call, and the four others cost their
/// metadata in the binary and nothing at run time.
#[cfg(feature = "tracing")]
macro_rules! opened {
  ($level:expr, $name:expr, $($field:tt)*) => {
    match $level {
      TracingLevel::Trace => ::tracing::span!(::tracing::Level::TRACE, $name, $($field)*),
      TracingLevel::Debug => ::tracing::span!(::tracing::Level::DEBUG, $name, $($field)*),
      TracingLevel::Info => ::tracing::span!(::tracing::Level::INFO, $name, $($field)*),
      TracingLevel::Warn => ::tracing::span!(::tracing::Level::WARN, $name, $($field)*),
      TracingLevel::Error => ::tracing::span!(::tracing::Level::ERROR, $name, $($field)*),
    }
  };
}

/// A UNIT'S BATCH: the operator walk over one delivery, in either engine.
///
/// It closes before the write, which is [`commit`]'s scope — the two are siblings rather than nested
/// because a batch that stages nothing still ran, and what a commit is timing is the store.
#[cfg(feature = "tracing")]
#[inline]
#[must_use]
pub(super) fn batch(level: TracingLevel, node: &super::NodeName, items: usize) -> Scope {
  Scope {
    span: opened!(level, "ingraph.batch", node = node.as_str(), items),
  }
}

/// [`batch`]'s shut twin.
#[cfg(not(feature = "tracing"))]
#[inline]
#[must_use]
pub(super) fn batch(_level: TracingLevel, _node: &super::NodeName, _items: usize) -> Scope {
  Scope::none()
}

/// A QUERY DOOR'S ROUND TRIP: the arm answering one invoke.
///
/// The scope is the SERVE — what the responder took to produce an answer — and not the reply's
/// handoff after it, which is a send on a channel of capacity one. That is also the call both
/// engines can wrap identically, the async engine's handoff being a second await the thread
/// engine's does not have.
///
/// `query` is the arm's own running count rather than an identity carried on the wire, because there
/// is none: an [`Invoke`](super::Invoke) is an input and a reply channel, and the input is the app's
/// own type with no bound this layer could render it through.
#[cfg(feature = "tracing")]
#[inline]
#[must_use]
pub(super) fn door(level: TracingLevel, node: &super::NodeName, query: usize) -> Scope {
  Scope {
    span: opened!(level, "ingraph.door", node = node.as_str(), query),
  }
}

/// [`door`]'s shut twin.
#[cfg(not(feature = "tracing"))]
#[inline]
#[must_use]
pub(super) fn door(_level: TracingLevel, _node: &super::NodeName, _query: usize) -> Scope {
  Scope::none()
}

/// A LAND COMMIT: the durable write of one batch's staged rows.
///
/// The scope is the WRITE and not the release that follows it. What a reader of this span is asking
/// is how long the store took; the ack law's own work — which items the window releases, what the
/// journal row says — is arithmetic over an answer that has already arrived.
///
/// # A PURE node opens none
///
/// The driver calls `land` once per batch whatever the node is, and for a node with no lane that
/// call is [`AsyncLands for ()`](super::AsyncLands) answering `Commit::all()` without doing anything. Timing
/// it would put a commit span on every batch of every pure operator in a graph, which is the
/// pepper-spraying the narrow list exists to prevent — so the effect decides, and a node that
/// claims no durability has no commit to time.
#[cfg(feature = "tracing")]
#[inline]
#[must_use]
pub(super) fn commit(
  level: TracingLevel,
  node: &super::NodeName,
  effect: super::Effect,
  rows: usize,
) -> Scope {
  if effect.is_pure() {
    return Scope::none();
  }
  Scope {
    span: opened!(level, "ingraph.commit", node = node.as_str(), rows),
  }
}

/// [`commit`]'s shut twin.
#[cfg(not(feature = "tracing"))]
#[inline]
#[must_use]
pub(super) fn commit(
  _level: TracingLevel,
  _node: &super::NodeName,
  _effect: super::Effect,
  _rows: usize,
) -> Scope {
  Scope::none()
}

/// A POLL TICK: one sweep of one declared poll source.
///
/// `source` is the source's POSITION in the node's declaration, which is what a
/// [`PollUpstream`](super::PollUpstream) means — a node may declare several entries at several cadences,
/// and a tick that did not say which would be a tick nobody could act on.
#[cfg(feature = "tracing")]
#[inline]
#[must_use]
pub(super) fn tick(
  level: TracingLevel,
  node: &super::NodeName,
  source: super::PollUpstream,
) -> Scope {
  Scope {
    span: opened!(
      level,
      "ingraph.tick",
      node = node.as_str(),
      source = source.index()
    ),
  }
}

/// [`tick`]'s shut twin.
#[cfg(not(feature = "tracing"))]
#[inline]
#[must_use]
pub(super) fn tick(
  _level: TracingLevel,
  _node: &super::NodeName,
  _source: super::PollUpstream,
) -> Scope {
  Scope::none()
}

/// THE SHIPPED TAP: every observation, as a `tracing` event.
///
/// ```
/// use std::sync::Arc;
///
/// use ingraph::{LifecycleTap, Taps, TracingTap};
///
/// let taps = Taps::new().with(Arc::new(TracingTap::new()) as Arc<dyn LifecycleTap>);
/// assert_eq!(taps.len(), 1);
/// ```
///
/// # The LEVEL is the arm's, and there is no knob
///
/// A failure is an error, a shed item and a retry are warnings, a lapped tick is an info line, and
/// an arrival is a debug line — because that is what those things ARE, and a tap that let a
/// deployment call every arrival an error would have made the level useless as a filter. The
/// `tracing(level = …)` word governs a node's SPANS, which are intervals whose interest genuinely
/// varies by node; an event's severity is a property of what happened.
///
/// Where a deployment wants them quieter or louder, that is `tracing`'s own business and it is
/// better at it than a knob here would be: a filter takes a target and a level, and these events
/// carry the module path every other event in this framework carries.
///
/// # What each arm says
///
/// Every event carries `node`. Beyond that:
///
/// | arm | level | fields |
/// |---|---|---|
/// | [`Arrived`](Observation::Arrived), settled | `DEBUG` | `station` |
/// | [`Arrived`](Observation::Arrived) at [`Skipped`](super::Station::Skipped) | `DEBUG` | `station`, `reason`, `content` |
/// | [`Arrived`](Observation::Arrived) at [`Failed`](super::Station::Failed) | `ERROR` | `station`, `error` |
/// | [`Retrying`](Observation::Retrying) | `WARN` | `attempt`, `after` |
/// | [`Shed`](Observation::Shed) | `WARN` | `key`, `decision`, `queued`, `capacity` |
/// | [`GivenUp`](Observation::GivenUp), carrying a [`Failure`](super::Failure) | `WARN` | `terminal`, `attempt`, `error` |
/// | [`GivenUp`](Observation::GivenUp) decided from a bare error | `WARN` | `terminal`, `attempt` |
/// | [`Expired`](Observation::Expired) | `WARN` | `terminal`, `budget` |
/// | [`PolicyFailed`](Observation::PolicyFailed) | `ERROR` | `concern`, `error` |
/// | [`Lapped`](Observation::Lapped) | `INFO` | `poll` |
///
/// The two ENDINGS are warnings and not errors, which is the same judgement the shed already makes:
/// a deployment that configured a retry ceiling or a query budget asked for exactly this to happen,
/// so an ending is a thing that was decided rather than a thing that went wrong. What went wrong, if
/// anything did, is the `error` a give-up carries — and that reaches the same tap as an
/// [`Arrived`](Observation::Arrived) at [`Failed`](super::Station::Failed) already, at the level a
/// failure deserves.
///
/// Three of those fields are optional and are spelled as `Option`s — a skip's `content`, a retry's
/// `after`, a shed's `capacity` on an unbounded edge. Every one of them is DECLARED at the callsite
/// whatever the value is and RECORDED only when there is one, which is `tracing`'s own rule for an
/// `Option` rather than a choice made here: `Value for Option<T>` visits nothing at `None`. So a
/// subscriber's metadata always names the field and its visitor sees it only where the framework had
/// something to say — measured against `tracing-core` 0.1's `field.rs`, not assumed.
///
/// A give-up's `error` is NOT a fourth of them, and that is what the arm's TWO ROWS above say: the
/// match has two callsites rather than one field spelled as an `Option`, so a give-up decided from a
/// bare error does not name `error` in its metadata AT ALL. A subscriber can tell the two absences
/// apart — a declared field with no value is not the same event as a field that was never declared —
/// so the distinction is a reader's to know rather than an implementation detail, and one row
/// promising `error` on every give-up would have been the table saying otherwise.
///
/// [`Observation`]: super::Observation
#[cfg(feature = "tracing")]
#[derive(Debug, Default, Clone, Copy)]
pub struct TracingTap {
  // A unit struct would be the same size and a worse door: `TracingTap::new()` is how every other
  // shipped tap is built, and a type constructed as a literal is one that cannot grow a field
  // without breaking the call sites that spelled it.
  _private: (),
}

#[cfg(feature = "tracing")]
impl TracingTap {
  /// A tap that says everything it is shown.
  #[inline]
  #[must_use]
  pub const fn new() -> Self {
    Self { _private: () }
  }
}

#[cfg(feature = "tracing")]
impl super::LifecycleTap for TracingTap {
  /// EXHAUSTIVE, and the missing wildcard is the contract: a new arm of
  /// [`Observation`](super::Observation) fails to compile here, which is what makes the shipped
  /// tap's promise to say everything checkable rather than a claim in a doc.
  ///
  /// Every arm reads its fields DIRECTLY, and that is the ending arms' shape rather than a style: a
  /// terminal arm carries the terminal's own payload, so there is no inner value whose variant
  /// could disagree with the arm holding it and no branch here for a pairing nothing can mint.
  fn observe(&self, node: &super::NodeName, observation: &super::Observation) {
    let node = node.as_str();
    match observation {
      super::Observation::Arrived(arrival) => {
        let station = arrival.station().as_str();
        if let Some(failure) = arrival.failure() {
          tracing::error!(node, station, error = %failure);
        } else if let Some(skip) = arrival.skip() {
          tracing::debug!(node, station, reason = %skip.reason(), content = skip.content());
        } else {
          tracing::debug!(node, station);
        }
      }
      super::Observation::Retrying { attempt, after } => {
        tracing::warn!(node, attempt, after = after.map(|after| after.to_string()),)
      }
      super::Observation::Shed {
        key,
        decision,
        occupancy,
      } => tracing::warn!(
        node,
        key = key.rendered(),
        decision = decision.as_str(),
        queued = occupancy.queued(),
        capacity = occupancy.capacity(),
      ),
      super::Observation::GivenUp {
        attempt,
        failure: Some(failure),
      } => tracing::warn!(
        node,
        terminal = super::Terminal::GivenUp.as_str(),
        attempt,
        error = %failure,
      ),
      super::Observation::GivenUp { attempt, .. } => {
        tracing::warn!(node, terminal = super::Terminal::GivenUp.as_str(), attempt)
      }
      super::Observation::Expired { budget } => tracing::warn!(
        node,
        terminal = super::Terminal::Expired.as_str(),
        budget = %budget,
      ),
      super::Observation::PolicyFailed { concern, error } => {
        tracing::error!(node, concern = concern.as_str(), error = %error);
      }
      // INFO, alone in this match, and the level is the judgement rather than the volume: a lap is
      // the at-most-one-in-flight default doing what the declaration asked by declining
      // `concurrent`. Nothing failed and nothing was lost — the next interval asks again.
      super::Observation::Lapped { poll } => {
        tracing::info!(node, poll = poll.index());
      }
    }
  }
}

#[cfg(test)]
mod tests;