ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
//! **THE RETIRED LEDGER**: the durable truth about which root keys are OUT, and the seam its
//! engines are written against.
//!
//! The registry one module over is a coordination CACHE — its own note says so — and this is what it
//! is a cache OF. A restarted deployment holds no state at all until something reads these rows, so
//! everything here is written to be read back: one table per ROOT FAMILY, one row per retired key,
//! and a `roll` that hands the whole family over at boot.
//!
//! ```text
//! __ig_retired_media
//! ┌───────────────┬────────────────┬────────────────────┬──────────┬──────────────┐
//! │ root          │ retired_second │ retired_nanosecond │ origin   │ late_tails   │
//! ├───────────────┼────────────────┼────────────────────┼──────────┼──────────────┤
//! │ <the key>     │ 1756…          │ 0                  │ vanished │ 0            │
//! └───────────────┴────────────────┴────────────────────┴──────────┴──────────────┘
//! ```
//!
//! # ROOT GRAIN, and it is the whole shape rather than a simplification
//!
//! There is no row-level anything here, and no per-store marker: a pipeline is ONE UNIT, so the
//! grain a deletion and a restore both have is the ROOT. That is what makes retire *one insert* and
//! restore *one delete* — a statement each, whatever a root's content grew to — and it is why no
//! record table in this framework carries a `deleted_at`.
//!
//! # THE FOUR VERBS, and why they are exactly four
//!
//! [`RetiredStore`] is [`ready`](RetiredStore::ready), [`record`](RetiredStore::record),
//! [`erase`](RetiredStore::erase) and [`roll`](RetiredStore::roll), and each one is a
//! [`Retirement`](crate::Retirement) or a boot read and nothing else:
//!
//! | verb | who calls it | what the state machine was saying |
//! |------|--------------|-----------------------------------|
//! | `ready` | the boot road, once per family | — the table has to exist before a row can |
//! | `record` | [`Keeper`] | [`Retirement::Recorded`](crate::Retirement::Recorded) |
//! | `erase` | [`Keeper`] | [`Retirement::Erased`](crate::Retirement::Erased) |
//! | `roll` | the boot road, once per family | — the cache is empty and the truth is here |
//!
//! Nothing else is on it, and the absences are deliberate. There is no *is this key retired* —
//! that question is the MIRROR's ([`RetiredMirror`]), answered without touching a database at all —
//! and no *list the retired keys matching …*, because the read side's own filtering is a later
//! knife with its own shape.
//!
//! # `record` IS AN UPSERT, and the update is ORIGIN-ONLY
//!
//! [`state`](crate::drive::registry::state)'s own arm decides this and this only mirrors it. A
//! retirement recorded a second time for a key already retired is a STRONGER ORIGIN restating who
//! stood the key down — `at` is when it went out, which did not change, and `late_tails` is a fact
//! about the drain that already happened, which did not change either. So the conflict clause sets
//! `origin` and touches nothing else, and the row keeps saying what it always said about the
//! retirement itself.
//!
//! # THE ORDERING: durable write FIRST, publish AFTER
//!
//! [`Keeper`] is where that lives and its own note argues it. What matters here is what the shape
//! makes possible: a store's verbs are awaited, and the mirror's publish is a separate act after the
//! await, so a crash between them leaves the LEDGER ahead of the mirror — the truth ahead of the
//! cache, which is the direction a restart repairs for free.
//!
//! # ENGINES ARE A SEAM, and the four here are its first occupants
//!
//! [`RetiredStore`] is a trait, not an enum. The four the framework ships — SQLite, PostgreSQL,
//! MySQL and MongoDB — are the four the graph word admits today; a KV store, a file, a service is
//! the same trait implemented elsewhere, and nothing in this module is written against a list of
//! backends. What IS closed is the WORD `retired(<engine>)` a declaration writes, because that word
//! is the framework's own vocabulary for the engines it can build unaided — see
//! [`RetiredEngine`].

use core::{future::Future, pin::Pin};

use std::sync::Arc;

use smol_bytes::Utf8Bytes;

use crate::{Timestamp, drive::registry::RetireOrigin};

mod keeper;
mod mirror;
// The DRIVER's feature and not the vocabulary's: `mongo` supplies `bson`, which is the document
// vocabulary a record's own row crosses into, and a store needs the client that talks to a server.
#[cfg(feature = "mongodb")]
mod mongo;
mod options;
#[cfg(any(feature = "sqlite", feature = "postgres", feature = "mysql"))]
mod sql;

#[cfg(test)]
mod tests;

pub use keeper::{Keeper, KeeperBody, KeeperReport, rebuild};
pub use mirror::{RetiredMirror, RetiredReader, RetiredReaders};
#[cfg(feature = "mongodb")]
pub use mongo::MongoRetired;
pub use options::{RetiredEngine, RetiredOptions, UnknownRetiredEngine};
#[cfg(any(feature = "sqlite", feature = "postgres", feature = "mysql"))]
pub use sql::{RetiredSql, SqlRetired};

/// The prefix every retired-ledger table's name carries — the framework's own reserved stem.
///
/// `__ig_` is what every identifier this framework mints rather than reads off a declaration wears,
/// one module over: a companion column, a derived index, an unnested alias. A ledger table is the
/// same kind of thing — nobody declared it, and the prefix is what says so to a person reading a
/// schema.
///
/// # It is PINNED to the reserved namespace, at compile time
///
/// The emission-prefix seat's own shape, taken the way a RUNTIME mint can take it. A ledger table's
/// name is not emitted: families are opened rather than declared, so the name is minted here, at run
/// time, from a family this crate learns at run time — and `EmissionPrefix` is `ingraph-codegen`'s
/// type, which this crate cannot name. What the seat asks of a mint is that the mark be ONE value
/// rather than a copy per site, and the catalog's own DDL prefix already answers that the only way
/// a runtime literal can: it states the bytes a reader can see whole and ASSERTS the agreement, so
/// a namespace that moves without this moving is a compile error rather than a schema that
/// disagrees with itself.
///
/// This is the SAME pin, over the same checker rather than a second copy of it — see
/// [`leads_with`](crate::query::sql). It rides `sqlx` because that is where the reserved namespace
/// const lives, which is every build that can put this name into a statement at all.
///
/// In the seat's own vocabulary the shape is `prefixed`, not `framed`: a ledger table is a minted
/// OBJECT rather than a synthesized column, so it wears the mark in front and no `__` behind.
pub const RETIRED_TABLE_PREFIX: &str = "__ig_retired_";

/// The stem this prefix adds to the reserved namespace — named so the pin below can measure it
/// rather than restate a number.
const RETIRED_STEM: &str = "_retired_";

#[cfg(feature = "sqlx")]
const _: () = assert!(
  crate::query::sql::leads_with(RETIRED_TABLE_PREFIX, crate::comparison::sql::ALIAS_PREFIX)
    && RETIRED_TABLE_PREFIX.len()
      == crate::comparison::sql::ALIAS_PREFIX.len() + RETIRED_STEM.len(),
  "the retired ledger's table prefix is the reserved namespace plus `_retired_`"
);

/// THE FUTURE A STORE'S VERB HANDS BACK — one act, landing.
///
/// Named for what it is rather than for what is owed, [`Owed`](crate::Owed) being a different
/// thing in this same namespace: an owed answer on an edge.
///
/// Boxed, and that is what a seam reached through `Arc<dyn RetiredStore<Root>>` costs: an
/// `impl Future` return is not dyn-safe, and the party that holds a store holds it erased — one
/// [`Keeper`] serves whatever engine the declaration named, and the ledger is a
/// once-per-retirement act rather than a hot path, so one allocation per act buys the erasure and
/// costs nothing anybody can measure.
pub type Landing<'a, T> = Pin<Box<dyn Future<Output = Result<T, RetiredStoreError>> + Send + 'a>>;

/// ONE ROW OF THE LEDGER: which key went out, when, on whose say-so, and whether anything was still
/// in flight when it did.
///
/// The three facts are exactly [`RootState::Retired`](crate::RootState::Retired)'s own fields, and
/// that is not a coincidence to be maintained — it is what makes the boot road total. A row read back reconstructs the state it was written from with nothing missing and
/// nothing invented, which is the whole of what *the registry is a cache and this is the truth*
/// means when a process restarts.
///
/// # There is NO late-arrival column, and the absence is a ruling
///
/// `late_tails` is FROZEN AT RETIREMENT. It says *work for this key may land after this row was
/// written*, which is a fact about the drain that ended — and a second column tracking whether such
/// work then actually arrived would be a row rewritten by the very tails it is warning about, from
/// every worker that had one out. What reads the flag is the sweep, and *look* is a complete
/// instruction.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct RetiredRow<Root> {
  /// The root key that went out.
  key: Root,
  /// When it went out — WALL time, because this is the reading a person opening the table wants.
  at: Timestamp,
  /// Who stood it down, and therefore whether a reappearance brings it back.
  origin: RetireOrigin,
  /// Whether it went out with answers still owed — see the type note for why nothing updates it.
  late_tails: bool,
}

impl<Root> RetiredRow<Root> {
  /// The row a retirement earns.
  #[inline]
  #[must_use]
  pub const fn new(key: Root, at: Timestamp, origin: RetireOrigin, late_tails: bool) -> Self {
    Self {
      key,
      at,
      origin,
      late_tails,
    }
  }

  /// The root key that went out.
  #[inline]
  #[must_use]
  pub const fn key(&self) -> &Root {
    &self.key
  }

  /// When it went out.
  #[inline]
  #[must_use]
  pub const fn at(&self) -> Timestamp {
    self.at
  }

  /// Who stood it down.
  #[inline]
  #[must_use]
  pub const fn origin(&self) -> &RetireOrigin {
    &self.origin
  }

  /// Whether it went out with answers still owed.
  #[inline]
  #[must_use]
  pub const fn late_tails(&self) -> bool {
    self.late_tails
  }

  /// The key, taken out of the row — what a boot rebuild wants once it has read the rest.
  #[inline]
  #[must_use]
  pub fn into_key(self) -> Root {
    self.key
  }
}

/// **WHAT A LEDGER COULD NOT DO** — one refusal, carrying whatever the engine said underneath.
///
/// One type rather than an associated `Error`, and the seam is what decides it: a
/// `Arc<dyn RetiredStore<Root>>` cannot project an associated type, and the party holding one — the
/// [`Keeper`] — has exactly one thing to do with a failure whatever engine produced it, which is to
/// report it and leave the mirror alone. So the engine's own error is a SOURCE rather than a
/// variant, reachable by a caller that knows which engine it built.
#[derive(Debug)]
pub struct RetiredStoreError {
  /// Which verb was refused — `record`, `erase`, `roll`, `ready`.
  verb: &'static str,
  /// The table it was refused on.
  table: Utf8Bytes,
  /// What the engine said.
  source: Box<dyn core::error::Error + Send + Sync + 'static>,
}

impl RetiredStoreError {
  /// The refusal an engine's own error becomes.
  #[must_use]
  pub fn new(
    verb: &'static str,
    table: impl Into<Utf8Bytes>,
    source: impl core::error::Error + Send + Sync + 'static,
  ) -> Self {
    Self {
      verb,
      table: table.into(),
      source: Box::new(source),
    }
  }

  /// Which verb was refused.
  #[inline]
  #[must_use]
  pub const fn verb(&self) -> &'static str {
    self.verb
  }

  /// The table it was refused on.
  #[inline]
  #[must_use]
  pub fn table(&self) -> &str {
    self.table.as_str()
  }
}

impl core::fmt::Display for RetiredStoreError {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    write!(
      f,
      "the retired ledger refused `{verb}` on `{table}`",
      verb = self.verb,
      table = self.table,
    )
  }
}

impl core::error::Error for RetiredStoreError {
  fn source(&self) -> Option<&(dyn core::error::Error + 'static)> {
    Some(self.source.as_ref())
  }
}

/// **THE ENGINE SEAM**: where one root family's retirement rows actually live.
///
/// Four verbs, argued in the [module note](self). An implementation is written per ENGINE and per
/// family — a `RetiredStore<MediaId>` is the ledger for `media` and knows its own table name — so
/// nothing here takes a family argument and nothing here is generic over more than the key.
///
/// # It is a SEAM, and the four shipped engines are its first occupants rather than its definition
///
/// A deployment that keeps its ledger somewhere this framework has never heard of writes this trait
/// and hands the result to [`Keeper::new`]. What such a store owes is exactly the four verbs and the
/// two orderings the [`Keeper`]'s note names; nothing about SQL, about a pool or about a dialect
/// reaches this declaration.
///
/// # Every verb is FALLIBLE, including the ones a caller cannot retry
///
/// A ledger that refused a `record` leaves the registry saying *retired* over a row that does not
/// exist, which a restart un-says. That is the honest outcome and it is why the refusal is
/// reported rather than swallowed — see [`KeeperReport`], which is where a deployment learns that
/// its cache and its truth have parted company.
pub trait RetiredStore<Root>: Send + Sync {
  /// **THE TABLE EXISTS** — create it if it does not, and answer once it does.
  ///
  /// Idempotent, and called on the boot road before [`roll`](Self::roll). A store whose home needs
  /// no creation — a document collection, a service — answers `Ok` having done nothing, which is
  /// the honest implementation rather than a stub.
  ///
  /// # Errors
  ///
  /// [`RetiredStoreError`], carrying whatever the engine said.
  fn ready(&self) -> Landing<'_, ()>;

  /// **ONE INSERT** — `row`'s key is retired as of `row`'s stamp.
  ///
  /// An UPSERT on the key, whose update sets the ORIGIN and nothing else: see the module note,
  /// where the state machine's own arm is what decides it.
  ///
  /// # Errors
  ///
  /// [`RetiredStoreError`], carrying whatever the engine said.
  fn record<'a>(&'a self, row: &'a RetiredRow<Root>) -> Landing<'a, ()>;

  /// **ONE DELETE** — `key` is not retired any more.
  ///
  /// Total on a key the ledger does not hold: erasing a row that is not there is the state the
  /// caller wanted, so it is `Ok` rather than a refusal. That is what makes a restore replayable
  /// after a crash between the delete and whatever was going to observe it.
  ///
  /// # Errors
  ///
  /// [`RetiredStoreError`], carrying whatever the engine said.
  fn erase<'a>(&'a self, key: &'a Root) -> Landing<'a, ()>;

  /// **EVERY ROW** — the whole family, for the rebuild that a boot is.
  ///
  /// Read-all rather than paged, and the shape is the grain's: a row per RETIRED ROOT is a
  /// deployment's retirees rather than its content, so the count is the number of things somebody
  /// stood down and never the number of records they held.
  ///
  /// # Errors
  ///
  /// [`RetiredStoreError`], carrying whatever the engine said.
  fn roll(&self) -> Landing<'_, Vec<RetiredRow<Root>>>;
}

/// A store held behind a pointer is a store — so a [`Keeper`] may be handed either.
impl<Root, S> RetiredStore<Root> for Arc<S>
where
  S: RetiredStore<Root> + ?Sized,
{
  #[inline]
  fn ready(&self) -> Landing<'_, ()> {
    S::ready(self)
  }

  #[inline]
  fn record<'a>(&'a self, row: &'a RetiredRow<Root>) -> Landing<'a, ()> {
    S::record(self, row)
  }

  #[inline]
  fn erase<'a>(&'a self, key: &'a Root) -> Landing<'a, ()> {
    S::erase(self, key)
  }

  #[inline]
  fn roll(&self) -> Landing<'_, Vec<RetiredRow<Root>>> {
    S::roll(self)
  }
}

/// **THE TABLE ONE FAMILY'S ROWS LIVE IN** — its name, and the five column words every engine
/// spells the same.
///
/// A value rather than an emitted type, and the reason is where a family comes from: families are
/// OPENED rather than declared (see [`Registries`](crate::Registries)), so the set of ledger tables
/// a deployment has is the set of families something attributed work to — which is a runtime fact.
/// What the graph word decides is the ENGINE; what the family decides is the NAME.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct RetiredTable {
  /// The family — `media`, `track` — as [`Attributes::FAMILY`](crate::Attributes::FAMILY) spells it.
  family: Utf8Bytes,
  /// `__ig_retired_<family>`, minted once so every statement reads the same string.
  name: Utf8Bytes,
}

impl RetiredTable {
  /// The column the root key itself is held in.
  pub const ROOT: &'static str = "root";

  /// The column holding the retirement's whole SECONDS — [`Timestamp`]'s own storage shape, which
  /// this framework already spells as two integers everywhere else a timestamp is a column.
  pub const RETIRED_SECOND: &'static str = "retired_second";

  /// The column holding the retirement's SUB-SECOND NANOSECONDS — the other half of the pair.
  pub const RETIRED_NANOSECOND: &'static str = "retired_nanosecond";

  /// The column holding [`RetireOrigin`]'s own word.
  pub const ORIGIN: &'static str = "origin";

  /// The column holding whether the key went out with answers still owed.
  pub const LATE_TAILS: &'static str = "late_tails";

  /// The table `family`'s rows live in.
  ///
  /// # It REFUSES a family it cannot name, and the refusal is the SQL audit
  ///
  /// A table name is SPLICED into every statement this ledger runs — an identifier cannot be bound
  /// — so the one place a caller-supplied string could reach a statement is here, and here is where
  /// it is stopped. `family` must be a bare identifier ([`slot`](crate::slot)'s own rule: ASCII
  /// `[A-Za-z0-9_]`, not digit-leading), which admits no quote, no comment, no semicolon and no
  /// whitespace — so a name this door accepted cannot end a string literal or start a second
  /// statement.
  ///
  /// It is a [`Result`] rather than an assertion for the reason the changes feed's own note gives
  /// one door over: a `const` refusal is right for a declaration and wrong for a name that reaches a
  /// store at run time. The families a graph actually opens come from
  /// [`Attributes::FAMILY`](crate::Attributes::FAMILY), which is a snake-cased Rust type name and
  /// therefore always accepted; this refusal is for the caller that built one by hand.
  ///
  /// # Errors
  ///
  /// [`UnnameableFamily`], carrying what was written.
  pub fn new(family: impl Into<Utf8Bytes>) -> Result<Self, UnnameableFamily> {
    let family = family.into();
    if !crate::slot::is_identifier(family.as_bytes()) {
      return Err(UnnameableFamily { family });
    }
    let name = Utf8Bytes::from(format!("{RETIRED_TABLE_PREFIX}{family}"));
    Ok(Self { family, name })
  }

  /// The family this table holds.
  #[inline]
  #[must_use]
  pub fn family(&self) -> &str {
    self.family.as_str()
  }

  /// `__ig_retired_<family>` — the identifier every statement names.
  #[inline]
  #[must_use]
  pub fn name(&self) -> &str {
    self.name.as_str()
  }

  /// The name, as the shared byte string the statements were built over.
  #[inline]
  #[must_use]
  pub fn named(&self) -> Utf8Bytes {
    self.name.clone()
  }
}

/// A family whose spelling cannot become a table name — see [`RetiredTable::new`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UnnameableFamily {
  /// What was written.
  family: Utf8Bytes,
}

impl UnnameableFamily {
  /// What was written.
  #[inline]
  #[must_use]
  pub fn family(&self) -> &str {
    self.family.as_str()
  }
}

impl core::fmt::Display for UnnameableFamily {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    write!(
      f,
      "`{family}` cannot name a retired-ledger table — a family is a bare identifier (ASCII        letters, digits and `_`, not digit-leading), because the table name is spliced into every        statement and an identifier cannot be bound",
      family = self.family,
    )
  }
}

impl core::error::Error for UnnameableFamily {}