ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! **THE CURATOR**: the party that occupies the registry's flip seat, and the one place the sealed
//! ordering is performed.
//!
//! [`GenerationLedger`](crate::GenerationLedger)'s contract is *queue the act and return* — it is
//! called from inside a resident's own loop, with the transition that closed a run just made, and an
//! implementation that blocked there would stop the worker whose last answer earned the flip. So the
//! seat is a CHANNEL and the work is a BODY: [`Curator`] is what the registry holds, [`CuratorBody`]
//! is what a runtime spends, and everything durable happens in the second.
//!
//! # THE ORDERING, and it is the whole reason this type exists
//!
//! ```text
//!   registry says CLOSED ──► Curator::flipped ──► store.point(row).await ──► store.sweep(…).await
//!                            (queue, return)      THE DURABLE FLIP           THE RETENTION DUTY
//! ```
//!
//! **The pointer lands FIRST, and nothing is erased until it has.** Both halves are asserted by the
//! shape rather than by discipline: the sweep is written after an `await` whose value is CHECKED, so
//! a refused flip cannot fall through to a `DELETE`, and a crash anywhere between them leaves a
//! family holding MORE generations than its retention allows.
//!
//! That direction is chosen and not incidental, and it is the retirement ledger's own argument with
//! the second half turned destructive. Extra generations are collected by the next flip's sweep for
//! free. A sweep that ran FIRST would delete rows the pointer still names, and no restart repairs
//! that: there is no second copy of a deleted generation to re-read, and the pointer would be naming
//! a run that no longer has any rows at all.
//!
//! # THE CEILING IS THE RUN THAT JUST LANDED, not the newest run in the table
//!
//! [`GenerationsStore::sweep`](super::GenerationsStore::sweep) takes the flip's OWN run as the
//! highest generation retention may count from, and this is the party that passes it. That is what
//! makes a concurrent re-analysis safe: its half-written rows are above the ceiling, so they are
//! neither counted towards `keep` nor deleted by it.
//!
//! # A REFUSED FLIP SWEEPS NOTHING, and says so
//!
//! A store that would not take the pointer leaves the closure reading its previous generation, which
//! is a deployment one run behind rather than one that lost a run — an honest state, and one the next
//! flip corrects. What is NOT done is the sweep: a retention bound spent against a pointer that never
//! moved would delete generations in favour of one nothing is looking at. The refusal is counted into
//! [`CuratorReport`] instead.

use std::sync::{
  Arc, RwLock,
  atomic::{AtomicU64, Ordering},
};

use crate::drive::registry::{Flip, GenerationLedger, Registry, RootKey};

use super::{
  Culled, CurrentRow, GenerationsOptions, GenerationsOverride, GenerationsStore,
  GenerationsStoreError, Scope,
};

/// **HOW THE GENERATIONS SEAT HAS BEEN GETTING ON** — three counts, read at any moment.
///
/// [`refused`](Self::refused) is the one that matters. Zero is a deployment whose pointers are where
/// its runs left them; anything else is the number of times a closure did not move, or moved and was
/// not collected behind.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct CuratorReport {
  /// Pointers written — runs that closed and were flipped to.
  pub flipped: u64,
  /// Rows the retention duty removed, across every table of the closure.
  pub swept: u64,
  /// Acts the engine refused, at either half. Nothing was erased for any of them.
  pub refused: u64,
}

/// The counters behind [`CuratorReport`], shared between the seat and its body.
#[derive(Debug, Default)]
struct Counters {
  flipped: AtomicU64,
  swept: AtomicU64,
  refused: AtomicU64,
}

impl Counters {
  /// The three, read together.
  ///
  /// `Relaxed` throughout, and it is the honest ordering: these are a REPORT, published by nothing
  /// and ordering nothing. A reader that catches one count a moment before another has read a
  /// deployment mid-act, which is what reading a live counter means.
  fn report(&self) -> CuratorReport {
    CuratorReport {
      flipped: self.flipped.load(Ordering::Relaxed),
      swept: self.swept.load(Ordering::Relaxed),
      refused: self.refused.load(Ordering::Relaxed),
    }
  }
}

/// **THE SEAT**: what [`Registry::curating`](crate::Registry::curating) is handed, and all it ever
/// does is queue.
///
/// Cheap to hold and cheap to call — a `send` on an unbounded line and a clone of a root key, which
/// is an id. The body on the other end is where a database is touched.
///
/// # The line is UNBOUNDED, and the grain is why
///
/// Backpressure exists to stop a producer outrunning a consumer, and the producer here is a CLOSED
/// RUN: one act per root whose analysis finished. A deployment that closes a million runs in a burst
/// has just finished re-analysing a million roots, which a bounded queue would turn into a stalled
/// resident rather than a slow flip — and stalling the resident is exactly what this seat's contract
/// forbids.
///
/// # NOT [`Clone`], and the omission is what makes the body's ending certain
///
/// [`CuratorBody::run`] ends when every sender on its line is gone, and the one seat that exists
/// lives in the family's own [`Registry`]. A clone would be a second sender somewhere else, holding a
/// body open over a store that had already been dropped — so the seat is minted once, handed to
/// [`Registry::curating`](crate::Registry::curating), and never copied.
pub struct Curator<Root> {
  /// Where a closed run goes. See the type note for why it is unbounded.
  owed: flume::Sender<(Root, Flip)>,
  /// The retention this seat is running, shared with the body that spends it — see
  /// [`apply_knobs`](Self::apply_knobs) for the tier this lock exists for.
  options: Arc<RwLock<GenerationsOptions>>,
  /// The counts, shared with the body that moves them.
  counters: Arc<Counters>,
}

impl<Root> core::fmt::Debug for Curator<Root> {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.debug_struct("Curator")
      .field("queued", &self.owed.len())
      .field("report", &self.counters.report())
      .finish()
  }
}

impl<Root> Curator<Root>
where
  Root: RootKey,
{
  /// **THE PAIR**: the seat a registry holds, and the body a runtime spends.
  ///
  /// `scope` is the closure this curator flips — see [`Scope`], where the grain is argued. One
  /// curator flips one closure, because one flip covering two closures would be a re-run of one
  /// invalidating the other.
  #[must_use]
  pub fn new(
    store: Arc<dyn GenerationsStore<Root>>,
    scope: Scope,
    options: GenerationsOptions,
  ) -> (Self, CuratorBody<Root>) {
    let (owed, taken) = flume::unbounded();
    let counters = Arc::new(Counters::default());
    let options = Arc::new(RwLock::new(options));
    let seat = Self {
      owed,
      options: Arc::clone(&options),
      counters: Arc::clone(&counters),
    };
    let body = CuratorBody {
      taken,
      store,
      scope,
      options,
      counters,
    };
    (seat, body)
  }

  /// **SEAT A CURATOR ON `registry`**, handing back the body to spend — or [`None`] where a curator
  /// is already seated there.
  ///
  /// [`Registry::curating`](crate::Registry::curating)'s refusal, forwarded whole: a second writer
  /// for one family's pointers is two parties disagreeing about which run is current, and the honest
  /// answer to the second is *no*. A caller that gets `None` back has a body it never built rather
  /// than one it must remember not to run.
  #[must_use = "the body is what performs the flips; a dropped one leaves the pointers untouched"]
  pub fn seated(
    registry: &Arc<Registry<Root>>,
    store: Arc<dyn GenerationsStore<Root>>,
    scope: Scope,
    options: GenerationsOptions,
  ) -> Option<CuratorBody<Root>> {
    let (seat, body) = Self::new(store, scope, options);
    registry
      .curating(Arc::new(seat) as Arc<dyn GenerationLedger<Root>>)
      .then_some(body)
  }

  /// **TURN THE LIVE DIAL**: fold `over`'s named knobs onto the retention this curator is running.
  ///
  /// The options layer's THIRD TIER, and the same fold
  /// [`Registry::apply_knobs`](crate::Registry::apply_knobs) performs one module over: a knob `over`
  /// leaves unnamed is unchanged. A depth raised here is spent by the NEXT sweep — the one now in
  /// flight is already bound — which is what makes the dial a policy change rather than a retraction
  /// of a delete already issued.
  pub fn apply_knobs(&self, over: &GenerationsOverride) {
    let mut current = self
      .options
      .write()
      .unwrap_or_else(|held| held.into_inner());
    if let Some(keep) = over.keep() {
      current.set_keep(keep);
    }
  }

  /// The retention this curator is running, AS OF THIS READ — a snapshot rather than a borrow,
  /// because [`apply_knobs`](Self::apply_knobs) may have moved on by the time a caller looks at what
  /// came back.
  #[must_use]
  pub fn options(&self) -> GenerationsOptions {
    self
      .options
      .read()
      .unwrap_or_else(|held| held.into_inner())
      .clone()
  }

  /// How the seat has been getting on — see [`CuratorReport`].
  #[inline]
  #[must_use]
  pub fn report(&self) -> CuratorReport {
    self.counters.report()
  }

  /// How many flips are waiting for the body.
  #[inline]
  #[must_use]
  pub fn queued(&self) -> usize {
    self.owed.len()
  }
}

/// **QUEUE AND RETURN** — the whole of what the seat does on the transition's own thread.
///
/// A send onto an unbounded line cannot block and cannot fail except by the body being gone, which is
/// a deployment already shutting down: an act with nowhere to go is dropped, and the registry it came
/// from is a cache that is about to be dropped too.
impl<Root> GenerationLedger<Root> for Curator<Root>
where
  Root: RootKey,
{
  #[inline]
  fn flipped(&self, key: &Root, act: Flip) {
    let _gone = self.owed.send((key.clone(), act));
  }
}

/// **THE BODY**: what actually writes, and the only party that erases.
///
/// Spend it on a runtime — `R::spawn_detach(body.run())`, or an ignition's own `spawn` where the join
/// is wanted. It ENDS when every [`Curator`] onto its line is gone, which is the same punctuation the
/// registry driver honours one module over: *shutdown is the receive failing because every sender has
/// been dropped.*
pub struct CuratorBody<Root: RootKey> {
  /// The far end of the seat's line.
  taken: flume::Receiver<(Root, Flip)>,
  /// Where the pointers go, and what the retention duty runs against.
  store: Arc<dyn GenerationsStore<Root>>,
  /// Which closure this body flips.
  scope: Scope,
  /// The retention, shared with the seat that dials it.
  options: Arc<RwLock<GenerationsOptions>>,
  /// The counts, shared with the seat that reports them.
  counters: Arc<Counters>,
}

impl<Root: RootKey> core::fmt::Debug for CuratorBody<Root> {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.debug_struct("CuratorBody")
      .field("queued", &self.taken.len())
      .field("scope", &self.scope)
      .field("report", &self.counters.report())
      .finish()
  }
}

impl<Root: RootKey> CuratorBody<Root> {
  /// **THE LOOP**: one flip at a time, until the line ends.
  ///
  /// Serial by construction, and that is the correct shape rather than a limitation. Two flips for
  /// ONE root must not race — a later run's pointer overtaken by an earlier one's would leave the
  /// closure reading a generation the deployment had already moved off, and the sweep behind it would
  /// then collect the newer one — and the registry has already serialised them into the order they
  /// closed in. Running them concurrently would be discarding that order at the one point it becomes
  /// durable.
  pub async fn run(self) {
    while let Ok((key, act)) = self.taken.recv_async().await {
      self.perform(&key, act).await;
    }
  }

  /// **ONE FLIP**: the pointer lands, and only then is anything collected behind it.
  ///
  /// Public because it is the unit the ordering is stated in, and therefore the unit a test asserts
  /// the ordering over. A caller driving it by hand is driving the same body [`run`](Self::run) does,
  /// one act at a time.
  pub async fn perform(&self, key: &Root, act: Flip) {
    let row = CurrentRow::new(key.clone(), self.scope.clone(), act.run(), act.at());
    // THE DURABLE HALF. A refusal returns, and what it returns from is everything below — see the
    // module note for why a flip that did not land must not be swept behind.
    if self.refused(self.store.point(&row).await) {
      return;
    }
    self.counters.flipped.fetch_add(1, Ordering::Relaxed);
    // THE RETENTION DUTY, second and never first, with the run that just landed as its ceiling.
    let keep = self.keep();
    match self.store.sweep(key, act.run(), keep).await {
      Ok(Culled { rows, .. }) => {
        self.counters.swept.fetch_add(rows, Ordering::Relaxed);
      }
      refusal => {
        // The FLIP still stands: it is durable, and a sweep that did not run leaves a family holding
        // more generations than its retention allows, which the next flip collects.
        let _counted = self.refused(refusal.map(|_swept| ()));
      }
    }
  }

  /// The retention as of now — read under the seat's own lock, once per act.
  fn keep(&self) -> core::num::NonZeroUsize {
    self
      .options
      .read()
      .unwrap_or_else(|held| held.into_inner())
      .keep()
  }

  /// Whether the act was refused — counted here so that both halves report it the same way, and
  /// `true` meaning *stop, erase nothing*.
  fn refused(&self, outcome: Result<(), GenerationsStoreError>) -> bool {
    let Err(refusal) = outcome else {
      return false;
    };
    self.counters.refused.fetch_add(1, Ordering::Relaxed);
    // The refusal is COUNTED and dropped rather than propagated, and that is what the seat's own
    // contract leaves available: `flipped` returns nothing, so there is nobody up the stack to hand a
    // `Result` to. What a deployment reads instead is `CuratorReport`, and what it reads under
    // `tracing` is this line.
    #[cfg(feature = "tracing")]
    {
      // The SOURCE is logged beside the refusal, not instead of it: `Display` on this error names
      // the verb and the table by convention and leaves the chain to the reader, and a log line is
      // the one reader that cannot walk one.
      let engine = core::error::Error::source(&refusal).map(ToString::to_string);
      tracing::error!(
        verb = refusal.verb(),
        table = refusal.table(),
        error = %refusal,
        engine = engine.as_deref().unwrap_or("—"),
        "the generations store refused an act; nothing was erased"
      );
    }
    #[cfg(not(feature = "tracing"))]
    let _unreported = refusal;
    true
  }

  /// Which closure this body flips.
  #[inline]
  #[must_use]
  pub const fn scope(&self) -> &Scope {
    &self.scope
  }

  /// How the seat has been getting on — see [`CuratorReport`].
  #[inline]
  #[must_use]
  pub fn report(&self) -> CuratorReport {
    self.counters.report()
  }
}

/// **THE BOOT ROAD**: the pointer table exists, and this is how many closures it is already pointing
/// for.
///
/// Two acts, in order:
///
/// 1. [`ready`](super::GenerationsStore::ready) — the table exists, because a `roll` over a table
///    that was never created is not an empty seat, it is an error.
/// 2. [`roll`](super::GenerationsStore::roll) — every pointer, counted.
///
/// Nothing is rebuilt into memory, and the absence is the seat's own shape rather than an omission:
/// there is no cache of *which run is current* anywhere in this process, the read path answering that
/// question inside its own plan. See the [module note](super) — visibility is the write.
///
/// # Errors
///
/// [`GenerationsStoreError`] from either act. A deployment that cannot read its pointers has no
/// honest generation to start on, so the refusal is handed back rather than absorbed — unlike a
/// refusal DURING a run, where there is no caller left to hand one to.
pub async fn prepare<Root>(
  store: &dyn GenerationsStore<Root>,
) -> Result<usize, GenerationsStoreError>
where
  Root: RootKey,
{
  store.ready().await?;
  store.roll().await.map(|pointers| pointers.len())
}