ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! The write lane: what a LANDING node's own write is, and what one write says about the items it
//! was given.
//!
//! The seam is [`AsyncLands`], and it is deliberately the smallest thing the loop needs — *take these
//! items, tell me which of them reached the store*. Nothing about keys, tables, transactions or
//! backends crosses it: which key a record replaces is the LANE's business, and a lane is what an
//! emission fills in, so this layer never has to name an app's content to make a write durable.
//!
//! # Why the answer is per item
//!
//! One delivery is one write, because a page of 256 rows written as 256 statements is 256 round
//! trips where a store already offers one. A batch that goes as one node can still land as several:
//! a row that violates a constraint, an object whose address is too long, a document one shard
//! refuses. So the answer is a [`Commit`] — a POSITIVE set of positions, exactly as
//! [`Ack::released`](super::Ack) is a positive set of keys, and for the same reason: what a lane
//! does not name stays held, so a lane that reports nothing costs a re-delivery and never a loss.
//!
//! POSITIONS and not keys. The socket hands the lane a slice and reads the answer back against
//! that slice, so the pairing is the delivery's own order and nothing in this layer has to be able
//! to name what a record IS.
//!
//! # Why the seam is async and both engines still drive it
//!
//! Every store face this crate carries — [`AsyncBlobEngine::put`](crate::AsyncBlobEngine),
//! [`VectorEngine`](crate::VectorEngine) — hands back a future, so a lane that wraps one is async
//! whether or not this seam is. Making the
//! seam synchronous would move that fact nowhere: it would put a block inside every lane instead of
//! inside the one engine that needs it.
//!
//! So the seam is async, and the thread engine drives it with [`block_on`] — which is a DRIVER and
//! not a runtime. A resident thread already blocks in `park` and blocks for room on a full edge;
//! blocking it on its own write is the same posture rather than a new one.

use core::future::{Future, ready};

use std::{collections::HashSet, convert::Infallible};

/// What ONE [`AsyncLands::land`] call did: which of the items it was given reached the store, and —
/// where some did not — why.
///
/// The two constructors that report less than everything both REQUIRE the refusal, and that is the
/// seam's one demand on a lane: an item that is not committed holds the input it descends from, so
/// something has to be able to say what happened, and the alternative is a delivery that stalls with
/// nothing written down.
#[derive(Debug)]
pub struct Commit<E> {
  reached: Reached,
  refusal: Option<E>,
}

/// The committed set, as a shape rather than as a set.
///
/// [`All`](Self::All) is the ordinary answer and carries no allocation: a lane that took the whole
/// batch says so in a word, and a page of 256 costs nothing to report.
#[derive(Debug)]
enum Reached {
  /// Every item.
  All,
  /// Exactly these positions in the slice the lane was given.
  Only(HashSet<usize>),
}

impl<E> Commit<E> {
  /// Every item reached the store.
  #[inline]
  #[must_use]
  pub const fn all() -> Self {
    Self {
      reached: Reached::All,
      refusal: None,
    }
  }

  /// None did, and `refusal` is why — the shape a dead connection or a rolled-back transaction
  /// takes.
  #[inline]
  #[must_use]
  pub fn none(refusal: E) -> Self {
    Self {
      reached: Reached::Only(HashSet::new()),
      refusal: Some(refusal),
    }
  }

  /// Only the items at `positions` did, and `refusal` is why the rest did not.
  ///
  /// The positions index the slice the lane was handed. One refusal covers every item it did not
  /// name, because the answer a held item needs is that it is held — the per-item detail belongs to
  /// whatever the lane's own error already carries.
  #[must_use]
  pub fn only(positions: impl IntoIterator<Item = usize>, refusal: E) -> Self {
    Self {
      reached: Reached::Only(positions.into_iter().collect()),
      refusal: Some(refusal),
    }
  }

  /// Whether the item at `position` reached the store.
  #[inline]
  #[must_use]
  pub fn committed(&self, position: usize) -> bool {
    match &self.reached {
      Reached::All => true,
      Reached::Only(positions) => positions.contains(&position),
    }
  }

  /// Why the uncommitted items are uncommitted, or `None` when there are none.
  #[inline]
  #[must_use]
  pub const fn refusal(&self) -> Option<&E> {
    self.refusal.as_ref()
  }

  /// Take the refusal — what the socket does to put it in a [`Lifecycle`](super::Lifecycle).
  #[inline]
  #[must_use]
  pub fn into_refusal(self) -> Option<E> {
    self.refusal
  }
}

/// A delivery's write did not commit everything it staged.
///
/// It is the framework's own wrapping of a lane's refusal and it exists to carry the COUNT, which
/// the lane cannot state: the lane knows why one write failed, and the socket is what knows that six
/// of two hundred and fifty-six are held. The lane's own error is kept as the
/// [`source`](core::error::Error::source) rather than rendered into a string, so a tap that knows
/// the concrete type can still walk to it.
#[derive(Debug, derive_more::Display)]
#[display("the write lane committed {committed} of {staged}: {refusal}")]
pub struct Unwritten<E> {
  staged: usize,
  committed: usize,
  refusal: E,
}

impl<E> Unwritten<E> {
  /// `committed` of `staged` items reached the store, and `refusal` is why the rest did not.
  #[inline]
  #[must_use]
  pub const fn new(staged: usize, committed: usize, refusal: E) -> Self {
    Self {
      staged,
      committed,
      refusal,
    }
  }

  /// How many items the delivery handed the lane.
  #[inline]
  #[must_use]
  pub const fn staged(&self) -> usize {
    self.staged
  }

  /// How many of them reached the store.
  #[inline]
  #[must_use]
  pub const fn committed(&self) -> usize {
    self.committed
  }

  /// The lane's own refusal.
  #[inline]
  #[must_use]
  pub const fn refusal(&self) -> &E {
    &self.refusal
  }
}

impl<E> core::error::Error for Unwritten<E>
where
  E: core::error::Error + 'static,
{
  #[inline]
  fn source(&self) -> Option<&(dyn core::error::Error + 'static)> {
    Some(&self.refusal)
  }
}

/// A node's own write — the socket a landing consumer's durability comes through.
///
/// A [`Resident`](super::Resident) with a lane answers each delivery's oneshot AFTER this call
/// returns, and releases exactly the input items whose writes committed. That ordering is the whole
/// reason the seam exists, and it is the channel half of the layer's one law: a position advances
/// only after the effect it covers is durable.
///
/// # What it does not ask for
///
/// No key, no table, no transaction, no backend. `land` receives the records themselves and answers
/// by POSITION, so the framework never names an app's content — the keyed replace that makes a
/// replay harmless is the lane's, written where the record's type is known. That is the same
/// division [`Skip::about`](crate::Skip::about) states one layer up.
///
/// # ONE WRITE, TWO LAYERS — and never two write verbs at one layer
///
/// This is the SOCKET and [`FulltextWrite::upsert`](crate::FulltextWrite::upsert) /
/// [`VectorWrite::upsert`](crate::VectorWrite::upsert) /
/// [`AsyncBlobEngine::put`](crate::AsyncBlobEngine::put) are
/// the ENGINES. They are not two vocabularies for one act and they are not alternatives: a LANE
/// implements this trait and CALLS an engine verb, and that composition is the only shape in which a
/// node's items reach a store. What each layer answers is what makes the pair irreducible — a socket
/// answers *which of these items are durable*, positionally, over a batch the framework handed it;
/// an engine answers *this entry is written*, over one entry whose identity the emission decided.
/// Neither can be derived from the other, and adding an item verb to an engine or an entry verb here
/// would be the duplication this arrangement exists to refuse.
///
/// **The framework ships no lane, and that is measured rather than an omission.** A generic
/// `FulltextLane<E: FulltextEngine>` cannot be written: `land` takes `&mut self` and writes N items
/// through one write scope, so it must REBORROW `E::WriteContext<'_>` per item — and a GAT
/// projection over an unknown backend has no reborrow. The shipped backend's scope is
/// `&mut SlotWriter` and reborrows as `&mut *writer`; another backend's may be an owned lease, a
/// token, a `Copy` handle. Nothing generic covers the three. So a lane is written where the write
/// scope is CONCRETE, which is the same place the record's type is known and the same place the
/// flatten into the engine's entry lives — one author for the whole crossing. `store::tantivy`'s
/// tests build one end to end.
///
/// **For a SQL backend that concrete scope is a POOL, and the shape is measured rather than
/// preferred**: the lane holds a `&Pool` — the one form an HRTB measurement over sqlx's `Acquire`
/// passed — and acquires PER LAND, the pool reaching it as a shared resource the launch built rather
/// than as a connection this layer hands over. None of it is written here, for the paragraph above's
/// reason; it is recorded so that the first lane written against a pool is written the way the
/// measurement passed.
///
/// The name is the one thing here still open: see the pull request's rename-candidates section.
///
/// # `&mut self`
///
/// [`Positions`](super::Positions)' reason, and it is the same resource: a lane that matters holds a
/// connection, and a keyed replace over a batch is not atomic across two callers. Taking the
/// exclusive borrow makes that serialisation the compiler's rather than a comment, and it costs
/// nothing — a lane belongs to exactly one resident.
///
/// The future is `Send` because a resident is a task on somebody's runtime, and it is spelled as
/// `impl Future + Send` rather than as `async fn` so the bound is part of the contract instead of
/// something each caller has to hope for.
pub trait AsyncLands<T> {
  /// What a refused write reports.
  type Error;

  /// Commit `items`, and say which of them reached the store.
  ///
  /// This is the RUNTIME half of the word: [`Resident::land`](super::Resident::land) installs the
  /// lane, and the driver calls this on it once per batch.
  fn land(&mut self, items: &[T]) -> impl Future<Output = Commit<Self::Error>> + Send;
}

/// [`AsyncLands`]' BLOCKING twin — the write socket a sync body reaches for.
///
/// # It is a FACE of an engine, not a lesser lane
///
/// A `block`, `spawn` or `rayon` poll entry runs a body that has no `.await` to spend, so a store it
/// writes through has to answer without one. Whether a given store CAN is the deployment engine's
/// own fact and nothing about it is decidable from a declaration: a tantivy index writes through an
/// `IndexWriter` and answers here; an sqlx pool is async to its bottom and answers only through
/// [`AsyncLands`]. So the legality of *sync body writes to this store* is settled at the TYPE layer, at
/// the line that reached, and the grammar refuses nothing — which is why the emitted grants contexts
/// forward BOTH families and each forward is conditional on the handle's own impl.
///
/// # The method keeps the bare name
///
/// `land` on both faces, measured rather than assumed: a body that imports one trait calls `land`
/// with no ceremony, and a body that imports BOTH and calls `land` on a handle implementing both is
/// `E0034` — *multiple applicable items in scope* — naming the two candidates and offering the
/// disambiguating spelling. That refusal is a good one: it arises only where a deployment genuinely
/// has two roads to one store and the body has not said which it is on, and it names both. A
/// suffixed second name (`land_blocking`) would buy silence at that one site and charge every other
/// site a word that says nothing the trait's own name does not.
///
/// # `&mut self`
///
/// [`AsyncLands`]' reason verbatim, and it is the same resource: a keyed replace over a batch is not
/// atomic across two callers, so the exclusive borrow is what serialises it.
#[diagnostic::on_unimplemented(
  message = "`{Self}` has no BLOCKING write socket for `{T}`",
  label = "this write needs `Lands<{T}>`",
  note = "a sync poll entry (`block`, `spawn`, `rayon`) writes through `Lands`, and an async \
          one (`async(inline)`, `async(spawn)`) through `AsyncLands` — the handle a deployment installed \
          implements whichever roads its engine actually has, so either give this clock a handle \
          that lands without awaiting, or write the entry in the async world"
)]
pub trait Lands<T> {
  /// What a refused write reports.
  type Error;

  /// Commit `items`, and say which of them reached the store.
  fn land(&mut self, items: &[T]) -> Commit<Self::Error>;
}

/// The unused lane, blocking face — [`AsyncLands for ()`](AsyncLands)' twin, and the same absence of a claim.
impl<T> Lands<T> for () {
  type Error = Infallible;

  #[inline(always)]
  fn land(&mut self, _items: &[T]) -> Commit<Self::Error> {
    Commit::all()
  }
}

/// The unused lane — what a resident that declared none carries.
///
/// It is [`AsyncBlobEngine for ()`](crate::AsyncBlobEngine)'s face, one layer up and for the same reason: a resident
/// is generic over its lane, and the absence of one has to be a type. It commits everything, which
/// is not a claim about durability but the absence of a claim — a node with no lane never stages
/// anything through this seam that its own body did not already handle, and what makes its answer
/// true is [`Effect`](super::Effect) rather than this call.
impl<T> AsyncLands<T> for () {
  type Error = Infallible;

  #[inline(always)]
  fn land(&mut self, _items: &[T]) -> impl Future<Output = Commit<Self::Error>> + Send {
    ready(Commit::all())
  }
}

/// Drive `future` to completion on THIS thread — the thread engine's half of a write.
///
/// The two engines already differ in how they park and how they wait for room; a write is the third
/// difference, and this is the thread side of it. It is a DRIVER and not a runtime: one future, no
/// timer, no spawn, no reactor and no dependency — `std::task::Wake` makes the waker and
/// `std::thread::park` is the wait, so an idle poll costs nothing and a wake from another thread is
/// what resumes it.
///
/// The park/unpark race is std's to answer and it does: an `unpark` that arrives before the `park`
/// leaves a token, so the wait returns at once rather than sleeping through its own wake.
pub(super) fn block_on<F>(future: F) -> F::Output
where
  F: Future,
{
  block_until(future, None).expect("a wait with no deadline ends only when the future answers")
}

/// [`block_on`] with a `deadline` — the same driver, giving up where the caller said to.
///
/// `None` is the expiry. It is what the THREAD engine's PARK spends: a resident's sweep is a
/// timeout, and a node with SEVERAL sources waits in a select that only a future can express, so
/// the deadline has to live in the driver rather than inside a receive call. See
/// [`park_on`](crate::conduit::park_on), which is this with the caller's sweep and is where the
/// public face of the same driver lives.
///
/// The unfinished future is DROPPED at the deadline, and that loses nothing: a receive future that
/// goes away deregisters its waker and passes the wake on, while the message it was waiting for is
/// in the channel's own queue rather than in the future, so the next park sees it.
pub(crate) fn block_until<F>(future: F, deadline: Option<std::time::Instant>) -> Option<F::Output>
where
  F: Future,
{
  use core::task::{Context, Poll};
  use std::{
    sync::Arc,
    task::{Wake, Waker},
    thread::{Thread, current, park, park_timeout},
    time::Instant,
  };

  struct Unpark(Thread);

  impl Wake for Unpark {
    #[inline]
    fn wake(self: Arc<Self>) {
      self.0.unpark();
    }

    #[inline]
    fn wake_by_ref(self: &Arc<Self>) {
      self.0.unpark();
    }
  }

  let mut future = core::pin::pin!(future);
  let waker = Waker::from(Arc::new(Unpark(current())));
  let mut cx = Context::from_waker(&waker);
  loop {
    if let Poll::Ready(value) = future.as_mut().poll(&mut cx) {
      return Some(value);
    }
    let Some(deadline) = deadline else {
      park();
      continue;
    };
    // A spurious unpark polls again rather than sleeping through what is left, which is why the
    // remaining time is recomputed each turn: the wait ends at the ANSWER or at the clock, and
    // never at a wake that meant nothing.
    park_timeout(deadline.checked_duration_since(Instant::now())?);
  }
}

#[cfg(test)]
mod tests;