ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! Tier B: the [`AsyncGather`] seam a batch of keys crosses to entities through, the [`RetrieveError`]
//! the two halves of a retrieve refuse in, and the fill loop that composes them.

use core::future::Future;

use super::{Folded, Ranked};

/// The store side of a retrieve: a batch of keys, the caller's filter, and the entities that
/// survive both.
///
/// # It is a SEAM rather than an emitted statement, and that is what keeps the tier ungated
///
/// The composed query is `WHERE <row-value keys predicate> AND <the user's comparison>` — the
/// second leg of *search → keys → hydrate*, which is a shape the SQL family already renders. What
/// it cannot be is EMITTED at a derived index's declaration: a derived index names no database (it
/// stores no rows), the upstream's row types belong to another expansion that this one cannot see,
/// and an expansion that named `sqlx` would make a declaration in a build with no SQL family fail
/// to compile. So the composition is written ONCE behind this trait — which is the shape the
/// retrieve face was ruled with, the orchestration living in one framework-internal core and the
/// emitted methods delegating to it — and the framework ships the SQL implementation of it.
///
/// A third party implements this for its own store: a document database, a key-value cache, an
/// in-memory corpus in a test. Nothing about the retrieve pipeline changes when one arrives.
///
/// # The answer is POSITIONAL, which is what restores the engine's order
///
/// One slot per key, in the order the keys were given. Engine order is the list's order and there
/// is no `ORDER BY` anywhere on this path — a ranked list re-sorted by a column would be a ranked
/// list with the ranking thrown away.
///
/// [`None`] is *this key produced no entity*, and it covers TWO cases on purpose:
///
/// - the user's filter pruned the row;
/// - the store has no row at that key at all — the engine knows an address the system of record has
///   since lost.
///
/// They are one case here because they are one case to the caller: the page is short by one and the
/// fill loop pulls another key. Distinguishing them would be a channel for a condition nobody can
/// act on differently.
///
/// # No `Send` on the returned future
///
/// The store faces demand it because a graph's drivers are multi-threaded. A gather is called from
/// wherever the caller's retrieve runs, so requiring it here would refuse a perfectly good
/// single-threaded store for a property this path never spends.
pub trait AsyncGather<K, Up> {
  /// The predicate this store takes — the node's own filter, at whatever type the store binds it.
  ///
  /// `?Sized` so a store may take an unsized predicate; an implementation naming a concrete
  /// comparison ignores the relaxation.
  type Filter: ?Sized;

  /// Why the store could not answer.
  type Error;

  /// The entities of `keys` that satisfy `filter`, ONE call per batch, positionally.
  ///
  /// The keys predicate and the filter are ONE question asked once — a store that ran them as two
  /// round trips would be paying twice for a composition this seam exists to make free.
  fn gather(
    &mut self,
    keys: &[K],
    filter: Option<&Self::Filter>,
  ) -> impl Future<Output = Result<Vec<Option<Up>>, Self::Error>>;
}

/// Why a retrieve did not answer: the ENGINE refused, or the STORE did.
///
/// Two parameters and no flattening, because the two halves are two backends a caller chose
/// independently and each already has an error vocabulary of its own. A single boxed error here
/// would take a tantivy `QueryParserError` — which tells a client its query text was malformed —
/// and a `SqlQueryError` — which tells an operator a bind ceiling was hit — and make them one thing
/// nobody can match on.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RetrieveError<E, G> {
  /// The index refused the query.
  Engine(E),
  /// The system of record refused the gather.
  Gather(G),
}

impl<E: core::fmt::Display, G: core::fmt::Display> core::fmt::Display for RetrieveError<E, G> {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Self::Engine(error) => write!(f, "the index refused the query: {error}"),
      Self::Gather(error) => write!(f, "the store refused the gather: {error}"),
    }
  }
}

impl<E, G> core::error::Error for RetrieveError<E, G>
where
  E: core::error::Error + 'static,
  G: core::error::Error + 'static,
{
  fn source(&self) -> Option<&(dyn core::error::Error + 'static)> {
    match self {
      Self::Engine(error) => Some(error),
      Self::Gather(error) => Some(error),
    }
  }
}

/// The FILL LOOP: ranked keys in, ranked entities out, one composed query per batch.
///
/// # What it does, in the order it does it
///
/// The ranked keys arrive already floored, deduped and stamped — tier A's whole answer, unbounded,
/// because it is this loop's own filter that decides how many of them become results. Then:
///
/// 1. take the next batch of keys — as many as the page still needs;
/// 2. ONE call to [`AsyncGather::gather`], which is ONE statement: the keys predicate AND the filter;
/// 3. keep the slots that came back filled, each wearing the score and rank its KEY carried;
/// 4. if the page is still short, go back to (1).
///
/// The loop ends at `limit`, or when the keys run out. The keys run out for two reasons and the
/// loop cannot tell them apart, which is correct: either the engine's stream was exhausted, or the
/// floor cut it. Both mean *there is no more above-quality material*, and both legitimately leave a
/// page UNDER-FULL. 宁短不凑 — a short final page is the ruling, not a case to pad around.
///
/// # The batch is what the page still needs, and that is the whole sizing rule
///
/// No constant, no knob. An unfiltered retrieve fills in one round trip; a filter that prunes half
/// a batch asks for the shortfall and no more. The alternative — a fixed batch — is a number that
/// is too big for the first case and too small for the second, chosen once for every corpus.
///
/// # Rank and score come from the KEY, never from the row
///
/// [`Ranked::map_node`] is the one move, so the numbers on an entity are the engine's answer about
/// its address. A row carries no ranking and inventing one from its position in the page would
/// re-densify exactly the holes tier A was careful to leave.
pub async fn filled<K, S, Up, G>(
  ranked: Vec<Ranked<K, S>>,
  store: &mut G,
  filter: Option<&G::Filter>,
  limit: usize,
) -> Result<Vec<Ranked<Up, S>>, G::Error>
where
  G: AsyncGather<K, Up>,
{
  // THE CARRY IS THE HIT WITH ITS NODE TAKEN OUT, so the numbers never leave the type that owns
  // them: what travels beside the keys is a `Ranked<(), S>`, and `map_node` is the one move that
  // puts an entity back where the key was. A carry spelled as a bare tuple would be the two numbers
  // re-assembled by hand at the rejoin, which is exactly what `map_node` exists to prevent.
  hydrated(
    ranked,
    |hit| {
      let (score, rank, key) = hit.into_components();
      (Ranked::new(score, rank, ()), key)
    },
    Ranked::map_node,
    store,
    filter,
    limit,
  )
  .await
}

/// The FILL LOOP AT THE SET GRAIN: [`filled`] over a [`Folded`] answer instead of a [`Ranked`] one.
///
/// Every word of [`filled`]'s note holds — the batch sizing, the positional contract, the short
/// final page, and the rule that the numbers come from the KEY and never from the row. The two
/// share [`hydrated`] rather than repeating the loop, because a loop written twice is a loop that
/// can drift at one of them, and the only difference between these two roads is WHAT rides beside
/// the key: two numbers on the single-vector road, and a folded score plus `m` per-stream sightings
/// here.
///
/// # What is carried is the whole provenance, not a re-derivation
///
/// [`Folded::map_node`] is the one move, exactly as [`Ranked::map_node`] is one road over: the
/// folded score and every [`StreamBest`](super::StreamBest) slot are the fold's answer about the
/// ADDRESS, and a row found under that address has nothing to say about either. A hit whose row the
/// system of record no longer holds drops out here, and the ordinals its slots carried stay holes in
/// the streams they came from — which is the same shape tier B already leaves in a `rank`.
pub async fn folded_filled<K, S, Up, G>(
  folded: Vec<Folded<K, S>>,
  store: &mut G,
  filter: Option<&G::Filter>,
  limit: usize,
) -> Result<Vec<Folded<Up, S>>, G::Error>
where
  G: AsyncGather<K, Up>,
{
  hydrated(
    folded,
    |hit: Folded<K, S>| {
      let (score, streams, key) = hit.into_components();
      (Folded::new(score, streams, ()), key)
    },
    Folded::map_node,
    store,
    filter,
    limit,
  )
  .await
}

/// THE FILL LOOP ITSELF — the batching, the positional read and the page cut, generic over what
/// rides beside a key.
///
/// Private, and the seam is deliberately a PAIR OF FUNCTIONS rather than a trait: the two callers
/// are this module's own, the split and the rejoin are two lines each, and a trait would put a
/// public vocabulary in front of an abstraction that exists to keep one `while` loop from being
/// written twice.
///
/// `C` is the hit WITH ITS NODE TAKEN OUT — a `Ranked<(), S>` on one road and a `Folded<(), S>` on
/// the other — and nothing in this body reads it. That is the property that makes the sharing safe:
/// the loop cannot pick a number up from the wrong hit, because it cannot see a number at all, and
/// the rejoin is each answer's own `map_node` rather than a constructor called with parts.
async fn hydrated<H, C, K, Out, Up, G>(
  hits: Vec<H>,
  split: impl Fn(H) -> (C, K),
  rejoin: impl Fn(C, Up) -> Out,
  store: &mut G,
  filter: Option<&G::Filter>,
  limit: usize,
) -> Result<Vec<Out>, G::Error>
where
  G: AsyncGather<K, Up>,
{
  // Split ONCE: the seam takes a contiguous `&[K]`, and what rides beside the keys travels in step.
  // The alternative — a key vector rebuilt per batch — would allocate per round trip for a slice
  // that is already there. The carry rides in an `Option` so each is MOVED out at the position it
  // belongs to, which is what keeps `S` unbounded: nothing here clones a score.
  let mut carried: Vec<Option<C>> = Vec::with_capacity(hits.len());
  let mut keys = Vec::with_capacity(hits.len());
  for hit in hits {
    let (carry, key) = split(hit);
    carried.push(Some(carry));
    keys.push(key);
  }

  let mut filled = Vec::new();
  let mut taken = 0usize;
  while filled.len() < limit && taken < keys.len() {
    let batch = (limit - filled.len()).min(keys.len() - taken);
    let found = store.gather(&keys[taken..taken + batch], filter).await?;
    // `take(batch)` is the length contract: a gather answering short leaves the tail read as
    // pruned, and one answering long has its surplus ignored. Either is a store that did not keep
    // the seam's positional promise, and neither can move a result onto the wrong key.
    for (offset, slot) in found.into_iter().take(batch).enumerate() {
      let Some(entity) = slot else {
        continue;
      };
      // The numbers are the KEY's, taken from the position the entity came back at — which is why
      // they were carried in a parallel list rather than recomputed from the page.
      let Some(carry) = carried[taken + offset].take() else {
        continue;
      };
      filled.push(rejoin(carry, entity));
      if filled.len() == limit {
        break;
      }
    }
    taken += batch;
  }
  Ok(filled)
}

#[cfg(test)]
mod tests;