ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! The fused walk, LIVE: a real Tantivy index and the in-tree vector face answer one question about
//! one corpus, and reciprocal rank fusion joins their two rankings at the media root.
//!
//! # Why this runs rather than asserts on a table of ordinals
//!
//! `fuse`'s own tests pin the arithmetic against hand-computed decimals, and they can do that because
//! they hand it the ordinals. What they cannot say is that a REAL engine's numbers are the ones that
//! arrive: two engines measuring in two incomparable currencies, ranking one corpus, joined at a root
//! neither of them knows about.
//!
//! The vector index carries one deliberate duplicate for that reason — see [`voiceprints`]. Without a
//! hole in the survivors' ordinals every rank equals its list position, and a fold reading the wrong
//! one passes here; that was MEASURED by mutating the fold rather than assumed, and the duplicate is
//! what makes these assertions able to tell the two apart on a live stream.
//!
//! # The two halves that are the EMISSION's, written by hand
//!
//! The evidence union and the root projection are what a graph root would emit, and no declaration
//! can name the engine it is stored in yet, so they are written here exactly as the emission will
//! write them — one variant per member carrying that member's own `Ranked`, and a root taken as the
//! key's root-first prefix. Everything between them is the shipped fold.

use super::corpus::SegmentKey;
use crate::{FulltextQuery, Fused, Fusion, HashJoin, HashSeen, Ranked, Rooted, retrieve::fuse};

/// THE EMITTED EVIDENCE UNION, by hand: one variant per member of the fused query, each carrying
/// that member's own `Ranked<Key, Score>` at that member's own score currency.
#[derive(Debug, Clone, PartialEq)]
enum Sighting {
  Transcript(Ranked<SegmentKey, f32>),
  Voiceprint(Ranked<SegmentKey, f32>),
}

impl Sighting {
  /// The hit the member is offering as its reason — the whole payload of the ruled variant.
  fn ranked(&self) -> &Ranked<SegmentKey, f32> {
    match self {
      Self::Transcript(hit) | Self::Voiceprint(hit) => hit,
    }
  }

  /// Which member said it, as a word a message can carry.
  fn member(&self) -> &'static str {
    match self {
      Self::Transcript(_) => "transcript",
      Self::Voiceprint(_) => "voiceprint",
    }
  }

  fn segment(&self) -> i64 {
    self.ranked().node_ref().id
  }
}

/// THE ROOT PROJECTION, by hand: a segment key's root-first prefix is its media step, so the grain
/// this fuses at is *one row per medium*.
fn rooted(
  hits: Vec<Ranked<SegmentKey, f32>>,
  variant: fn(Ranked<SegmentKey, f32>) -> Sighting,
) -> Vec<Rooted<String, Sighting>> {
  hits
    .into_iter()
    .map(|hit| {
      let root = hit.node_ref().media.clone();
      let rank = hit.rank();
      Rooted::new(root, rank, variant(hit))
    })
    .collect()
}

/// Every medium the seeded corpus holds, in order.
const MEDIA: [&str; 3] = ["m1", "m2", "m3"];

/// The shipped fold's term for a hit at `rank`, written out here rather than reached for, so this
/// file's expectations are not the implementation's own arithmetic handed back.
fn term(rank: u64) -> f64 {
  1.0 / (60.0 + rank as f64)
}

fn roots(fused: &[Fused<String, Sighting>]) -> Vec<&str> {
  fused.iter().map(|row| row.root_ref().as_str()).collect()
}

/// The whole full-text answer for `sunset`, unfloored.
async fn transcripts() -> Vec<Ranked<SegmentKey, f32>> {
  let (_dir, index, mut engine) = super::fulltext::index(&[]).await;
  super::fulltext::retrieve_keys::<HashSeen<_>>(
    &mut engine,
    &index,
    &FulltextQuery::new("sunset", 0.0_f32),
    usize::MAX,
  )
  .await
}

/// The whole vector answer, unfloored — with a HOLE in its ordinals, on purpose.
///
/// Deterministic by construction: the fixture's embeddings lean progressively further from the query
/// direction, so the stream is segments 1 through 6 in order. What is added here is a SECOND
/// voiceprint of segment 1, leaning just enough to land second in the raw stream — where the dedup
/// drops it, consuming ordinal 2 and leaving it behind.
///
/// That hole is what makes this file able to say anything about the ENGINE ordinal at all. Without
/// it every survivor's rank equals its position in the list, the two readings of *rank* coincide, and
/// a fold reading the wrong one passes here — measured, by mutating the fold and watching these
/// tests stay green. With it the survivors wear `1, 3, 4, 5, 6, 7` against positions `1..6`, and the
/// two readings answer differently.
async fn voiceprints() -> Vec<Ranked<SegmentKey, f32>> {
  let mut engine = super::vector::index([super::vector::Voiceprint::of(
    "m1",
    1,
    [1.0, 0.10, 0.0, 0.0],
  )]);
  super::vector::retrieve_keys::<HashSeen<_>>(&mut engine, &super::vector::query(0.0), usize::MAX)
    .await
}

/// **The cross-modal claim, live.** A lexical index and a vector index rank the same corpus by two
/// incomparable measures, and one ranked list comes back — every row carrying BOTH members' reasons.
///
/// The scores are re-derived from the ordinals the ENGINES actually produced rather than written
/// down, because a live BM25 order is a property of the corpus and the analyzer; what is written
/// down is the RULE, and the rule is applied to whatever the engines said.
#[tokio::test]
async fn both_flavours_fuse_into_one_ranking() {
  let lexical = transcripts().await;
  let similar = voiceprints().await;
  // The two members disagree, which is what makes the fusion do work rather than confirm an order.
  assert_ne!(
    lexical
      .iter()
      .map(|hit| hit.node_ref().id)
      .collect::<Vec<_>>(),
    similar
      .iter()
      .map(|hit| hit.node_ref().id)
      .collect::<Vec<_>>(),
  );
  // And the vector member's ordinals are NOT its list positions, which is what lets the expectation
  // below distinguish the two readings of `rank` on a live stream.
  assert_eq!(
    similar.iter().map(Ranked::rank).collect::<Vec<_>>(),
    vec![1, 3, 4, 5, 6, 7],
    "the deduped duplicate left its ordinal behind",
  );

  // Each member's FIRST sighting of each medium — what the root grain keeps, computed here from the
  // live lists so the expectation is independent of the fold's own bookkeeping.
  let best = |hits: &[Ranked<SegmentKey, f32>], medium: &str| -> Option<u64> {
    hits
      .iter()
      .find(|hit| hit.node_ref().media == medium)
      .map(Ranked::rank)
  };
  let expected: Vec<(&str, f64)> = MEDIA
    .iter()
    .map(|medium| {
      let lexically = best(&lexical, medium).map(term).unwrap_or_default();
      let similarly = best(&similar, medium).map(term).unwrap_or_default();
      (*medium, lexically + similarly)
    })
    .collect();

  let fused = fuse::<_, _, HashJoin<_>>(
    vec![
      rooted(lexical, Sighting::Transcript),
      rooted(similar, Sighting::Voiceprint),
    ],
    Fusion::rrf(),
  );

  assert_eq!(fused.len(), 3, "three media, three rows");
  for (medium, score) in expected {
    let row = fused
      .iter()
      .find(|row| row.root_ref() == medium)
      .unwrap_or_else(|| panic!("{medium} is in the answer"));
    assert!(
      (row.score().get() - score).abs() < 1e-12,
      "{medium}: {} is not {score}",
      row.score().get()
    );
    let mut members: Vec<&str> = row
      .evidence()
      .iter()
      .map(|held| Sighting::member(held.seen()))
      .collect();
    members.sort_unstable();
    assert_eq!(
      members,
      vec!["transcript", "voiceprint"],
      "{medium} carries BOTH members' reasons"
    );
  }

  assert!(
    fused
      .windows(2)
      .all(|pair| pair[0].score() >= pair[1].score()),
    "{:?}",
    fused.iter().map(Fused::score).collect::<Vec<_>>()
  );
}

/// **The root grain keeps the BEST segment of each medium**, live and exactly.
///
/// The vector stream is segments 1..6 in order, so each medium is sighted twice — 1 and 2 for `m1`,
/// 3 and 4 for `m2`, 5 and 6 for `m3`. The fold counts the FIRST of each pair and drops the second
/// whole, so the voiceprint's reason for each row is the odd-numbered segment: the moment to open
/// when a user clicks the video.
///
/// This is what makes *search videos, show the moment that matched* fall out of the fusion rather
/// than out of a second query.
#[tokio::test]
async fn each_row_carries_the_best_segment_of_its_medium() {
  let similar = voiceprints().await;
  assert_eq!(
    similar
      .iter()
      .map(|hit| hit.node_ref().id)
      .collect::<Vec<_>>(),
    vec![1, 2, 3, 4, 5, 6],
    "the fixture's stream, which the expectation below reads",
  );
  assert_eq!(
    similar.iter().map(Ranked::rank).collect::<Vec<_>>(),
    vec![1, 3, 4, 5, 6, 7],
    "with the dedup's hole in it",
  );

  let fused = fuse::<_, _, HashJoin<_>>(vec![rooted(similar, Sighting::Voiceprint)], Fusion::rrf());

  assert_eq!(roots(&fused), MEDIA);
  assert_eq!(
    fused
      .iter()
      .map(|row| {
        assert_eq!(row.evidence().len(), 1, "one member, counted once");
        row.evidence()[0].seen().segment()
      })
      .collect::<Vec<_>>(),
    vec![1, 3, 5],
    "the second sighting of each medium is dropped, not added",
  );
}

/// **A member the floor silenced contributes NOTHING, and the fused answer is the rest.**
///
/// The full-text floor is set above the corpus's own best score, so that member's walk terminates on
/// its first hit and hands back an empty list. The fused answer is then the vector member's alone,
/// and it is HAND-COMPUTABLE end to end because that member's stream is deterministic:
///
/// ```text
/// voiceprint:  m1 first at rank 1   m2 first at rank 4   m3 first at rank 6
///
/// m1 = 1/61 = 0.01639344262295082
/// m2 = 1/64 = 0.015625
/// m3 = 1/66 = 0.015151515151515152
/// ```
///
/// The ordinals are 1, 4 and 6 rather than 1, 3 and 5 because of the deduped duplicate: it consumed
/// ordinal 2, so every later hit's engine ordinal runs one ahead of its list position. Reading
/// positions instead would put `1/63` and `1/65` here, which is what makes these three decimals a
/// LIVE assertion about the engine's own numbering rather than about the fixture's length.
///
/// Which is also the *roadless member sits out* posture arriving from the other direction: a member
/// with nothing above its floor is indistinguishable from a member that never ran, and neither is an
/// error.
#[tokio::test]
async fn a_member_the_floor_silenced_leaves_the_answer_to_the_rest() {
  let (_dir, index, mut engine) = super::fulltext::index(&[]).await;
  let unfloored = super::fulltext::retrieve_keys::<HashSeen<_>>(
    &mut engine,
    &index,
    &FulltextQuery::new("sunset", 0.0_f32),
    usize::MAX,
  )
  .await;
  let above_everything = unfloored[0].score() + 1.0;

  let silenced = super::fulltext::retrieve_keys::<HashSeen<_>>(
    &mut engine,
    &index,
    &FulltextQuery::new("sunset", above_everything),
    usize::MAX,
  )
  .await;
  assert!(silenced.is_empty(), "the floor took the whole stream");

  let fused = fuse::<_, _, HashJoin<_>>(
    vec![
      rooted(silenced, Sighting::Transcript),
      rooted(voiceprints().await, Sighting::Voiceprint),
    ],
    Fusion::rrf(),
  );

  assert_eq!(roots(&fused), MEDIA);
  let scores: Vec<f64> = fused.iter().map(|row| row.score().get()).collect();
  for (got, hand_computed) in scores.iter().zip([
    0.016_393_442_622_950_82,
    0.015_625,
    0.015_151_515_151_515_152,
  ]) {
    assert!(
      (got - hand_computed).abs() < 1e-15,
      "{got} is not {hand_computed}"
    );
  }
  assert!(
    fused.iter().all(|row| row
      .evidence()
      .iter()
      .all(|saw| saw.seen().member() == "voiceprint")),
    "a silenced member offers no evidence",
  );
}