ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! The SINK a full-text converter writes into — one `push` per SEGMENT of the document.
//!
//! A field whose value is not already text reaches a full-text index through a converter the index
//! node's declaration names, and that converter takes a [`BoxWriter`] rather than returning a string:
//!
//! ```
//! use ingraph::BoxWriter;
//!
//! struct Label(&'static str);
//!
//! fn join_labels(out: &mut BoxWriter, labels: &Vec<Label>) {
//!   for label in labels {
//!     out.push(label.0);
//!   }
//! }
//!
//! let mut out = BoxWriter::new();
//! join_labels(&mut out, &vec![Label("red car"), Label("blue sky")]);
//! assert_eq!(out.segments().collect::<Vec<_>>(), ["red car", "blue sky"]);
//! ```
//!
//! It names no engine, and nothing gates it. A declaration names its converter, so the signature an
//! author writes has to be spellable in a build with no backend at all — the type is reached from
//! the DECLARING side, which has no features of the framework's to speak of.
//!
//! # Why a sink, and not `-> String`
//!
//! **Because the SEGMENTATION has to survive, and a returned string destroys it.** A positional
//! full-text engine separates the VALUES of one field so that an exact phrase cannot match across the
//! join. Tantivy 0.26.1 is the measured case: `src/postings/postings_writer.rs` declares
//! `const POSITION_GAP: u32 = 1` (line 18) and, after each value's token stream, advances
//! `indexing_position.end_position = end_position + POSITION_GAP` (line 158) — so the next value's
//! first token sits one position PAST where a continuation would have been, and a phrase query
//! spanning the boundary matches nothing. A converter returning one `String` collapses every value
//! into a single append, and then a phrase CAN match the tail of one label and the head of the next: a
//! hit for a phrase the corpus does not contain. Pushing per segment is what keeps that impossible,
//! and it puts the choice of where the boundaries are with the author, who is the only one who knows
//! whether two values are one utterance or two.
//!
//! **And because the signature has to be NAMABLE.** `fn(&mut BoxWriter, &T)` is concrete, so
//! generated code can write the call and the compiler can check the declared path against it at the
//! site. Returning `impl Iterator<Item = &str>` — the shape that would preserve segmentation without a
//! sink — has no spelling an expansion can write; returning `String` has one, and pays an allocation
//! per call for the collapse described above.
//!
//! # What it is not
//!
//! Nothing hands the buffer out as ONE string. That accessor would be exactly the concatenation the
//! segmentation exists to prevent, and it would read like "the document" while being a different
//! document from the one the converter described. [`BoxWriter::segments`] is the whole read face.
//!
//! The name is the ONE-BOX destination: today a document has a single text field, so every segment a
//! converter pushes lands in the same box and a writer needs no field to address. Per-field
//! (`named`) documents are a declaration this grammar does not have yet, and until it does there is
//! nothing for a second destination to be.
//!
//! # The two builtins
//!
//! Most converters are one of two loops, so both are here as methods rather than written out by
//! every author: [`BoxWriter::push_each`] and [`BoxWriter::push_joined`]. The pair is the WHOLE
//! choice a segmented sink offers, and it is the choice the header is about — `each` keeps the values
//! apart so a phrase cannot span two of them, `join` puts them in one value so it can, through a
//! separator the caller names. A hand-written converter is for everything else, and neither builtin
//! can do anything one could not.
//!
//! The index node's derive HAS that grammar now, and the two loops are what its two shorthand words
//! spend: `multi` is [`push_each`](BoxWriter::push_each) and `join = "…"` is
//! [`push_joined`](BoxWriter::push_joined). The pair was written before anything picked between
//! them and is unchanged by being picked between — which is the whole of what *expressible before
//! anything consumes it* was worth.
//!
//! # The THREE ROADS, and [`Searchable`] is the one that can refuse
//!
//! A `searchable` field reaches this sink one of three ways: the value is text already (IDENTITY),
//! a builtin walks a multi-valued field (SHORTHAND), or the declaration names a converter
//! (`from`/`try_from`). Each road carries its own bound, so a field the road does not fit is a
//! compile error at the field rather than a promise the parser could not check — and the identity
//! one is spelled through [`Searchable`] rather than through [`push`](BoxWriter::push) so that the
//! refusal can name the other two.

#[cfg(test)]
mod tests;

#[cfg(test)]
mod searchable_tests;

/// The sink a full-text converter pushes SEGMENTS of text into.
///
/// One [`push`](Self::push) is one segment. Segments are kept apart rather than joined, because a
/// positional engine's value boundary is what stops a phrase query matching across two of them — the
/// module header carries the measurement.
///
/// Backed by ONE `String` plus each segment's end offset, not a `Vec<String>`: a push is a byte copy
/// into a buffer that grows amortized, so a converter over N values pays no per-value allocation.
/// That is the half of the "not `-> String`" argument that is about cost; the other half is about
/// correctness and is the reason this type exists at all.
#[derive(Debug, Clone, Default)]
pub struct BoxWriter {
  /// Every segment's bytes, back to back. Never handed out whole.
  text: String,
  /// Each segment's END offset into [`text`](Self::text), so the length IS the segment count and
  /// segment `i` spans `ends[i - 1]..ends[i]` (the first from `0`).
  ends: Vec<usize>,
}

impl BoxWriter {
  /// A writer with no segments.
  pub const fn new() -> Self {
    Self {
      text: String::new(),
      ends: Vec::new(),
    }
  }

  /// Push one SEGMENT of the document.
  ///
  /// Takes `impl AsRef<str>` so a `&str`, a `String`, a `SmolStr` or any other text newtype pushes
  /// without the caller converting first. An EMPTY push is recorded as an empty segment rather than
  /// dropped: the writer reports what the converter did, and dropping it would silently change the
  /// segmentation the converter chose. A converter that means to skip a value does not push it.
  pub fn push(&mut self, text: impl AsRef<str>) {
    self.text.push_str(text.as_ref());
    self.ends.push(self.text.len());
  }

  /// Push ONE segment per value — the whole body of the `each` builtin.
  ///
  /// A loop over [`push`](Self::push) and nothing else, which is the point: the builtin exists so
  /// that the common converter does not have to be written out, not so that it can do something a
  /// hand-written one could not.
  ///
  /// The bound is where the TYPE is checked. A field whose values are not renderable as text does not
  /// satisfy `I::Item: AsRef<str>`, and the refusal is an `E0277` at the call naming the field's own
  /// type — the declaration has no type information and needs none.
  ///
  /// Zero values push NOTHING, which is the honest reading of "one segment per value" and matches
  /// [`push_joined`](Self::push_joined)'s opposite: a join of nothing is still one append.
  pub fn push_each<I>(&mut self, values: I)
  where
    I: IntoIterator,
    I::Item: AsRef<str>,
  {
    for value in values {
      self.push(value);
    }
  }

  /// Push ONE segment, the values joined by `separator` — the whole body of the `join` builtin.
  ///
  /// [`push_each`](Self::push_each)'s opposite, and the difference is the one the declaration is
  /// about: these values end up in a SINGLE value, so a phrase CAN match across two of them, through
  /// the separator the author chose. That is a legitimate thing to want ("red, blue" as one caption)
  /// and it is also the only thing a single-value engine can do — which is why `join` is honoured in
  /// every face and `each` is not.
  ///
  /// Writes THROUGH the buffer rather than building a `String` and pushing it, so the joined form
  /// keeps the no-per-value-allocation property the type's header claims for the segmented one.
  ///
  /// Zero values push ONE EMPTY segment, because `join` is one append and the join of nothing is the
  /// empty string — `[].join(sep)` in std's own terms. It is [`push`](Self::push)'s rule about an
  /// empty push, reached by a different route.
  pub fn push_joined<I>(&mut self, values: I, separator: &str)
  where
    I: IntoIterator,
    I::Item: AsRef<str>,
  {
    for (i, value) in values.into_iter().enumerate() {
      if i > 0 {
        self.text.push_str(separator);
      }
      self.text.push_str(value.as_ref());
    }
    self.ends.push(self.text.len());
  }

  /// The segments, in the order they were pushed.
  ///
  /// [`ExactSizeIterator`], so a caller that needs the segment count asks `segments().len()` — the
  /// count is the iterator's rather than a second accessor's, so there is one authority on it.
  pub fn segments(&self) -> impl ExactSizeIterator<Item = &str> {
    let mut start = 0;
    self.ends.iter().map(move |&end| {
      let segment = &self.text[start..end];
      start = end;
      segment
    })
  }

  /// Whether nothing has been pushed.
  ///
  /// Not "holds no text": a writer handed one empty segment is NOT empty, because one segment is what
  /// it was told to record.
  pub fn is_empty(&self) -> bool {
    self.ends.is_empty()
  }
}

/// THE IDENTITY ROAD: a value that is ALREADY the document text.
///
/// One segment, pushed as it stands. It is what a `searchable` field takes when its engine's block
/// names no road at all, and it is the road that has to be able to say NO — a field whose value is
/// not text has no identity road, and the useful answer names the two roads that would have carried
/// it.
///
/// # It is a bound, and the bound IS the check
///
/// The derive performs no type test on a `searchable` field and must not: it has no type
/// information, and the one check a parser could make is a spelling allowlist, which false-refuses
/// `SmolStr`, a newtype, an alias and `Vec<String>` alike — and this crate calls a false refusal a
/// defect of its own. So the road is checked where the types are known. A field the identity road
/// does not fit is `E0277` at that field's own type token, carrying the note below.
///
/// # Blanket over `AsRef<str>`, and that is where the surface stays open
///
/// Nothing implements this trait by hand and nothing needs to: the blanket covers every type that
/// can already be read as text, which is the same seam [`BoxWriter::push`] takes. A domain newtype
/// joins the road by implementing `AsRef<str>` — one impl on the writer's own type, no orphan to
/// argue with — and the blanket carries it here. Publishing a second membership trait beside the
/// std one would be a second list to keep in step with the first.
#[diagnostic::on_unimplemented(
  message = "`{Self}` has no road into a full-text document",
  label = "this field is `searchable`, and nothing says how it is read as text",
  note = "a searchable field reaches the document by one of THREE roads, and the ENGINE'S OWN \
          BLOCK is where the last two are written — IDENTITY: the value is text already \
          (anything `AsRef<str>`; a domain type joins by implementing it); SHORTHAND, for a \
          multi-valued field: `join = \", \"` puts the values in ONE segment (a phrase can match \
          across them) and `multi` gives each its own (a phrase cannot); DECLARED: \
          `from = \"path::to::fn\"`, or `try_from = \"…\"` where the conversion can fail — a \
          converter takes `(&mut BoxWriter, &Field)` and pushes whatever segments it means"
)]
pub trait Searchable {
  /// Push this value into `into` as ONE segment.
  fn write(&self, into: &mut BoxWriter);
}

impl<T: AsRef<str> + ?Sized> Searchable for T {
  fn write(&self, into: &mut BoxWriter) {
    into.push(self.as_ref());
  }
}