ingraph 0.1.0

A declaration-driven indexing framework: records and nodes are written as Rust types, and the storage rows, the dataflow between them and the GraphQL query surface are emitted from the declaration.
Documentation
//! THE SEGMENTER: a text in, a sequence of script runs out — UAX #24's resolution, stated once.

use core::iter::FusedIterator;

use super::WritingSystem;

#[cfg(test)]
mod tests;

/// **CUT `text` INTO SCRIPT RUNS** — the one implementation both sides of a fan call.
///
/// ```
/// use ingraph::{WritingSystem, script_runs};
///
/// let runs: Vec<_> = script_runs("Rust 2024 年鉴").collect();
///
/// assert_eq!(runs.len(), 2);
/// assert_eq!(runs[0].script(), &WritingSystem::Latn);
/// assert_eq!(runs[0].text(), "Rust 2024 ");
/// assert_eq!(runs[1].script(), &WritingSystem::Hani);
/// assert_eq!(runs[1].text(), "年鉴");
/// ```
///
/// # THE RESOLUTION, in the three sentences it takes
///
/// 1. **A contextual character continues the run it is in.** Spaces, digits, punctuation and
///    combining marks belong to every script and to none
///    ([`WritingSystem::is_contextual`]), so they extend whatever preceded them rather than cutting it.
///    The space and the digits in the example above are Latin because the word before them was.
/// 2. **At the head, a contextual character joins what FOLLOWS.** There is nothing before it to
///    continue, so the leading run of them is attached to the first concrete script that appears —
///    `" 年鉴"` is one Han run and not a fallback run plus a Han one.
/// 3. **A text with no concrete script at all is one run in the fallback bucket.** `"123 !!!"` is
///    entirely contextual, so there is nothing for rules 1 and 2 to attach it to, and it lands in
///    [`WritingSystem::Zzzz`] beside every character the table does not name.
///
/// **PRECEDING WINS OVER FOLLOWING**, and rule 2 is the exception rather than a second rule. It is
/// UAX #24's own order and it is the one that matches how text is written: a trailing space, comma
/// or full stop is part of the sentence it ends, and a script change begins at the character that
/// changed it. The alternative — attaching to the following — would put the space in `"日本 word"`
/// into the Latin run, which is a boundary no writer of either script would draw.
///
/// # WHAT IT PROMISES, exactly
///
/// - **Total and non-overlapping**: the runs are contiguous, in order, and cover every byte of
///   `text` exactly once. Concatenating their texts reproduces the input.
/// - **Non-empty**: no run is ever zero bytes, so an empty input yields no runs at all rather than
///   one empty one.
/// - **Deterministic**: the same text always cuts the same way. Nothing here reads a locale, a
///   clock or an environment.
/// - **Zero-allocation**: it borrows `text` and yields slices of it, which is what lets a write path
///   run it per delivery.
///
/// # WHAT IT DOES NOT DO
///
/// It does not tokenise, lowercase, normalise or fold anything, and it has no opinion about
/// language — see the `script` module's note for where each of those lives. It also does not merge
/// runs of different scripts that a deployment considers one lane: a Japanese sentence cuts into Han,
/// Hiragana and Katakana runs, and putting them back together is a router's policy over an answer
/// this function gives faithfully.
#[inline]
#[must_use]
pub fn script_runs(text: &str) -> ScriptRuns<'_> {
  ScriptRuns { text, at: 0 }
}

/// ONE RUN — a maximal stretch of `text` written in one script.
///
/// # It carries the SLICE and the OFFSET, and both are spent
///
/// The slice is what a fan feeds to the analyzer that holds this script, and the offset is where it
/// came from — which is what lets a hit found in a per-script index be pointed back at a position in
/// the transcript that produced it. Neither can be derived from the other without the original text,
/// so both are here.
///
/// # There is no `with_` / `set_` pair, and the absence is the invariant
///
/// The house four-piece is for a value a holder may narrow after construction. These three seats are
/// ONE FACT — this stretch, at this place, in this script — and a run whose text was replaced
/// without its offset would be a lie the type had helped tell. A caller that wants a different run
/// builds one.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ScriptRun<'a> {
  /// The script this stretch is written in.
  script: WritingSystem,
  /// Where it starts, as a byte offset into the text it was cut from.
  at: usize,
  /// The stretch itself.
  text: &'a str,
}

impl<'a> ScriptRun<'a> {
  /// This stretch, at this offset, in this script.
  ///
  /// Public because a hand-written router's own tests build one, and because nothing here is the
  /// segmenter's to keep — the invariants above are promises [`script_runs`] makes about what IT
  /// produces, not conditions this constructor checks.
  #[inline]
  #[must_use]
  pub const fn new(script: WritingSystem, at: usize, text: &'a str) -> Self {
    Self { script, at, text }
  }

  /// The script this run is written in.
  #[inline]
  #[must_use]
  pub const fn script(&self) -> &WritingSystem {
    &self.script
  }

  /// Where this run starts, as a byte offset into the text it was cut from.
  #[inline]
  #[must_use]
  pub const fn at(&self) -> usize {
    self.at
  }

  /// Where this run ends — the offset one past its last byte, and the next run's [`at`](Self::at).
  #[inline]
  #[must_use]
  pub const fn end(&self) -> usize {
    self.at + self.text.len()
  }

  /// The stretch of text itself — never empty.
  #[inline]
  #[must_use]
  pub const fn text(&self) -> &'a str {
    self.text
  }

  /// All three seats, owned — what a router spends when it routes the script and forwards the text.
  #[inline]
  #[must_use]
  pub fn into_components(self) -> (WritingSystem, usize, &'a str) {
    (self.script, self.at, self.text)
  }
}

/// The runs of one text, cut lazily — [`script_runs`]' own iterator.
///
/// Lazy rather than collected, because a write path runs this per delivery and a query path runs it
/// per question: the runs are slices of the text already in hand, so there is nothing to allocate
/// unless a caller asks for a `Vec` by collecting one.
#[derive(Debug, Clone)]
pub struct ScriptRuns<'a> {
  /// The whole text, kept so every run's offset is against one origin.
  text: &'a str,
  /// The byte offset the next run starts at.
  at: usize,
}

impl<'a> Iterator for ScriptRuns<'a> {
  type Item = ScriptRun<'a>;

  fn next(&mut self) -> Option<Self::Item> {
    if self.at >= self.text.len() {
      return None;
    }

    let start = self.at;
    // THE RUN'S SCRIPT IS NOT KNOWN AT ITS FIRST CHARACTER, which is what rule 2 costs: a leading
    // stretch of contextual characters belongs to the first concrete script that appears after it,
    // so the answer is pending until one does.
    let mut script: Option<WritingSystem> = None;
    let mut end = self.text.len();

    for (offset, character) in self.text[start..].char_indices() {
      let at_character = WritingSystem::of(character);
      if at_character.is_contextual() {
        continue;
      }
      match &script {
        // Rule 2: the pending leading stretch joins this one, so the run's start does not move.
        None => script = Some(at_character),
        // Rule 1 is the `continue` above; this is the same script again, which extends the run.
        Some(carried) if *carried == at_character => {}
        // THE BOUNDARY, and it is at the character that changed the script rather than after the
        // contextual run before it — which is rule 1 read from the other side.
        Some(_) => {
          end = start + offset;
          break;
        }
      }
    }

    self.at = end;
    // Rule 3: nothing concrete was ever seen, so the whole stretch is contextual and lands in the
    // fallback bucket beside every character the table does not name.
    Some(ScriptRun::new(
      script.unwrap_or(WritingSystem::Zzzz),
      start,
      &self.text[start..end],
    ))
  }

  /// At least one run where any text remains, and at most one per character.
  ///
  /// The lower bound is what a `collect` allocates against, and it is `1` rather than a division:
  /// a run is bounded below by one character and the text's byte length is not its character count,
  /// so any tighter number would be a guess.
  fn size_hint(&self) -> (usize, Option<usize>) {
    let remaining = self.text.len().saturating_sub(self.at);
    (usize::from(remaining > 0), Some(remaining))
  }
}

/// Once the cursor has reached the end it stays there, so a spent iterator keeps answering [`None`].
impl FusedIterator for ScriptRuns<'_> {}