hypersteeldb 0.5.4

A database that compiles questions instead of guessing answers: typed vocabulary discovered from your documents, queries type-checked before they run, roaring-bitmap set algebra over reified hyperedges, and Dempster-Shafer evidence with an explicit conflict guard.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

//! Projector — the pluggable ingest seam, analogous to DuckDB's table functions / replacement scans.
//!
//! DuckDB queries CSV/Parquet/JSON uniformly because every reader lowers to a common DataChunk that
//! the SQL engine consumes. Here every file format lowers to a common **Situation stream** (tokens +
//! display cells) that the IKL/bitmap engine consumes — so adding a format never touches the query
//! layer. Projection is push-based: a reader emits one Situation per source record into a sink.
//!
//! The core owns the tabular/columnar/text projectors (CSV, JSONL text situations; Parquet/plain-text
//! next). Heavy document formats (PDF/DOCX/PPTX/HTML) reuse the existing producer pipeline, which emits
//! the same Situation stream over a sidecar bridge — no re-implementation of document parsing here.

use serde::Serialize;

#[derive(Serialize, Clone, Copy, PartialEq)]
#[serde(rename_all = "lowercase")]
pub enum CorpusKind {
    Csv,
    Text,
}

/// One projected record: the tokens it asserts (for set-algebra), the cells to show for it, and any
/// numeric fields it carries (for `(num <field> <op> <value>)` range predicates — e.g. a CSV numeric
/// column or a canonicalised quantity span like `qty-length` = 70 m).
pub struct Situation {
    pub tokens: Vec<String>,
    pub display: Vec<String>,
    pub numbers: Vec<(String, f64)>,
    /// **Infon polarity** per token (paper §1.2): `(token, i)` where `i ∈ {-1, -0.5, +0.5, +1}`. Only
    /// tokens whose polarity differs from the default `+1` need an entry; the rest are asserted-positive.
    /// Populated from the tagger's epistemic head and consumed by `InfonIndex::add_infon_polar`.
    pub beliefs: Vec<(String, f32)>,
}

impl Situation {
    /// Token + display record with no numeric fields (the common case).
    pub fn new(tokens: Vec<String>, display: Vec<String>) -> Situation {
        Situation { tokens, display, numbers: Vec::new(), beliefs: Vec::new() }
    }
}

/// A push-based reader over one source. `project` drives the sink once per source record.
pub trait Projector {
    fn columns(&self) -> Vec<String>;
    fn kind(&self) -> CorpusKind;
    fn source(&self) -> String;
    fn project(self: Box<Self>, sink: &mut dyn FnMut(Situation)) -> std::io::Result<()>;
}

/// slugify a value into a token-safe form: trim, lowercase, collapse whitespace/`/` to `-`.
pub fn slug(s: &str) -> String {
    let t = s.trim().to_lowercase();
    let mut out = String::with_capacity(t.len());
    let mut prev_dash = false;
    for ch in t.chars() {
        if ch.is_whitespace() || ch == '/' {
            if !prev_dash && !out.is_empty() {
                out.push('-');
                prev_dash = true;
            }
        } else {
            out.push(ch);
            prev_dash = false;
        }
    }
    while out.ends_with('-') {
        out.pop();
    }
    out
}