steeldb/

lib.rs

//! **A database that compiles questions instead of guessing answers.**
//!
//! Most systems answer a question about documents by *similarity*: find the nearest text and return it. That
//! works until the question involves a combination (`A but not B`), a complete count, or something the
//! documents simply do not contain — where a similarity search still returns its closest guess, and a guess is
//! indistinguishable from an answer.
//!
//! SteelDB learns which categories your documents actually support, type-checks a question against them before
//! anything runs, and executes the survivors as bitwise set algebra over compressed bitmaps. A question the
//! data cannot answer is **refused**, with the alternatives that do exist.
//!
//! # Start here
//!
//! ```no_run
//! use steeldb::SteelDb;
//!
//! # let documents: Vec<String> = Vec::new();
//! // No model files needed: the vocabulary is discovered from the text.
//! let db = SteelDb::ingest(documents)?;
//!
//! // Category names come from the words the documents use, so read them before writing a query.
//! for c in db.categories() {
//!     println!("can ask about {}", c.wildcard());
//! }
//!
//! match db.query("(and elevation/* (not state/negated))") {
//!     Ok(answer)   => println!("{} situations", answer.len()),
//!     Err(refused) => println!("{refused}"),   // says what the data does contain
//! }
//! # Ok::<(), steeldb::Error>(())
//! ```
//!
//! # Three verbs
//!
//! | verb | what it needs | what it costs |
//! |---|---|---|
//! | [`SteelDb::ingest`] | nothing — no models, no network | deterministic and free |
//! | [`SteelDb::query`] | nothing | microseconds |
//! | [`learn`] | credentials and a network | a model call, and a bill |
//!
//! The asymmetry is deliberate. `ingest` and `query` are pure; `learn` calls a language model, so it lives in
//! its own module, is `async`, is feature-gated, and returns a *proposal* rather than changing your vocabulary.
//! You review it and [`SteelDb::adopt`] it, at which point the same gate that governs local discovery decides
//! what survives — a model cannot add a category a deterministic test would have rejected.
//!
//! [`SteelDb`] is the whole API for most uses. [`Answer`] is a *complete* set rather than a ranked sample, so
//! counting it means something. [`Refused`] is an error rather than an empty result because those are different
//! facts, and conflating them is how a confident wrong answer gets produced.
//!
//! # The query language
//!
//! Queries are s-expressions — operation first, nested lists, as in Lisp. The whole grammar:
//!
//! | form | meaning |
//! |---|---|
//! | `category/value` | situations carrying that exact tag |
//! | `category/*` | any value in that category |
//! | `(and A B)` | intersection |
//! | `(or A B)` | union |
//! | `(not A)` | difference |
//! | `(num field op value)` | numeric comparison; `op` is `ge gt le lt eq ne` |
//! | `(evidence A :min-bel f)` | only where belief in `A` reaches `f` |
//! | `(s-path :s n (source A) (target B))` | situations on a chain sharing ≥ `n` tags per step |
//! | `(combine-ds :max-conflict f …)` | fuse independent evidence, or refuse |
//!
//! There is deliberately almost no syntax to get wrong, which matters when the author is a language model.
//!
//! # Beyond the basics
//!
//! - [`evidence`] — Dempster–Shafer belief intervals, and the conflict metric that refuses to fuse
//!   contradictory sources rather than averaging them into a consensus nobody holds.
//! - [`programs`] — higher-order structure: s-paths, and the primal/dual s-filtration.
//! - [`emergent`] — how the vocabulary is discovered from prose, with no model.
//! - [`models`] — where trained weights come from. Nothing downloads without being asked.
//! - [`linter`] — the type-checker, if you want to validate without executing.
//!
//! # Installing
//!
//! The crate is published as **`hypersteeldb`** and imported as `steeldb`:
//!
//! ```toml
//! [dependencies]
//! hypersteeldb = "0.2"
//! ```
//!
//! (The bare name `steeldb` was taken on crates.io in 2023 by an unrelated project, so the package carries the
//! longer name while the import stays short.)
//!
//! # Features
//!
//! The default build is pure Rust with no model dependencies and compiles to `wasm32`.
//!
//! | feature | adds |
//! |---|---|
//! | `embed` | static embeddings + optimal-transport discovery (links a C regex library) |
//! | `onnx` | the trained span tagger |
//! | `native` | candle: HRM training and inference |
//! | `needle` | the Cactus needle3 query planner |
//! | `agent`, `bedrock`, `paddock` | LLM-driven query planning |
//! | `wasm` | browser bindings |

pub mod api;
pub mod artifact;
pub use api::{Answer, Error, Interval, Options, Refused, SteelDb};

pub mod agent;
pub mod bitmap;
pub mod db;
pub mod vocabulary;
#[cfg(feature = "wasm")]
#[doc(hidden)]
pub mod wasm;
#[doc(hidden)]
pub mod discover;
#[cfg(feature = "embed")]
pub mod discover_ontology;
#[doc(hidden)]
pub mod dsl_trajectories;
pub mod evidence;
pub mod emergent;
#[cfg(feature = "docs")]
pub mod docs;
#[cfg(feature = "ocr")]
pub mod ocr;
pub mod grow;
#[cfg(feature = "native")]
pub mod hrm;
#[doc(hidden)]
pub mod ikl_trajectories;
pub mod index;
pub mod jsonl;
pub mod learn;
pub mod linter;
#[cfg(feature = "native")]
pub mod needle_model;
pub mod mece;
pub mod models;
pub mod paths;
pub mod programs;
pub mod spans;
pub mod registry;
pub mod projector;
pub mod projectors;
#[doc(hidden)]
pub mod trajectories;
#[doc(hidden)]
pub mod tagger_data;
#[cfg(feature = "native")]
pub mod relation_train;
#[cfg(feature = "native")]
pub mod tagger_train;
pub mod text;
pub mod tokenql;
pub mod units;
pub mod dimensions;

pub use bitmap::{Postings, RoarPostings, SetPostings};
pub use db::{Corpus, FolderReport, Hit, QueryOut, Stats};
pub use index::InfonIndex;
pub use projector::{CorpusKind, Projector, Situation};
pub use tokenql::{evaluate, parse, Node, TokenStore};