Crate steeldb

Expand description

A database that compiles questions instead of guessing answers.

Most systems answer a question about documents by similarity: find the nearest text and return it. That works until the question involves a combination (A but not B), a complete count, or something the documents simply do not contain — where a similarity search still returns its closest guess, and a guess is indistinguishable from an answer.

SteelDB learns which categories your documents actually support, type-checks a question against them before anything runs, and executes the survivors as bitwise set algebra over compressed bitmaps. A question the data cannot answer is refused, with the alternatives that do exist.

§Start here

use steeldb::SteelDb;

// No model files needed: the vocabulary is discovered from the text.
let db = SteelDb::ingest(documents)?;

// Category names come from the words the documents use, so read them before writing a query.
for c in db.categories() {
    println!("can ask about {}", c.wildcard());
}

match db.query("(and elevation/* (not state/negated))") {
    Ok(answer)   => println!("{} situations", answer.len()),
    Err(refused) => println!("{refused}"),   // says what the data does contain
}

§Three verbs

verb	what it needs	what it costs
`SteelDb::ingest`	nothing — no models, no network	deterministic and free
`SteelDb::query`	nothing	microseconds
`learn`	credentials and a network	a model call, and a bill

The asymmetry is deliberate. ingest and query are pure; learn calls a language model, so it lives in its own module, is async, is feature-gated, and returns a proposal rather than changing your vocabulary. You review it and SteelDb::adopt it, at which point the same gate that governs local discovery decides what survives — a model cannot add a category a deterministic test would have rejected.

SteelDb is the whole API for most uses. Answer is a complete set rather than a ranked sample, so counting it means something. Refused is an error rather than an empty result because those are different facts, and conflating them is how a confident wrong answer gets produced.

§The query language

Queries are s-expressions — operation first, nested lists, as in Lisp. The whole grammar:

form	meaning
`category/value`	situations carrying that exact tag
`category/*`	any value in that category
`(and A B)`	intersection
`(or A B)`	union
`(not A)`	difference
`(num field op value)`	numeric comparison; `op` is `ge gt le lt eq ne`
`(evidence A :min-bel f)`	only where belief in `A` reaches `f`
`(s-path :s n (source A) (target B))`	situations on a chain sharing ≥ `n` tags per step
`(combine-ds :max-conflict f …)`	fuse independent evidence, or refuse

There is deliberately almost no syntax to get wrong, which matters when the author is a language model.

§Beyond the basics

evidence — Dempster–Shafer belief intervals, and the conflict metric that refuses to fuse contradictory sources rather than averaging them into a consensus nobody holds.
programs — higher-order structure: s-paths, and the primal/dual s-filtration.
emergent — how the vocabulary is discovered from prose, with no model.
models — where trained weights come from. Nothing downloads without being asked.
linter — the type-checker, if you want to validate without executing.

§Installing

The crate is published as hypersteeldb and imported as steeldb:

[dependencies]
hypersteeldb = "0.1"

(The bare name steeldb was taken on crates.io in 2023 by an unrelated project, so the package carries the longer name while the import stays short.)

§Features

The default build is pure Rust with no model dependencies and compiles to wasm32.

feature	adds
`embed`	static embeddings + optimal-transport discovery (links a C regex library)
`onnx`	the trained span tagger
`native`	candle: HRM training and inference
`needle`	the Cactus needle3 query planner
`agent`, `bedrock`, `paddock`	LLM-driven query planning
`wasm`	browser bindings

Re-exports§

pub use api::Answer;
pub use api::Error;
pub use api::Interval;
pub use api::Options;
pub use api::Refused;
pub use api::SteelDb;
pub use bitmap::Postings;
pub use bitmap::RoarPostings;
pub use bitmap::SetPostings;
pub use db::Corpus;
pub use db::FolderReport;
pub use db::Hit;
pub use db::QueryOut;
pub use db::Stats;
pub use index::InfonIndex;
pub use projector::CorpusKind;
pub use projector::Projector;
pub use projector::Situation;
pub use tokenql::evaluate;
pub use tokenql::parse;
pub use tokenql::Node;
pub use tokenql::TokenStore;

Modules§

agent: Agent layer — retrieval-as-reasoning over a Corpus, driven by a configurable LLM provider. The loop and tools are provider-agnostic; backends (Bedrock / Paddock) are selected by ProviderConfig and gated by cargo features so neither is a mandatory dependency.
api: The public API. One type to learn, and a return type that makes refusal impossible to ignore.
artifact: Artefacts — what learn leaves behind, and what ingest and query follow.
bitmap: Postings — an exact set of unsigned situation ids with roaring-compatible set algebra.
db: Corpus — the query facade the app (and, later, the C ABI / napi bindings) drives.
dimensions: Vocabulary Space V — the canonical URI taxonomy (paper §1, §2). Every producer’s output is normalised into one of six hierarchical, slash-delimited dimensions. The hierarchy is load-bearing, not cosmetic: subtree wildcards only work if the URIs have depth. glob_match already globs the whole token, so qty/temp/*, time/2026/*, geo/apac/*, rel/supplies/+ all resolve the moment emission is hierarchical — whereas a flat qty/27c or time/q3-2026 is unreachable by prefix.
discover_ontology: Sinkhorn-OT ontology discovery — the text counterpart of structured-data discover. Extract candidate terms from a corpus, embed them with the model2vec static embedder, and cluster them into a MECE facet codebook via k-means++ prototypes + entropy-regularised optimal transport (src/text/ot.rs). No hand-authored schema — the facets emerge from the corpus. The TUI view (src/bin/ontology.rs) animates the Sinkhorn plan sharpening and the transport cost falling as it converges.
docs: Any-doc bridge — lower PDF / DOCX / PPTX / HTML / MD / TXT to plain text, natively in Rust, so the folder ingest can project every readable document into the hypergraph (not just tabular/text files). The extracted text feeds the same TextEngine (SPO tagger + SPLADE + gazetteer) as .txt.
emergent: Emergent ontology from prose — discover facets from unlabelled natural language, with no model and no planted hints.
evidence: Dempster–Shafer evidence combination (paper §4.1–4.2).
grow: Step 3: ontology growth — the MDL-style split criterion from design.py.
hrm: HRM two-timescale reasoning core — a faithful candle port of the reference python/splade/hrm_core.py + spo_tagger.py::HRMTagger.
index: InfonIndex — the sparse incidence store: one posting set per infon token = the situations where it holds. Generic over the Postings backend so the same query path runs on either the HashSet baseline or roaring. atom resolves a token or a glob pattern to the UNION of matching postings, matching the TS atom / fnmatch semantics.
jsonl: Streaming JSONL loader. Each line is one situation { "tokens": [...], "num": {...} }; the line index (0-based) is the situation id. Builds token -> ascending sids in a single pass — the sids land in id order for free, so the index build hits the fast sorted path.
learn: Learn — the third verb, and the one that works differently on purpose.
linter: The linter — SteelDB’s compile-time boundary between a probabilistic agent and the deterministic engine (paper §2). Before any IKL query executes, every atom is validated against the corpus Vocabulary Space V (the Bitmap Symbol Table). Unknown terms are rejected with a “did you mean” nearest-term correction; wildcards are expanded to their real child terms; and unbalanced parentheses are repaired. This is what turns a hallucinated URI into a caught compile error instead of a silently wrong answer.
mece: MECE gate — the information-gain test that decides whether a facet earns its place (design.py §“MECE gate: Collectively-Exhaustive (coverage) × Mutually-Exclusive (orthogonality)”).
models: Where the weights come from.
needle_model: Pure-Rust (candle) port of the Cactus needle-1 Simple Attention Network — the encoder-decoder we CAN faithfully reimplement (its architecture is documented; needle-2’s novel MHC/engram/MTP stack is not). This is the model the finetuning harness trains: load the published bf16 safetensors, run teacher-forced, and serve fully in-Rust (no cactus C++ wheel, no Python).
ocr: OCR fallback for scanned/image PDFs with no text layer — the bundled PP-OCRv6 pipeline, natively in Rust. Two ONNX stages: DB detection (image → text-region probability map) and CRNN recognition (each cropped region → text via CTC). Pages are rasterized with poppler pdftoppm. Gated behind the ocr feature (implies docs/onnx).
paths: Location-independent model resolution. An installed steeldb binary must find its bundled models whatever the working directory is — so we search, in priority order: an explicit env override, the per-user cache (~/.steeldb/models, %LOCALAPPDATA%\steeldb\models), next-to-the-executable (<exe>/models, and <exe>/../../models for the dev target/<profile>/ layout), then ./models.
programs: Bitmap-program analytics over the infon index — the deterministic template library the agent composes beyond bare retrieve. Ported from the mother’s hypergraph/programs.ts + query-engine.ts (fd46262): partitions (breakdown/crosstab), salience (rank), the structural s-graph (cooccurs/s_path/s_clusters), and stepwise answerability (narrow).
projector: Projector — the pluggable ingest seam, analogous to DuckDB’s table functions / replacement scans.
projectors: Concrete projectors. Each lowers one file format to the common Situation stream.
registry: Step 4: the registry — version the spec, its metrics, and the growth log so an ontology is reproducible and auditable (design.py: “version the model + facet spec + metrics in a filesystem registry (models/<id>/, registry.json)”).
relation_train: Head C: the biaffine relation scorer — the head that makes dimension 2 real.
spans: Span boundary repair — turn model predictions over sub-word pieces into spans over whole words.
tagger_train: Step 2: tagger finetuning in Rust (candle) — the port of the reference spo_tagger.py::train.
text: Native text projection primitives (multilingual, CPU-only), ported from the TS ingest pipeline. • model2vec — the EN/JA/KO static span embedder (reuses the shipped potion.f32 artifact). • ot — Sinkhorn optimal transport + k-means codebook for auto ontology sensing. The SPLADE / SPO-tagger ONNX stages plug in next via the ort crate (ONNX Runtime), running the same exported .onnx graphs unchanged.
tokenql: Token-only IKL — the single retrieval language, an s-expression over the postings store. Ported 1:1 from the TS tokenql.ts evaluator: <atom> a tag or glob pattern → UNION of matching tokens’ postings, ∩ scope (and A B …) → ∩ (also the default for a bare list) (or A B …) → ∪ (not A) → scope − A (closed-world inversion) (The num typed range predicate is stubbed here — it belongs to the columnar numeric layer, which is a separate benchmark; set-algebra is what we’re measuring.)
units: Units + numeric parsing — turns raw cell values and tagged quantity spans into comparable numbers so IKL’s (num <field> <op> <value>) range predicate can filter on magnitude (e.g. “braking distance ≤ 70 m”, “range ≥ 800”, “price < 30000”). Two entry points: • parse_number — a bare numeric (CSV cell): strips thousands separators, currency, %. • parse_quantity — a value+unit span (from the QTY tagger): canonicalises to an SI dimension so “100 km/h” and “27.8 m/s” compare, tagged under a dimensional field (qty-length, qty-mass…).
vocabulary: Step 0: the facet spec — the corpus’s Vocabulary Space V, discovered before ingest and persisted beside the data. Ported from the reference design.py (Step-0 ontology design + growth).

Crate steeldb

Crate steeldb Copy item path

§Start here

§Three verbs

§The query language

§Beyond the basics

§Installing

§Features

Re-exports§

Modules§

Crate steeldb