HyperSteelDb

[dependencies]
hypersteeldb = "0.1"

use steeldb::SteelDb;

let db = SteelDb::ingest([
    "Morty Shade defeated Wallace Gale at Ecruteak City in 2025.",
    "A habitat survey recorded Aggron near Sootopolis City at 1082 m.",
])?;

match db.query("(and battle/* (not state/negated))") {
    Ok(answer)   => println!("{} matching situations", answer.len()),
    Err(refused) => println!("{refused}"),   // tells you what the data *does* contain
}

No model files. No network. No configuration. The vocabulary is derived from your documents.

The problem this solves

You have a pile of documents and a question like "battles in Johto that weren't part of the Indigo tournament."

A vector search converts your question to numbers, finds the nearest documents, and returns them. It works well for "anything about battles?" and fails for that question in three specific ways:

That third one is the expensive one. A confident wrong answer looks exactly like a right one, so you cannot tell which you received — and neither can an agent calling you in a loop.

What this does instead

It learns which categories your documents actually support, checks your question against them before running anything, then executes the survivors as bit arithmetic.

for c in db.categories() {
    println!("you can ask about {}", c.wildcard());   // battle/*, survey/*, series/* …
}

db.query("gene/brca1")?;   // Err: refused, and lists the categories that exist

A question the data cannot answer returns Err, not an empty list. That is the whole point: an empty result and an unanswerable question are different facts, and merging them is how a wrong answer gets produced.

Three verbs

ingest

let db = SteelDb::ingest(documents)?;        // from memory
let db = SteelDb::open("corpus/")?;          // from a directory of .md / .txt

Reads your documents, works out the categories, and indexes everything. Pure Rust — no ONNX, no Python, no downloads. Compiles to wasm32 and runs in a browser.

query

Queries are s-expressions, borrowed from Lisp: operation first, brackets for grouping. The whole language:

Results are complete sets, never ranked samples, so counting them means something:

let answer = db.query("(and survey/* (num elevation_m gt 1000))")?;
for (id, text) in db.resolve(&answer) {
    println!("{id}: {text}");         // every result traces back to its document
}

learn (optional)

ingest finds categories from word statistics. A language model can spot ones it missed. That needs a model, so learn is separate, explicit, and returns a suggestion rather than changing anything:

use steeldb::learn::Teacher;

let teacher  = Teacher::ollama("qwen2.5:0.5b")?;      // local, free, nothing leaves the machine
let proposal = teacher.propose_categories(&db).await?;

println!("{proposal}");                                // review it first
let verdicts = db.adopt(&proposal);                    // you decide

Adopting runs each suggestion through the same test ingest uses: enough coverage, and not a near-duplicate of something already there. A model cannot add a category that the deterministic test would reject — so using a small local model is safe. A bad suggestion is rejected, not absorbed.

You can also point it at Bedrock (Teacher::bedrock), which does cost money per call.

Artefacts: run learn once, not every time

learn is slow, needs a model, and may answer slightly differently each time. You do not want that on every startup. So save what it produced:

db.save(".hypersteeldb")?;                              // after learning

.hypersteeldb/
  .gitignore         excludes training/ — written before anything else
  manifest.json      what produced this, when, and which parts hold document text
  vocabulary.json    the categories and their words
  gazetteer.json     multi-word names kept whole
  relations.json     relation verbs
  training/          only if you asked for it — see below
    spans.jsonl      labelled passages for a tagger finetune

Then every run after that follows those files:

let db = SteelDb::ingest_using(documents, ".hypersteeldb")?;

Same vocabulary, same answers, no model, no network. Commit the directory next to your code — it is small, diffable, and a vocabulary change shows up in review like any other change.

The vocabulary files contain no document text

vocabulary.json, gazetteer.json and relations.json are derived vocabulary: category names, signal words, name surfaces, verbs. Not your documents, and not an index of them. Tests assert it — both against whole documents and against six-word fragments, since a partial copy leaks just as surely as a whole one.

You still need the documents to ingest. The artefacts describe how to read them, not what they said.

The finetune set is the one exception, so it is kept apart

let n = db.save_with_training(".hypersteeldb")?;   // n labelled passages

A finetune set is labelled spans, and a span label is meaningless without the words it points at. So this one file does contain your document text. Three things keep that from becoming an accident:

• it goes to training/, not next to the vocabulary;
• save writes the .gitignore excluding it before writing the data, so the rule cannot be missing;
• manifest.json lists it under contains_document_text, so a CI check can refuse to publish a directory without needing to know the layout.

After cloning a repository you will have the vocabulary and not the training set. That is the intended state: ingest and query never need it.

What you get for free, and what needs a download

The default build has no model dependencies at all. Everything above works with cargo add hypersteeldb and nothing else.

Optional features add trained models, which are fetched on request, never automatically:

Trained heads (1.8 MB) ship inside the crate. Larger weights are resolved by steeldb::models::resolve, which searches STEELDB_MODELS, then ~/.steeldb/models, then ./models, and returns an error naming the repository, revision, size and licence if it finds nothing. It never downloads on its own.

Beyond the basics

• Evidence. db.belief("state/negated") returns [belief, plausibility] rather than one number, so "nobody said" is distinguishable from "sources disagree". (combine-ds …) refuses to merge contradictory sources instead of averaging them into a consensus nobody holds.
• Structure. db.s_path(a, b, 2) finds connections that share at least two tags, which suppresses the drift you get from following single links. db.filtration(6) sweeps that threshold and shows real structure separating from coincidence.
• Bounded by construction. Every tag is attached to a situation, so evaluation is flat set intersection rather than an open-ended graph walk. Cost is rows ÷ register width, and no query can escape it.

Try it without installing anything

An interactive version of the paper runs the real engine in your browser, compiled to WebAssembly: https://huggingface.co/spaces/cp500/steeldb-ontology-sensing

The name

Registeel is the steel golem of the Pokémon Regi trio — a sealed thing that only opens for the right sequence. Fitting for an engine whose whole argument is that it refuses malformed questions.

Licence

MIT. Third-party models keep their own licences, recorded in steeldb::models::ARTIFACTS with attribution and a pinned revision.