HyperSteelDb

A database that compiles your question instead of guessing an answer.

Ask it something your documents cannot answer and it tells you so, listing what they can answer. It does not hand back an empty list and leave you to work out which of the two just happened.

[dependencies]
hypersteeldb = "0.2"

No model files, no network, no configuration, no build script. Pure Rust. It also compiles to wasm32 and runs in a browser — try it without installing anything.

Sixty seconds

use steeldb::SteelDb;

let db = SteelDb::ingest(documents)?;

// Category names come from words your documents actually use,
// so read them before writing a query.
for c in db.categories() {
    println!("you can ask about {}", c.wildcard());
}

match db.query("(and elevation/* (not state/negated))") {
    Ok(answer)   => println!("{} situations", answer.len()),
    Err(refused) => println!("{refused}"),
}

That is cargo run --example quickstart. Its real output:

indexed 8 situations

you can ask about:
  defeated/*     defeated, city, ecruteak, indigo
  elevation/*    elevation, habitat, sootopolis, survey
  permitted/*    permitted, milotic, play, season

(and elevation/* (not state/negated))  ->  3 situations
  [3] A habitat survey recorded Aggron near Sootopolis City at
  [4] A habitat survey recorded Salamence near Sootopolis City

refused: survey/*
  dimension 'survey' is not in this corpus; facets are: defeated,
    elevation, entity, permitted, quantity, rel, state, time
  available: defeated/*, elevation/*, permitted/*

Two things to take from that.

The categories are defeated, elevation and permitted — not battle or survey. They are named after words in your text, so always print db.categories() first. Guessing a category name is the most common way to waste an afternoon here.

And survey/* is refused, not answered with nothing. A category that does not exist and a category with no matches are different facts, and merging them is how a wrong answer gets produced.

How many documents do you need?

Categories come from words that recur across documents, so two documents discover nothing. Around eight gives useful categories; more is better. entity/*, time/*, quantity/* and state/* work at any size, because those come from patterns rather than from statistics.

The problem this solves

You have a pile of documents and a question like "battles in Johto that weren't part of the Indigo tournament."

Vector search turns your question into numbers, finds the nearest documents, and returns them. That works well for "anything about battles?" and fails for the question above in three specific ways:

The third one is the expensive one. A confident wrong answer looks exactly like a right one, so you cannot tell which you got — and neither can an agent calling you in a loop.

Three verbs

ingest

let db = SteelDb::ingest(documents)?;   // from memory
let db = SteelDb::open("corpus/")?;     // a dir of .md / .txt

Reads your documents, works out the categories, indexes everything.

Categories are found by optimal transport. A term's position is the set of documents it appears in; k-means proposes prototypes; entropy-regularised transport assigns each term to one. The target marginal is uniform, so no group can take more than its share and swallow the corpus. That is why this needs no similarity threshold tuned by hand, and no embedding model.

query

Queries are s-expressions: operation first, brackets for grouping. The whole language:

Results are complete sets, never ranked samples, so counting them means something:

let answer = db.query("(and survey/* (num elevation_m gt 1000))")?;
for (id, text) in db.resolve(&answer) {
    println!("{id}: {text}");   // each result traces to its document
}

Because relations record who was on each side, reversing a claim stops it matching rather than merely ranking it lower.

learn (optional)

ingest finds categories from word statistics. A language model can spot ones it missed. That needs a model, so learn is separate, explicit, and returns a suggestion rather than changing anything:

use steeldb::learn::Teacher;

// local, free, nothing leaves the machine
let teacher  = Teacher::ollama("qwen2.5:0.5b")?;
let proposal = teacher.propose_categories(&db).await?;

println!("{proposal}");             // review it first
let verdicts = db.adopt(&proposal); // you decide

Adopting runs each suggestion through the same test ingest uses: enough coverage, and not a near-duplicate of something already there. A model cannot add a category the deterministic test would reject, which is what makes a small local model safe here. A bad suggestion is rejected, not absorbed.

Teacher::bedrock points the same call at Amazon Bedrock, which does cost money per call.

Artefacts: run learn once, not every time

learn is slow, needs a model, and may answer differently each time. You do not want that on every startup, so save what it produced:

db.save(".hypersteeldb")?;

.hypersteeldb/
  .gitignore       excludes training/ — written first
  manifest.json    what produced this, and which parts hold text
  vocabulary.json  the categories and their words
  gazetteer.json   multi-word names kept whole
  relations.json   relation verbs
  motifs.json      latent themes
  training/        only if you asked for it
    spans.jsonl    labelled passages for a tagger finetune

Every run after that follows those files:

let db = SteelDb::ingest_using(documents, ".hypersteeldb")?;

Same vocabulary, same answers, no model, no network. A test asserts that ingest and ingest_using produce identical tags, because a reload that drifted would defeat the point of saving.

Commit the directory next to your code. It is small, diffable, and a vocabulary change shows up in review like any other change.

The vocabulary files contain no document text

vocabulary.json, gazetteer.json, relations.json and motifs.json are derived vocabulary: category names, signal words, name surfaces, verbs, theme labels. Not your documents, and not an index of them. A test walks every file and asserts it, checking whole documents and six-word fragments — a partial copy leaks just as surely as a whole one.

You still need the documents to ingest. The artefacts describe how to read them, not what they said.

The finetune set is the one exception, so it is kept apart

let n = db.save_with_training(".hypersteeldb")?;

A finetune set is labelled spans, and a span label is meaningless without the words it points at. So this one file does contain your document text. Three things stop that becoming an accident:

• it goes to training/, not next to the vocabulary;
• save writes the .gitignore excluding it before writing the data, so the rule cannot be missing;
• manifest.json lists it under contains_document_text, so a CI check can refuse to publish without knowing the layout.

After cloning a repository you have the vocabulary and not the training set. That is intended: ingest and query never need it.

What needs a download

The default build has no model dependencies. Everything above works with cargo add hypersteeldb and nothing else.

Trained heads (1.8 MB) ship inside the crate. Larger weights are resolved by steeldb::models::resolve, which searches STEELDB_MODELS, then ~/.steeldb/models, then ./models, and returns an error naming the repository, revision, size and licence if it finds nothing. It never downloads on its own.

Beyond the basics

• Evidence. db.belief("state/negated") returns [belief, plausibility] rather than one number, so "nobody said" stays distinguishable from "sources disagree". (combine-ds …) refuses to merge contradictory sources instead of averaging them into a consensus nobody holds.
• Structure. db.s_path(a, b, 2) finds connections sharing at least two tags, which suppresses the drift you get from following single links. db.filtration(6) sweeps that threshold so real structure separates from coincidence.
• Bounded by construction. Every tag is attached to a situation, so evaluation is flat set intersection rather than an open-ended graph walk. Cost is rows ÷ register width, and no query escapes it.

The name

Registeel is the steel golem of the Pokémon Regi trio — a sealed thing that opens only for the right sequence. Fitting for an engine whose whole argument is that it refuses malformed questions.

Licence

MIT. Third-party models keep their own licences, recorded in steeldb::models::ARTIFACTS with attribution and a pinned revision.