HyperSteelDb

A database that compiles your question instead of guessing an answer.

Ask it something your documents cannot answer and it tells you so, listing what they can answer. It does not hand back an empty list and leave you to work out which of the two just happened.

[dependencies]
hypersteeldb = "0.5"

No model files, no network, no configuration, no build script. Pure Rust. It also compiles to wasm32 and runs in a browser — try it without installing anything.

A terminal tool, if you would rather not write code

cargo install hypersteeldb --features cli
steel ./corpus

 steel · ./corpus · 21 situations · 4 categories · learn: qwen3:1.7b
┌ you can ask about ───────┐┌ result ───────────────────────────────┐
│▸ battle/*                ││ (and battle/* (not state/negated))    │
│    indigo, defeated      ││   →  9 situations  (41 µs)            │
│  city/*                  ││                                       │
│    species, survey       ││ [2] Morty Shade defeated Wallace Gale │
│  competition/*           ││ [5] Bea Strike defeated Iris Draco    │
└──────────────────────────┘└───────────────────────────────────────┘

The categories are always on screen, because their names come from your text and cannot be guessed. Tab moves between the query line and the category list, Enter runs, l asks a model for a category the deterministic pass missed, s saves the artefact set, ? explains the rest. A refusal is drawn as prominently as an answer.

A saved artefact set at <corpus>/.hypersteeldb is followed automatically on later runs, so a category you learn and save is still there next time — the header says artefact or discovered so you always know which you are looking at. --rediscover ignores it; --artifact <dir> follows a specific one. learn picks the best local model it can find; --model overrides it.

Sixty seconds

use steeldb::SteelDb;

let db = SteelDb::ingest(documents)?;

// Category names come from words your documents actually use,
// so read them before writing a query.
for c in db.categories() {
    println!("you can ask about {}", c.wildcard());
}

match db.query("(and elevation/* (not state/negated))") {
    Ok(answer)   => println!("{} situations", answer.len()),
    Err(refused) => println!("{refused}"),
}

That is cargo run --example quickstart. Its real output:

indexed 8 situations

you can ask about:
  defeated/*     defeated, city, ecruteak, indigo
  elevation/*    elevation, habitat, sootopolis, survey
  permitted/*    permitted, milotic, play, season

(and elevation/* (not state/negated))  ->  3 situations
  [3] A habitat survey recorded Aggron near Sootopolis City at
  [4] A habitat survey recorded Salamence near Sootopolis City

refused: survey/*
  dimension 'survey' is not in this corpus; facets are: defeated,
    elevation, entity, permitted, quantity, rel, state, time
  available: defeated/*, elevation/*, permitted/*

Two things to take from that.

The categories are defeated, elevation and permitted — not battle or survey. They are named after words in your text, so always print db.categories() first. Guessing a category name is the most common way to waste an afternoon here.

And survey/* is refused, not answered with nothing. A category that does not exist and a category with no matches are different facts, and merging them is how a wrong answer gets produced.

How many documents do you need?

Categories come from words that recur across documents, so two documents discover nothing. Around eight gives useful categories; more is better. entity/*, time/*, quantity/* and state/* work at any size, because those come from patterns rather than from statistics.

What this is for

A record of observations, not transactions. Reports, filings, signals, sightings — things that happened, or that someone said happened. You append them; you never edit or delete them. Some of them are wrong, and some are wrong on purpose.

That last part is why a plain triple does not work. Storing (acme, supplies, defence-ministry) asserts it is true. But you did not observe a fact, you observed a claim, and tomorrow another source may claim the opposite. Both are real observations and both have to stay.

So every observation becomes a situation: an n-ary relation with named roles and a polarity, indexed as bits.

cargo run --example observations ingests eight such observations — some asserting a supply relationship, some denying permission for one — and prints what got indexed:

rel/permitted/-
rel/permitted/-/defence-ministry
rel/supplies/+
rel/supplies/+/acme-corp
rel/supplies/+/beta-corp
rel/supplies/-
rel/supplies/-/defence-ministry

Both directions of both relations are present, bound to who was on each side. Nothing was overwritten when the third observation contradicted the first two.

Contradiction is then a measurement, not an error. db.belief(tag) returns two numbers rather than one:

state/asserted   [0.75, 1.00]   ignorance 0.25
state/negated    [0.00, 0.75]   ignorance 0.75

The gap between belief and plausibility is what is not yet ruled out, so "nobody said" stays distinguishable from "sources disagree" — a single probability collapses both to the middle. (combine-ds …) refuses to merge two sources when their conflict exceeds its threshold, rather than averaging them into a consensus nobody holds.

If your data is rows that get updated and deleted, use a database. This is for the case where the disagreement between observations is the signal.

Why an agent can use this without drowning

Give an agent a normal database and two things blow up its context window: it has to learn the data model first, and then it accumulates result rows to reason over.

Both are addressed by the same property — the vocabulary is closed, and a query is type-checked against it before anything runs.

> db.query("gene/brca1")

refused: gene/brca1
  dimension 'gene' is not in this corpus; facets are: entity, field,
    permitted, quantity, rel, state, supplies, time
  available: supplies/*, field/*, permitted/*

That refusal is the schema. An agent learns what is askable in a few dozen tokens, from the error, without a schema dump or example queries. And because results are complete sets rather than ranked samples, answer.len() is exact — an agent can count without pulling rows into its context.

The columns are also disentangled, which is the difference from a RAG embedding: one column is membership of one reified relation, or one entity, or one unit. So first-order logic over the index is set arithmetic, and (and A (not B)) means what it says.

Where a vector search goes wrong on this

The third is the expensive one. A confident wrong answer looks exactly like a right one, so you cannot tell which you got — and neither can an agent calling you in a loop.

Three verbs

ingest

let db = SteelDb::ingest(documents)?;   // from memory
let db = SteelDb::open("corpus/")?;     // a dir of .md / .txt

Reads your documents, works out the categories, indexes everything.

Categories are found by optimal transport. A term's position is the set of documents it appears in; k-means proposes prototypes; entropy-regularised transport assigns each term to one. The target marginal is uniform, so no group can take more than its share and swallow the corpus. That is why this needs no similarity threshold tuned by hand, and no embedding model.

query

Queries are s-expressions: operation first, brackets for grouping. The whole language:

Results are complete sets, never ranked samples, so counting them means something:

let answer = db.query("(and survey/* (num elevation_m gt 1000))")?;
for (id, text) in db.resolve(&answer) {
    println!("{id}: {text}");   // each result traces to its document
}

Because relations record who was on each side, reversing a claim stops it matching rather than merely ranking it lower.

learn (optional)

ingest finds categories from word statistics. A language model can spot ones it missed. That needs a model, so learn is separate, explicit, and returns a suggestion rather than changing anything:

use steeldb::learn::Teacher;

// local, free, nothing leaves the machine
let teacher  = Teacher::ollama("qwen3.5:0.8b")?;
let proposal = teacher.propose_categories(&db).await?;

println!("{proposal}");             // review it first
let verdicts = db.adopt(&proposal); // you decide

The model must support tool calling. learn asks for a structured ontology, not prose. A model without tool calling replies in prose or not at all, and you get:

model did not emit an ontology. `learn` needs a model that supports
tool calling; one that does not will answer in prose or not at all.

Measured against a local ollama, smallest first:

Adopting runs each suggestion through the same test ingest uses: enough coverage, and not a near-duplicate of something already there. A model cannot add a category the deterministic test would reject, which is what makes a small local model safe here.

That is not a claim, it is the observed behaviour. qwen3.5:0.8b on the eight documents above proposes two categories and the gate refuses both:

drop  mortal_sand_castle   detectors never fired on the sample
drop  indigo_invitational  gain 0.000 < threshold 0.050
                           (coverage 0.375, maxcos 1.000 vs 'defeated')

One was invented outright; the other restated a category already present. A bad suggestion is rejected, not absorbed.

Teacher::bedrock points the same call at Amazon Bedrock, which does cost money per call.

learn with the span tagger (optional, better categories)

Word statistics find categories whose names are single recurring words. A trained span tagger finds multi-word domain entities they cannot — on a 7000-document technical corpus the difference is name/*, date/* versus sensing-modality/*, platform/*, energy-storage/*.

cargo install hypersteeldb --features cli,onnx,embed
huggingface-cli download cp500/steeldb-models \
  --local-dir ~/.steeldb/models
steel ./corpus --neural            # reads 100 documents by default

Three stages, and each does a different kind of work:

Stage 3 is not optional and it is the interesting one. A raw cluster's label is a value: a cluster of 6g, 5g, next-generation is not a facet called 6g, it is a facet called network-generation whose values include 6g. Curation abstracts values up to kinds, merges synonyms, and drops boilerplate. Every surviving facet still has to pass the same gate as a locally-discovered one, so the model can enrich the vocabulary but not pollute it.

QTY never enters the codebook: a measurement is a value on a scale, not a kind of thing, so it becomes a numeric field instead — which is the only reason (num …) can be asked later.

The models run once, over the sample. Then save, and every later run is the fast model-free path:

steel ./corpus        # follows .hypersteeldb — no tagger, no curator

Needs a curator model. Any local OpenAI-compatible server works (ollama serve), and Bedrock is measurably better at the job:

steel ./corpus --neural \
  --model bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0

On the same clusters, Haiku abstracts values up to kinds — 6g becomes network-generation, cost-effective becomes cost-attribute — where a small local model keeps the value as the facet name. Needs the bedrock feature and AWS credentials on the standard chain.

Artefacts: run learn once, not every time

learn is slow, needs a model, and may answer differently each time. You do not want that on every startup, so save what it produced:

db.save(".hypersteeldb")?;

.hypersteeldb/
  .gitignore       excludes training/ — written first
  manifest.json    what produced this, and which parts hold text
  vocabulary.json  the categories and their words
  gazetteer.json   multi-word names kept whole
  relations.json   relation verbs
  motifs.json      latent themes
  training/        only if you asked for it
    spans.jsonl    labelled passages for a tagger finetune

Every run after that follows those files:

let db = SteelDb::ingest_using(documents, ".hypersteeldb")?;

Same vocabulary, same answers, no model, no network. A test asserts that ingest and ingest_using produce identical tags, because a reload that drifted would defeat the point of saving.

Commit the directory next to your code. It is small, diffable, and a vocabulary change shows up in review like any other change.

The vocabulary files contain no document text

vocabulary.json, gazetteer.json, relations.json and motifs.json are derived vocabulary: category names, signal words, name surfaces, verbs, theme labels. Not your documents, and not an index of them. A test walks every file and asserts it, checking whole documents and six-word fragments — a partial copy leaks just as surely as a whole one.

You still need the documents to ingest. The artefacts describe how to read them, not what they said.

The finetune set is the one exception, so it is kept apart

let n = db.save_with_training(".hypersteeldb")?;

A finetune set is labelled spans, and a span label is meaningless without the words it points at. So this one file does contain your document text. Three things stop that becoming an accident:

• it goes to training/, not next to the vocabulary;
• save writes the .gitignore excluding it before writing the data, so the rule cannot be missing;
• manifest.json lists it under contains_document_text, so a CI check can refuse to publish without knowing the layout.

After cloning a repository you have the vocabulary and not the training set. That is intended: ingest and query never need it.

What needs a download

The default build has no model dependencies. Everything above works with cargo add hypersteeldb and nothing else.

Trained heads (1.8 MB) ship inside the crate. Larger weights are resolved by steeldb::models::resolve, which searches STEELDB_MODELS, then ~/.steeldb/models, then ./models, and returns an error naming the repository, revision, size and licence if it finds nothing. It never downloads on its own.

Beyond the basics

• Evidence. db.belief("state/negated") returns [belief, plausibility] rather than one number, so "nobody said" stays distinguishable from "sources disagree". (combine-ds …) refuses to merge contradictory sources instead of averaging them into a consensus nobody holds.
• Structure. db.s_path(a, b, 2) finds connections sharing at least two tags, which suppresses the drift you get from following single links. db.filtration(6) sweeps that threshold so real structure separates from coincidence.
• Bounded by construction. Every tag is attached to a situation, so evaluation is flat set intersection rather than an open-ended graph walk. Cost is rows ÷ register width, and no query escapes it.

The name

Registeel is the steel golem of the Pokémon Regi trio — a sealed thing that opens only for the right sequence. Fitting for an engine whose whole argument is that it refuses malformed questions.

Licence

MIT. Third-party models keep their own licences, recorded in steeldb::models::ARTIFACTS with attribution and a pinned revision.