# HyperSteelDb
<p align="center">
<img src="https://huggingface.co/spaces/cp500/steeldb-ontology-sensing/resolve/main/registeel.jpg" alt="Registeel EX" width="260">
</p>
**A database that compiles your question instead of guessing an answer.**
Ask it something your documents cannot answer and it tells you so, listing
what they *can* answer. It does not hand back an empty list and leave you
to work out which of the two just happened.
```toml
[dependencies]
hypersteeldb = "0.5"
```
No model files, no network, no configuration, no build script. Pure Rust.
It also compiles to `wasm32` and runs in a browser —
[try it without installing anything](https://huggingface.co/spaces/cp500/steeldb-ontology-sensing).
## A terminal tool, if you would rather not write code
```bash
cargo install hypersteeldb --features cli
steel ./corpus
```
```text
steel · ./corpus · 21 situations · 4 categories · learn: qwen3:1.7b
┌ you can ask about ───────┐┌ result ───────────────────────────────┐
│▸ battle/* ││ (and battle/* (not state/negated)) │
│ indigo, defeated ││ → 9 situations (41 µs) │
│ city/* ││ │
│ species, survey ││ [2] Morty Shade defeated Wallace Gale │
│ competition/* ││ [5] Bea Strike defeated Iris Draco │
└──────────────────────────┘└───────────────────────────────────────┘
```
The categories are always on screen, because their names come from your
text and cannot be guessed. `Tab` moves between the query line and the
category list, `Enter` runs, `l` asks a model for a category the
deterministic pass missed, `s` saves the artefact set, `?` explains the
rest. A refusal is drawn as prominently as an answer.
A saved artefact set at `<corpus>/.hypersteeldb` is followed automatically on
later runs, so a category you learn and save is still there next time — the
header says `artefact` or `discovered` so you always know which you are
looking at. `--rediscover` ignores it; `--artifact <dir>` follows a specific
one. `learn` picks the best local model it can find; `--model` overrides it.
## Sixty seconds
```rust
use steeldb::SteelDb;
let db = SteelDb::ingest(documents)?;
// Category names come from words your documents actually use,
// so read them before writing a query.
for c in db.categories() {
println!("you can ask about {}", c.wildcard());
}
match db.query("(and elevation/* (not state/negated))") {
Ok(answer) => println!("{} situations", answer.len()),
Err(refused) => println!("{refused}"),
}
```
That is `cargo run --example quickstart`. Its real output:
```text
indexed 8 situations
you can ask about:
defeated/* defeated, city, ecruteak, indigo
elevation/* elevation, habitat, sootopolis, survey
permitted/* permitted, milotic, play, season
(and elevation/* (not state/negated)) -> 3 situations
[3] A habitat survey recorded Aggron near Sootopolis City at
[4] A habitat survey recorded Salamence near Sootopolis City
refused: survey/*
dimension 'survey' is not in this corpus; facets are: defeated,
elevation, entity, permitted, quantity, rel, state, time
available: defeated/*, elevation/*, permitted/*
```
Two things to take from that.
The categories are `defeated`, `elevation` and `permitted` — not `battle`
or `survey`. They are named after words in your text, so **always print
`db.categories()` first**. Guessing a category name is the most common way
to waste an afternoon here.
And `survey/*` is **refused**, not answered with nothing. A category that
does not exist and a category with no matches are different facts, and
merging them is how a wrong answer gets produced.
### How many documents do you need?
Categories come from words that recur *across* documents, so two documents
discover nothing. Around eight gives useful categories; more is better.
`entity/*`, `time/*`, `quantity/*` and `state/*` work at any size, because
those come from patterns rather than from statistics.
---
## What this is for
A record of **observations**, not transactions. Reports, filings, signals,
sightings — things that happened, or that someone *said* happened. You append
them; you never edit or delete them. Some of them are wrong, and some are
wrong on purpose.
That last part is why a plain triple does not work. Storing
`(acme, supplies, defence-ministry)` asserts it is true. But you did not
observe a fact, you observed a *claim*, and tomorrow another source may claim
the opposite. Both are real observations and both have to stay.
So every observation becomes a **situation**: an n-ary relation with named
roles and a polarity, indexed as bits.
`cargo run --example observations` ingests eight such observations — some
asserting a supply relationship, some denying permission for one — and prints
what got indexed:
```text
rel/permitted/-
rel/permitted/-/defence-ministry
rel/supplies/+
rel/supplies/+/acme-corp
rel/supplies/+/beta-corp
rel/supplies/-
rel/supplies/-/defence-ministry
```
Both directions of both relations are present, bound to who was on each side.
Nothing was overwritten when the third observation contradicted the first two.
Contradiction is then a **measurement**, not an error. `db.belief(tag)`
returns two numbers rather than one:
```text
state/asserted [0.75, 1.00] ignorance 0.25
state/negated [0.00, 0.75] ignorance 0.75
```
The gap between belief and plausibility is what is *not yet ruled out*, so
"nobody said" stays distinguishable from "sources disagree" — a single
probability collapses both to the middle. `(combine-ds …)` **refuses** to
merge two sources when their conflict exceeds its threshold, rather than
averaging them into a consensus nobody holds.
If your data is rows that get updated and deleted, use a database. This is
for the case where the disagreement between observations is the signal.
## Why an agent can use this without drowning
Give an agent a normal database and two things blow up its context window: it
has to learn the data model first, and then it accumulates result rows to
reason over.
Both are addressed by the same property — the vocabulary is closed, and a
query is type-checked against it *before* anything runs.
```text
> db.query("gene/brca1")
refused: gene/brca1
dimension 'gene' is not in this corpus; facets are: entity, field,
permitted, quantity, rel, state, supplies, time
available: supplies/*, field/*, permitted/*
```
That refusal **is** the schema. An agent learns what is askable in a few
dozen tokens, from the error, without a schema dump or example queries. And
because results are complete sets rather than ranked samples, `answer.len()`
is exact — an agent can count without pulling rows into its context.
The columns are also disentangled, which is the difference from a RAG
embedding: one column is membership of one reified relation, or one entity,
or one unit. So first-order logic over the index is set arithmetic, and
`(and A (not B))` means what it says.
## Where a vector search goes wrong on this
| *A but **not** B* | similarity has no "not" — documents with both rank **higher** |
| *how many in total?* | you get the top 10; you cannot know if there were 11 |
| *which suppliers shipped part X?* | if nothing mentions part X, you still get answers |
The third is the expensive one. A confident wrong answer looks exactly like a
right one, so you cannot tell which you got — and neither can an agent
calling you in a loop.
## Three verbs
| `ingest` | nothing | deterministic, free | every run |
| `query` | nothing | microseconds | every request |
| `learn` | a model | one model call | once, optional |
### ingest
```rust
let db = SteelDb::ingest(documents)?; // from memory
let db = SteelDb::open("corpus/")?; // a dir of .md / .txt
```
Reads your documents, works out the categories, indexes everything.
Categories are found by **optimal transport**. A term's position is the set
of documents it appears in; k-means proposes prototypes; entropy-regularised
transport assigns each term to one. The target marginal is uniform, so no
group can take more than its share and swallow the corpus. That is why this
needs no similarity threshold tuned by hand, and no embedding model.
### query
Queries are s-expressions: operation first, brackets for grouping. The
whole language:
| `battle/defeated` | situations carrying that exact tag |
| `battle/*` | any value in the `battle` category |
| `(and A B)` | both |
| `(or A B)` | either |
| `(not A)` | exclude |
| `rel/defeated/+/morty-shade` | the *acting* side of a relation |
| `motif/series` | a theme two documents share without a shared word |
| `(num elevation_m gt 1000)` | numeric comparison |
| `(evidence A :min-bel 0.8)` | only where evidence for A is strong |
| `(s-path :s 2 (source A) (target B))` | linked by ≥2 shared tags |
| `(combine-ds :max-conflict 0.2 …)` | merge sources, or refuse |
Results are **complete sets**, never ranked samples, so counting them means
something:
```rust
let answer = db.query("(and survey/* (num elevation_m gt 1000))")?;
for (id, text) in db.resolve(&answer) {
println!("{id}: {text}"); // each result traces to its document
}
```
Because relations record who was on each side, reversing a claim stops it
matching rather than merely ranking it lower.
### learn (optional)
`ingest` finds categories from word statistics. A language model can spot
ones it missed. That needs a model, so `learn` is separate, explicit, and
**returns a suggestion rather than changing anything**:
```rust
use steeldb::learn::Teacher;
// local, free, nothing leaves the machine
let teacher = Teacher::ollama("qwen3.5:0.8b")?;
let proposal = teacher.propose_categories(&db).await?;
println!("{proposal}"); // review it first
let verdicts = db.adopt(&proposal); // you decide
```
**The model must support tool calling.** `learn` asks for a structured
ontology, not prose. A model without tool calling replies in prose or not at
all, and you get:
```text
model did not emit an ontology. `learn` needs a model that supports
tool calling; one that does not will answer in prose or not at all.
```
Measured against a local ollama, smallest first:
| `qwen2.5:0.5b` | **no** | returns `tool_calls: null` for every request |
| `granite3-moe:1b` | no | answers in prose |
| `qwen3:1.7b` | no | empty reply |
| `granite3-moe:3b` | yes | 3.6 s, proposed nothing |
| `functiongemma` | yes | 23 s, proposed nothing |
| `qwen3.5:0.8b` | yes | 7 s, and actually proposes |
Adopting runs each suggestion through the same test `ingest` uses: enough
coverage, and not a near-duplicate of something already there. **A model
cannot add a category the deterministic test would reject**, which is what
makes a small local model safe here.
That is not a claim, it is the observed behaviour. `qwen3.5:0.8b` on the
eight documents above proposes two categories and the gate refuses both:
```text
drop mortal_sand_castle detectors never fired on the sample
drop indigo_invitational gain 0.000 < threshold 0.050
(coverage 0.375, maxcos 1.000 vs 'defeated')
```
One was invented outright; the other restated a category already present.
A bad suggestion is rejected, not absorbed.
`Teacher::bedrock` points the same call at Amazon Bedrock, which does cost
money per call.
### learn with the span tagger (optional, better categories)
Word statistics find categories whose names are single recurring words. A
trained span tagger finds multi-word domain entities they cannot — on a
7000-document technical corpus the difference is `name/*`, `date/*` versus
`sensing-modality/*`, `platform/*`, `energy-storage/*`.
```bash
cargo install hypersteeldb --features cli,onnx,embed
huggingface-cli download cp500/steeldb-models \
--local-dir ~/.steeldb/models
steel ./corpus --neural # reads 100 documents by default
```
Three stages, and each does a different kind of work:
| 1. tag | the span tagger reads each sentence | typed spans: `ENT`, `GEO`, `REL`, `TIME`, `QTY` |
| 2. group | optimal transport, per kind | raw clusters — **candidate values, not facets** |
| 3. curate | a model merges and names | facet **types**, noise dropped |
Stage 3 is not optional and it is the interesting one. A raw cluster's label
is a *value*: a cluster of `6g, 5g, next-generation` is not a facet called
`6g`, it is a facet called `network-generation` whose values include `6g`.
Curation abstracts values up to kinds, merges synonyms, and drops boilerplate.
Every surviving facet still has to pass the same gate as a locally-discovered
one, so the model can enrich the vocabulary but not pollute it.
`QTY` never enters the codebook: a measurement is a value on a scale, not a
kind of thing, so it becomes a numeric field instead — which is the only
reason `(num …)` can be asked later.
The models run **once, over the sample**. Then save, and every later run is
the fast model-free path:
```bash
steel ./corpus # follows .hypersteeldb — no tagger, no curator
```
Needs a curator model. Any local OpenAI-compatible server works
(`ollama serve`), and Bedrock is measurably better at the job:
```bash
steel ./corpus --neural \
--model bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0
```
On the same clusters, Haiku abstracts values up to kinds — `6g` becomes
`network-generation`, `cost-effective` becomes `cost-attribute` — where a
small local model keeps the value as the facet name. Needs the `bedrock`
feature and AWS credentials on the standard chain.
## Artefacts: run learn once, not every time
`learn` is slow, needs a model, and may answer differently each time. You
do not want that on every startup, so save what it produced:
```rust
db.save(".hypersteeldb")?;
```
```text
.hypersteeldb/
.gitignore excludes training/ — written first
manifest.json what produced this, and which parts hold text
vocabulary.json the categories and their words
gazetteer.json multi-word names kept whole
relations.json relation verbs
motifs.json latent themes
training/ only if you asked for it
spans.jsonl labelled passages for a tagger finetune
```
Every run after that follows those files:
```rust
let db = SteelDb::ingest_using(documents, ".hypersteeldb")?;
```
Same vocabulary, same answers, no model, no network. A test asserts that
`ingest` and `ingest_using` produce identical tags, because a reload that
drifted would defeat the point of saving.
**Commit the directory next to your code.** It is small, diffable, and a
vocabulary change shows up in review like any other change.
### The vocabulary files contain no document text
`vocabulary.json`, `gazetteer.json`, `relations.json` and `motifs.json` are
derived vocabulary: category names, signal words, name surfaces, verbs,
theme labels. Not your documents, and not an index of them. A test walks
every file and asserts it, checking whole documents *and* six-word
fragments — a partial copy leaks just as surely as a whole one.
You still need the documents to `ingest`. The artefacts describe *how* to
read them, not *what* they said.
### The finetune set is the one exception, so it is kept apart
```rust
let n = db.save_with_training(".hypersteeldb")?;
```
A finetune set is labelled spans, and a span label is meaningless without
the words it points at. So this one file **does** contain your document
text. Three things stop that becoming an accident:
- it goes to `training/`, not next to the vocabulary;
- `save` writes the `.gitignore` excluding it **before** writing the data,
so the rule cannot be missing;
- `manifest.json` lists it under `contains_document_text`, so a CI check
can refuse to publish without knowing the layout.
After cloning a repository you have the vocabulary and not the training
set. That is intended: `ingest` and `query` never need it.
## What needs a download
The default build has no model dependencies. Everything above works with
`cargo add hypersteeldb` and nothing else.
| — (default) | ingest, query, artefacts, evidence, topology | none |
| `cli` | the `steel` terminal tool (includes `paddock`) | yours |
| `paddock` | `learn` via any OpenAI-compatible server | yours |
| `bedrock` | `learn` via Amazon Bedrock | hosted |
| `onnx` | span tagger for `--neural` discovery | 168 MB |
| `embed` | embedding-based discovery | 15 MB |
| `native` | in-process inference | varies |
Trained heads (1.8 MB) ship inside the crate. Larger weights are resolved
by `steeldb::models::resolve`, which searches `STEELDB_MODELS`, then
`~/.steeldb/models`, then `./models`, and returns an error naming the
repository, revision, size and licence if it finds nothing. **It never
downloads on its own.**
## Beyond the basics
- **Evidence.** `db.belief("state/negated")` returns `[belief,
plausibility]` rather than one number, so "nobody said" stays
distinguishable from "sources disagree". `(combine-ds …)` refuses to
merge contradictory sources instead of averaging them into a consensus
nobody holds.
- **Structure.** `db.s_path(a, b, 2)` finds connections sharing at least
two tags, which suppresses the drift you get from following single
links. `db.filtration(6)` sweeps that threshold so real structure
separates from coincidence.
- **Bounded by construction.** Every tag is attached to a situation, so
evaluation is flat set intersection rather than an open-ended graph
walk. Cost is rows ÷ register width, and no query escapes it.
## The name
Registeel is the steel golem of the Pokémon Regi trio — a sealed thing that
opens only for the right sequence. Fitting for an engine whose whole
argument is that it refuses malformed questions.
## Licence
MIT. Third-party models keep their own licences, recorded in
`steeldb::models::ARTIFACTS` with attribution and a pinned revision.