# HyperSteelDb
<p align="center">
<img src="https://huggingface.co/spaces/cp500/steeldb-ontology-sensing/resolve/main/registeel.jpg" alt="Registeel EX" width="260">
</p>
**A database that compiles your question instead of guessing an answer.**
Ask it something your documents cannot answer and it tells you so, listing
what they *can* answer. It does not hand back an empty list and leave you
to work out which of the two just happened.
```toml
[dependencies]
hypersteeldb = "0.5"
```
No model files, no network, no configuration, no build script. Pure Rust.
It also compiles to `wasm32` and runs in a browser —
[try it without installing anything](https://huggingface.co/spaces/cp500/steeldb-ontology-sensing).
## A terminal tool, if you would rather not write code
```bash
cargo install hypersteeldb --features cli
steel ./corpus
```
```text
steel · ./corpus · 21 situations · 4 categories · learn: qwen3:1.7b
┌ you can ask about ───────┐┌ result ───────────────────────────────┐
│▸ battle/* ││ (and battle/* (not state/negated)) │
│ indigo, defeated ││ → 9 situations (41 µs) │
│ city/* ││ │
│ species, survey ││ [2] Morty Shade defeated Wallace Gale │
│ competition/* ││ [5] Bea Strike defeated Iris Draco │
└──────────────────────────┘└───────────────────────────────────────┘
```
The categories are always on screen, because their names come from your
text and cannot be guessed. `Tab` moves between the query line and the
category list, `Enter` runs, `l` asks a model for a category the
deterministic pass missed, `s` saves the artefact set, `?` explains the
rest. A refusal is drawn as prominently as an answer.
A saved artefact set at `<corpus>/.hypersteeldb` is followed automatically on
later runs, so a category you learn and save is still there next time — the
header says `artefact` or `discovered` so you always know which you are
looking at. `--rediscover` ignores it; `--artifact <dir>` follows a specific
one. `learn` picks the best local model it can find; `--model` overrides it.
## Sixty seconds
```rust
use steeldb::SteelDb;
let db = SteelDb::ingest(documents)?;
// Category names come from words your documents actually use,
// so read them before writing a query.
for c in db.categories() {
println!("you can ask about {}", c.wildcard());
}
match db.query("(and elevation/* (not state/negated))") {
Ok(answer) => println!("{} situations", answer.len()),
Err(refused) => println!("{refused}"),
}
```
That is `cargo run --example quickstart`. Its real output:
```text
indexed 8 situations
you can ask about:
defeated/* defeated, city, ecruteak, indigo
elevation/* elevation, habitat, sootopolis, survey
permitted/* permitted, milotic, play, season
(and elevation/* (not state/negated)) -> 3 situations
[3] A habitat survey recorded Aggron near Sootopolis City at
[4] A habitat survey recorded Salamence near Sootopolis City
refused: survey/*
dimension 'survey' is not in this corpus; facets are: defeated,
elevation, entity, permitted, quantity, rel, state, time
available: defeated/*, elevation/*, permitted/*
```
Two things to take from that.
The categories are `defeated`, `elevation` and `permitted` — not `battle`
or `survey`. They are named after words in your text, so **always print
`db.categories()` first**. Guessing a category name is the most common way
to waste an afternoon here.
And `survey/*` is **refused**, not answered with nothing. A category that
does not exist and a category with no matches are different facts, and
merging them is how a wrong answer gets produced.
### How many documents do you need?
Categories come from words that recur *across* documents, so two documents
discover nothing. Around eight gives useful categories; more is better.
`entity/*`, `time/*`, `quantity/*` and `state/*` work at any size, because
those come from patterns rather than from statistics.
---
## The problem this solves
You have a pile of documents and a question like *"battles in Johto that
weren't part of the Indigo tournament."*
Vector search turns your question into numbers, finds the nearest
documents, and returns them. That works well for *"anything about
battles?"* and fails for the question above in three specific ways:
| *A but **not** B* | similarity has no "not" — documents with both rank **higher** |
| *how many in total?* | you get the top 10; you cannot know if there were 11 |
| *which trainers used a Mewtwo?* | if nothing mentions Mewtwo, you still get answers |
The third one is the expensive one. A confident wrong answer looks exactly
like a right one, so you cannot tell which you got — and neither can an
agent calling you in a loop.
## Three verbs
| `ingest` | nothing | deterministic, free | every run |
| `query` | nothing | microseconds | every request |
| `learn` | a model | one model call | once, optional |
### ingest
```rust
let db = SteelDb::ingest(documents)?; // from memory
let db = SteelDb::open("corpus/")?; // a dir of .md / .txt
```
Reads your documents, works out the categories, indexes everything.
Categories are found by **optimal transport**. A term's position is the set
of documents it appears in; k-means proposes prototypes; entropy-regularised
transport assigns each term to one. The target marginal is uniform, so no
group can take more than its share and swallow the corpus. That is why this
needs no similarity threshold tuned by hand, and no embedding model.
### query
Queries are s-expressions: operation first, brackets for grouping. The
whole language:
| `battle/defeated` | situations carrying that exact tag |
| `battle/*` | any value in the `battle` category |
| `(and A B)` | both |
| `(or A B)` | either |
| `(not A)` | exclude |
| `rel/defeated/+/morty-shade` | the *acting* side of a relation |
| `motif/series` | a theme two documents share without a shared word |
| `(num elevation_m gt 1000)` | numeric comparison |
| `(evidence A :min-bel 0.8)` | only where evidence for A is strong |
| `(s-path :s 2 (source A) (target B))` | linked by ≥2 shared tags |
| `(combine-ds :max-conflict 0.2 …)` | merge sources, or refuse |
Results are **complete sets**, never ranked samples, so counting them means
something:
```rust
let answer = db.query("(and survey/* (num elevation_m gt 1000))")?;
for (id, text) in db.resolve(&answer) {
println!("{id}: {text}"); // each result traces to its document
}
```
Because relations record who was on each side, reversing a claim stops it
matching rather than merely ranking it lower.
### learn (optional)
`ingest` finds categories from word statistics. A language model can spot
ones it missed. That needs a model, so `learn` is separate, explicit, and
**returns a suggestion rather than changing anything**:
```rust
use steeldb::learn::Teacher;
// local, free, nothing leaves the machine
let teacher = Teacher::ollama("qwen3.5:0.8b")?;
let proposal = teacher.propose_categories(&db).await?;
println!("{proposal}"); // review it first
let verdicts = db.adopt(&proposal); // you decide
```
**The model must support tool calling.** `learn` asks for a structured
ontology, not prose. A model without tool calling replies in prose or not at
all, and you get:
```text
model did not emit an ontology. `learn` needs a model that supports
tool calling; one that does not will answer in prose or not at all.
```
Measured against a local ollama, smallest first:
| `qwen2.5:0.5b` | **no** | returns `tool_calls: null` for every request |
| `granite3-moe:1b` | no | answers in prose |
| `qwen3:1.7b` | no | empty reply |
| `granite3-moe:3b` | yes | 3.6 s, proposed nothing |
| `functiongemma` | yes | 23 s, proposed nothing |
| `qwen3.5:0.8b` | yes | 7 s, and actually proposes |
Adopting runs each suggestion through the same test `ingest` uses: enough
coverage, and not a near-duplicate of something already there. **A model
cannot add a category the deterministic test would reject**, which is what
makes a small local model safe here.
That is not a claim, it is the observed behaviour. `qwen3.5:0.8b` on the
eight documents above proposes two categories and the gate refuses both:
```text
drop mortal_sand_castle detectors never fired on the sample
drop indigo_invitational gain 0.000 < threshold 0.050
(coverage 0.375, maxcos 1.000 vs 'defeated')
```
One was invented outright; the other restated a category already present.
A bad suggestion is rejected, not absorbed.
`Teacher::bedrock` points the same call at Amazon Bedrock, which does cost
money per call.
### learn with the span tagger (optional, better categories)
Word statistics find categories whose names are single recurring words. A
trained span tagger finds multi-word domain entities they cannot — on a
7000-document technical corpus the difference is `name/*`, `date/*` versus
`sensing-modality/*`, `platform/*`, `energy-storage/*`.
```bash
cargo install hypersteeldb --features cli,onnx,embed
huggingface-cli download cp500/steeldb-models \
--local-dir ~/.steeldb/models
steel ./corpus --neural # reads 100 documents by default
```
Three stages, and each does a different kind of work:
| 1. tag | the span tagger reads each sentence | typed spans: `ENT`, `GEO`, `REL`, `TIME`, `QTY` |
| 2. group | optimal transport, per kind | raw clusters — **candidate values, not facets** |
| 3. curate | a model merges and names | facet **types**, noise dropped |
Stage 3 is not optional and it is the interesting one. A raw cluster's label
is a *value*: a cluster of `6g, 5g, next-generation` is not a facet called
`6g`, it is a facet called `network-generation` whose values include `6g`.
Curation abstracts values up to kinds, merges synonyms, and drops boilerplate.
Every surviving facet still has to pass the same gate as a locally-discovered
one, so the model can enrich the vocabulary but not pollute it.
`QTY` never enters the codebook: a measurement is a value on a scale, not a
kind of thing, so it becomes a numeric field instead — which is the only
reason `(num …)` can be asked later.
The models run **once, over the sample**. Then save, and every later run is
the fast model-free path:
```bash
steel ./corpus # follows .hypersteeldb — no tagger, no curator
```
Needs a curator model. Any local OpenAI-compatible server works
(`ollama serve`), and Bedrock is measurably better at the job:
```bash
steel ./corpus --neural \
--model bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0
```
On the same clusters, Haiku abstracts values up to kinds — `6g` becomes
`network-generation`, `cost-effective` becomes `cost-attribute` — where a
small local model keeps the value as the facet name. Needs the `bedrock`
feature and AWS credentials on the standard chain.
## Artefacts: run learn once, not every time
`learn` is slow, needs a model, and may answer differently each time. You
do not want that on every startup, so save what it produced:
```rust
db.save(".hypersteeldb")?;
```
```text
.hypersteeldb/
.gitignore excludes training/ — written first
manifest.json what produced this, and which parts hold text
vocabulary.json the categories and their words
gazetteer.json multi-word names kept whole
relations.json relation verbs
motifs.json latent themes
training/ only if you asked for it
spans.jsonl labelled passages for a tagger finetune
```
Every run after that follows those files:
```rust
let db = SteelDb::ingest_using(documents, ".hypersteeldb")?;
```
Same vocabulary, same answers, no model, no network. A test asserts that
`ingest` and `ingest_using` produce identical tags, because a reload that
drifted would defeat the point of saving.
**Commit the directory next to your code.** It is small, diffable, and a
vocabulary change shows up in review like any other change.
### The vocabulary files contain no document text
`vocabulary.json`, `gazetteer.json`, `relations.json` and `motifs.json` are
derived vocabulary: category names, signal words, name surfaces, verbs,
theme labels. Not your documents, and not an index of them. A test walks
every file and asserts it, checking whole documents *and* six-word
fragments — a partial copy leaks just as surely as a whole one.
You still need the documents to `ingest`. The artefacts describe *how* to
read them, not *what* they said.
### The finetune set is the one exception, so it is kept apart
```rust
let n = db.save_with_training(".hypersteeldb")?;
```
A finetune set is labelled spans, and a span label is meaningless without
the words it points at. So this one file **does** contain your document
text. Three things stop that becoming an accident:
- it goes to `training/`, not next to the vocabulary;
- `save` writes the `.gitignore` excluding it **before** writing the data,
so the rule cannot be missing;
- `manifest.json` lists it under `contains_document_text`, so a CI check
can refuse to publish without knowing the layout.
After cloning a repository you have the vocabulary and not the training
set. That is intended: `ingest` and `query` never need it.
## What needs a download
The default build has no model dependencies. Everything above works with
`cargo add hypersteeldb` and nothing else.
| — (default) | ingest, query, artefacts, evidence, topology | none |
| `cli` | the `steel` terminal tool (includes `paddock`) | yours |
| `paddock` | `learn` via any OpenAI-compatible server | yours |
| `bedrock` | `learn` via Amazon Bedrock | hosted |
| `onnx` | span tagger for `--neural` discovery | 168 MB |
| `embed` | embedding-based discovery | 15 MB |
| `native` | in-process inference | varies |
Trained heads (1.8 MB) ship inside the crate. Larger weights are resolved
by `steeldb::models::resolve`, which searches `STEELDB_MODELS`, then
`~/.steeldb/models`, then `./models`, and returns an error naming the
repository, revision, size and licence if it finds nothing. **It never
downloads on its own.**
## Beyond the basics
- **Evidence.** `db.belief("state/negated")` returns `[belief,
plausibility]` rather than one number, so "nobody said" stays
distinguishable from "sources disagree". `(combine-ds …)` refuses to
merge contradictory sources instead of averaging them into a consensus
nobody holds.
- **Structure.** `db.s_path(a, b, 2)` finds connections sharing at least
two tags, which suppresses the drift you get from following single
links. `db.filtration(6)` sweeps that threshold so real structure
separates from coincidence.
- **Bounded by construction.** Every tag is attached to a situation, so
evaluation is flat set intersection rather than an open-ended graph
walk. Cost is rows ÷ register width, and no query escapes it.
## The name
Registeel is the steel golem of the Pokémon Regi trio — a sealed thing that
opens only for the right sequence. Fitting for an engine whose whole
argument is that it refuses malformed questions.
## Licence
MIT. Third-party models keep their own licences, recorded in
`steeldb::models::ARTIFACTS` with attribution and a pinned revision.