Expand description
Artefacts — what learn leaves behind, and what ingest and query follow.
learn is the only step that needs a model. Running it on every startup would make the system slow,
non-deterministic, and dependent on a service being reachable. So it runs once and writes its result to
a directory of small JSON files; every run after that reads those files and is offline and reproducible.
.hypersteeldb/
manifest.json what produced this set, when, and a digest of each part
vocabulary.json the categories and the words each claims
gazetteer.json multi-word mentions kept whole
relations.json relation verbs, with polarity
motifs.json latent motifs, so `motif/*` tokens survive a reloaduse steeldb::SteelDb;
// once — discovers (or learns) and records the result
let db = SteelDb::ingest(["…documents…"])?;
db.save(".hypersteeldb")?;
// every time after — same vocabulary, no model, no network, byte-identical results
let db = SteelDb::ingest_using(["…documents…"], ".hypersteeldb")?;§What is deliberately not in here
No document text. An artefact set is derived vocabulary: category names, the words that signal them, mention surfaces, relation verbs. It is not a copy of the corpus, and it is not an index. Two reasons that matters: an artefact set is safe to commit to a repository next to code, and shipping one does not ship the documents it was learned from. A test asserts the files contain no sentence from the corpus.
You still need the documents to ingest. The artefacts say how to read them, not what they said.
Structs§
- Artifacts
- The complete set. Small, human-readable, and diffable on purpose — a vocabulary change should show up in review like any other change.
- Category
Record - One category and the words that put a document in it.
- Registration
- A registered entity mention: how it appears, and what it resolves to.
Enums§
- Artifact
Error - Why an artefact set could not be read or written.
Constants§
- GAZETTEER
- IGNORE_
FILE - MANIFEST
- File names inside an artefact directory.
- MOTIFS
- Latent motifs: the themes a situation can join without sharing a keyword (the paper’s sixth dimension).
- RELATIONS
- SCHEMA
- Current on-disk layout version. Bumped when a field’s meaning changes, so an old set is refused rather than silently misread.
- TRAINING_
DIR - Subdirectory for the finetuning set. Separate because, unlike everything else here, it necessarily contains document text.
- TRAINING_
FILE - VOCABULARY