Expand description
The version-matched Teksilo documentation corpus.
This crate is data, not logic: one prebuilt retrieval index over the hand-written guides and the worked examples — their text, chunked, with BM25 statistics and a dense vector per chunk — embedded at build time and parsed on first use. It carries no ML dependency; the vectors are computed at release time, so a consumer only ever encodes the query.
It is versioned in lockstep with the framework, which is the whole point:
cargo install cargo-teksilo --version X resolves teksilo-corpus X
through cargo’s own resolver, so an answer can never come from a
different teksilo than the one an app pinned.
§The corpus is one file
corpus/index.json is the whole crate’s data, embedded by
INDEX_JSON. Every Chunk carries its own text, and its path
names the original file in the teksilo repository —
docs/scroll-area.md, examples/simple_button/src/main.rs — so a
search result cites something that exists and resolves on GitHub.
It did not always work that way. An earlier layout mirrored all 158
guides and example sources into corpus/ and stored only a line range
into the copy. That put a second copy of every guide in the tree, one
fuzzy-open away from the real one, and an edit made in the copy was
discarded without a word by the next regeneration. Moving the text into
the index was size-neutral — index.json grew by roughly what the
copies weighed — and int8-quantised vectors had already bought the
headroom that splitting the text out was originally for.
Chunk::line_start / Chunk::line_end survive as provenance:
an inclusive, 0-based range naming where in path this text was found,
so a reader can be sent to the right lines. tools/build_corpus.py
verifies every range against its source file before emitting the index,
and the line split is str::lines’ — on \n only, with a trailing
\r stripped — so Rust and the generator agree on what a line index
means.
§Term strings are interned
Index::terms is the sorted table of every token in the corpus;
Chunk::tokens and Index::df are keyed by a term’s index into it
rather than repeating the string thousands of times. Resolve one with
Index::term, or a whole chunk’s with Index::chunk_terms. In
JSON those maps are objects with decimal string keys ({"17":5}),
since JSON object keys must be strings; serde deserializes them
straight into the u32 keys used here.
§The index is built in two passes, in this order
python3 tools/build_corpus.pychunks the guides and examples and writesindex.jsonwithencoder,encoder_dimand everyChunk::embeddingasnull. It regenerates the file from scratch, so it necessarily discards any vectors already there — except the ones it can carry forward by chunk-text hash, which is every vector whose text an edit did not touch.cargo teksilo build-vectorsencodes each chunk, quantises the result, and writes the vectors back into that sameindex.json, settingencoderandencoder_dim.
Running them in the other order leaves a corpus with no vectors, which
degrades to lexical search rather than breaking — but it is a
regression, so the order is part of the release procedure, and
build_corpus.py --check is deliberately blind to the three vector
fields: it checks that the chunking is current, which is the thing
that goes stale when a guide is edited.
Structs§
- Chunk
- One retrieval chunk of the corpus, as written by
tools/build_corpus.py. - Chunk
Embedding - A chunk’s dense vector: int8 values plus the scale that decodes them.
- Index
- The full retrieval index, embedded as
INDEX_JSON.
Enums§
- Corpus
Error - An error parsing the embedded corpus index.
Constants§
- CORPUS_
VERSION - The teksilo version this corpus was generated from.
Statics§
- INDEX_
JSON - The corpus index, embedded at build time.
Functions§
- index
- Parse and cache the embedded corpus index.