Skip to main content

Crate teksilo_corpus

Crate teksilo_corpus 

Source
Expand description

The version-matched Teksilo documentation corpus.

This crate is data, not logic: one prebuilt retrieval index over the hand-written guides and the worked examples — their text, chunked, with BM25 statistics and a dense vector per chunk — embedded at build time and parsed on first use. It carries no ML dependency; the vectors are computed at release time, so a consumer only ever encodes the query.

It is versioned in lockstep with the framework, which is the whole point: cargo install cargo-teksilo --version X resolves teksilo-corpus X through cargo’s own resolver, so an answer can never come from a different teksilo than the one an app pinned.

§The corpus is one file

corpus/index.json is the whole crate’s data, embedded by INDEX_JSON. Every Chunk carries its own text, and its path names the original file in the teksilo repository — docs/scroll-area.md, examples/simple_button/src/main.rs — so a search result cites something that exists and resolves on GitHub.

It did not always work that way. An earlier layout mirrored all 158 guides and example sources into corpus/ and stored only a line range into the copy. That put a second copy of every guide in the tree, one fuzzy-open away from the real one, and an edit made in the copy was discarded without a word by the next regeneration. Moving the text into the index was size-neutral — index.json grew by roughly what the copies weighed — and int8-quantised vectors had already bought the headroom that splitting the text out was originally for.

Chunk::line_start / Chunk::line_end survive as provenance: an inclusive, 0-based range naming where in path this text was found, so a reader can be sent to the right lines. tools/build_corpus.py verifies every range against its source file before emitting the index, and the line split is str::lines’ — on \n only, with a trailing \r stripped — so Rust and the generator agree on what a line index means.

§Term strings are interned

Index::terms is the sorted table of every token in the corpus; Chunk::tokens and Index::df are keyed by a term’s index into it rather than repeating the string thousands of times. Resolve one with Index::term, or a whole chunk’s with Index::chunk_terms. In JSON those maps are objects with decimal string keys ({"17":5}), since JSON object keys must be strings; serde deserializes them straight into the u32 keys used here.

§The index is built in two passes, in this order

  1. python3 tools/build_corpus.py chunks the guides and examples and writes index.json with encoder, encoder_dim and every Chunk::embedding as null. It regenerates the file from scratch, so it necessarily discards any vectors already there — except the ones it can carry forward by chunk-text hash, which is every vector whose text an edit did not touch.
  2. cargo teksilo build-vectors encodes each chunk, quantises the result, and writes the vectors back into that same index.json, setting encoder and encoder_dim.

Running them in the other order leaves a corpus with no vectors, which degrades to lexical search rather than breaking — but it is a regression, so the order is part of the release procedure, and build_corpus.py --check is deliberately blind to the three vector fields: it checks that the chunking is current, which is the thing that goes stale when a guide is edited.

Structs§

Chunk
One retrieval chunk of the corpus, as written by tools/build_corpus.py.
ChunkEmbedding
A chunk’s dense vector: int8 values plus the scale that decodes them.
Index
The full retrieval index, embedded as INDEX_JSON.

Enums§

CorpusError
An error parsing the embedded corpus index.

Constants§

CORPUS_VERSION
The teksilo version this corpus was generated from.

Statics§

INDEX_JSON
The corpus index, embedded at build time.

Functions§

index
Parse and cache the embedded corpus index.