# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.90+, edition 2024, 64-bit only (turbovec refuses 32-bit). Nothing has to
be installed first: turbovec 1.0.0 dropped BLAS, so there is no OpenBLAS step on
Linux any more, and the TLS stack is rustls rather than the system OpenSSL. The
packaged Linux binary must keep needing nothing but glibc and libstdc++ —
`release.yml` reads its `NEEDED` entries and fails on `libssl`, `libcrypto` or
`libopenblas`. First build compiles bundled SQLite and downloads ONNX Runtime —
minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo test --test endpoint # the HTTP MCP endpoint; no model, so it is in that set
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --test image -- --ignored # images; downloads CLIP as well
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
cargo test --release --test retrieval -- --ignored --nocapture # retrieval quality
```
`tests/retrieval.rs` is the harness behind every retrieval claim this repository
makes. It runs a fixed set of 107 questions with ground-truth spans from
`tests/fixtures/retrieval/questions.yaml` — identifier-shaped, concept-shaped and
multi-hop — and prints hit@1, hit@3, hit@8, bytes per answer, the graph list's
marginal contribution, and from 0.23.0 calls and tokens per answered question for
`brief` against the search-then-read path. It asserts two things: the wrong-yes
count for `path` is zero, and `tools/list` costs under 1 120 tokens — thirteen
tools measure 4 441 bytes, about 1 111.
The questions are split development/sealed by a recorded seed in
`tests/fixtures/retrieval/split.yaml`, and the sealed set is scored only when
`SEMLITH_RETRIEVAL_SEALED` is set. Development work sees the development set and
nothing else; a number tuned against is not a measurement. Change ranking, chunking or the
tool schemas and run it; a number moving is the report, and an adjective is not.
The question set's spans are line ranges in the 0.14.0 merge, and the `symbol`
field is the durable anchor when one moves — the file's header says how to
re-derive it.
`tests/endpoint.rs` is the exception in the other direction: it drives the
daemon's `/mcp` route and its credential, neither of which needs an embedding
model, so it runs in the default set and a break in the agent key's reach fails
CI rather than an `--ignored` run somebody remembered to do.
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
- **`exact.f32`** — from 0.23.0, the full-precision copy of what the codes
approximate. Fixed-size `[id][dim f32]` records appended in id order, read back
by binary search for the handful of candidates a query is about to reorder. A
store written before 0.23.0 has none and is ranked by the codes alone, which
`semlith stats` says out loud.
A store that has indexed an image holds a third, `images/`, in the same shard
layout at CLIP's 512 dimensions. It is opened lazily, so a store that never
meets an image never grows the directory.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched `4 * k` deep, then fused by reciprocal rank
(`1 / (60 + rank)`). Scores are fusion scores, not cosines — meaningful for
ordering within one result set, meaningless across queries. Query text reaches
FTS5 as bare terms, never as typed, because `MATCH` is a query language.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way. A store that holds
images loads CLIP's two encoders on top of that, on the first image it indexes
or is searched for — measured per store rather than folded into the number
above, which is about the text path.
Module responsibilities:
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/brief.rs` | `semlith brief` / `semlith_brief`: the ranked spans, their text and one-hop edges, composed from the primitives and fitted to a token budget. No engine of its own |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/chunk.rs` | File bytes → text → chunks. Cut at definitions where tree-sitter found them and at headings in Markdown; 800 chars and 2 overlap lines are the fallback and the budget; 8 MiB cap |
| `src/formats.rs` | Readers for the thirteen non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/graph.rs` | tree-sitter extraction, the bounded traversals over the edges, and the ranked walk search expands through |
| `src/pattern.rs` | `semlith pattern`: one tree-sitter query over the indexed files of one language |
| `src/ledger.rs` | The one place a retrieval is recorded, whichever surface answered it |
| `src/image.rs` | Image support: the five extensions, and the CLIP pair that makes a picture comparable with a sentence |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/add.rs` | `semlith add`: the one outbound fetch, its refusals, and where a fetched file lands |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | `docs/clients.md`'s client stanzas, parsed, so the portal shows the tested text |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **There is no cookie.** The portal's session token travels in a
`Semlith-Token` header, because every port on `localhost` is the same site and
`SameSite=Strict` never separated this daemon from a page served by anything
else on `127.0.0.1`. Nothing may set a cookie or read one. A write is refused
before the token is compared unless it carries a JSON content type and, from a
client that sends fetch metadata, `Sec-Fetch-Site: same-origin` — which is why
`http::answer` checks the origin rule first and returns 403 with an empty body.
The page and its own static assets are the one credential-free surface, because
a browser attaches no header to a stylesheet.
- **The home directory is read in one place.** `home::user_home` is the only
function in `src/` that reads `HOME`, and on Windows it is the only one that
reads `USERPROFILE` or `HOMEDRIVE`+`HOMEPATH`. Nine call sites used to ask the
environment themselves and each had a different answer when it was not there,
which is the whole of issues #70 to #73: the store home, the model cache, the
portal's directory browser, the deny-list, the index boundary and `semlith
setup` disagreed about where home was. `tests/home.rs` fails on a tenth
lookup, so one cannot be added by accident.
- **Nothing falls back to the working directory.** `user_home`, `home_or_error`,
`stores_root`, `bin_dir`, `registry_path`, `agent_key_path`,
`Registry::dir_of` and `model_cache_dir` all return a `Result`, and no variant
of any of them returns a default. A machine with no home gets an error naming
`SEMLITH_HOME`; it does not get a store, a registry, an agent key and 52 MB of
weights written into whatever directory the process started in. The rules that
depend on the home fail closed with it: `filter::denied` refuses a path it
cannot check against `~/.ssh`, and the portal's `dirs` route answers 500 rather
than rooting at a volume.
- **A path on its way out is plain.** `plain()` strips the Windows verbatim
`\\?\` prefix where a path becomes text — hits, file rows, image rows, symbol
rows, call sites — and nowhere else. The store keeps the verbatim form, which
is what makes a path longer than 260 characters work, and `semlith read`
accepts either.
- **The deny-list is one table.** `filter::DENIED_DIRS` and
`filter::DENIED_NAMES` in `filter.rs`, applied by `Boundary::refuses` to walked
entries and explicitly named paths alike. A second copy anywhere is a rule that
will disagree with itself. The hidden-file rule applies to an explicit path
too: that is the whole of what `~/.ssh/id_rsa` used to slip past.
- **A store outside the store home is opened only when the registry trusts it.**
`Registry::trusts` is the one predicate; `home::resolve` and `home::all_dirs`
both ask it, and `Discovery::read` asks it before it will read a `daemon.json`
at all. `--store` and `SEMLITH_STORE` bypass it deliberately — naming a store
is an instruction rather than a discovery.
- **A store connection refuses writes until a write path asks.** `store::open`
sets `query_only`, and `Semlith::writing` lifts it around `index_set`,
`forget_held` and the ledger's `record_retrieval`. A fourth write path has to
take the same guard or SQLite will refuse it, which is the point.
- **The two Linux release binaries load ONNX Runtime from beside themselves.**
`embed::link_runtime` runs before anything asks for a model and is a no-op in
every build without the `dynamic-ort` feature. Anything that loads a model
through a path that does not call it will work on every platform except the one
the release ships.
- **A chunk starts where the code does, and the fixed window is the fallback.**
From 0.22.0 `chunk::chunk_file` cuts a file at the definitions `graph::extract`
found, walking each one upwards over the doc comment and attributes attached to
it, so a constant and the six lines saying what it is for are one chunk instead
of two halves of two. Markdown cuts at its headings. Everything else — a
language with no grammar, a document a reader turned into prose, a plain text
file — keeps the 800-character window it always had. The budget still ends a
chunk, so a function longer than the budget is split as anything else is; what
a cut buys is that a definition never *begins* mid-chunk. The two-line overlap
is dropped where the next chunk starts on a cut, because that boundary is not
arbitrary and repeating across it is the blurring the cut exists to remove.
This is why `graph::extract` now runs above the chunking in `index_set_writing`
rather than below it.
- **`Chunk::context` is shown to the model and stored nowhere.** A Markdown
chunk three headings deep is written as though the reader has the two above it
in mind, so the heading path is prepended to what is embedded. It cannot be
part of `chunk.text`: `Semlith::read` maps every line of a chunk to
`start_line + offset`, and one invented line at the front would shift every
span the store can answer with. FTS5 indexes the stored text, so the path is
not keyword-searchable either — which is correct, because it is not in the
file.
- **Definition chunking is a store format, and it is format 3.** A store written
by 0.21.0 holds fixed windows, opens unchanged and answers; its files have not
changed, the rule for what a chunk is has, so the hash check would leave it on
the old rule for ever. The first index pass under 0.22.0 therefore re-chunks
everything it walks, and writes the format row only at the end of a pass that
swept the whole store and was not stopped — a pass over one directory leaves
the rest of the store on the old rule, and the row would be a claim about files
the run never opened. `semlith stats` says which rule a store is on.
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. From 0.18.0 this is what every
registration semlith writes launches, which is why no client configuration
carries the agent key and why a rotation reconfigures nothing: the key is read
from `~/.semlith/agent.key` by this process, not expanded from the
environment by the client. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the three halves can never disagree about eligibility — graph
expansion resolves its neighbours through `store::symbols_by_names`, which
takes the same predicate, so a chunk outside the filter cannot arrive through
the graph either.
- **The graph list is a ranked walk, not a hop.** From 0.16.0 `graph::expand` is
a personalised PageRank seeded with each fused hit's own contribution: a chunk
both lists found seeds twice as hard as one only a single list found, and a
chunk holding three symbols splits its mass between them rather than seeding
each as though it were a hit of its own. Neighbours are read through a closure,
one name at a time, and cached — a name is asked about once however many rounds
run, so the walk holds the frontier and never the whole graph, and `MAX_NODES`
is still the bound. The maths takes a closure rather than a database handle
precisely so it is unit-tested against a literal adjacency map with no store
and no model.
- **The rerank has no model and no learned weights, and its factors are
tiebreaks.** Graph distance from the seeds, and freshness. The whole span is
under 1.5x — asserted, not intended — so a chunk the query matched badly cannot
climb over one it matched well. A third factor, a lift for a chunk inside a
named definition, was specified, built, measured and removed in the same
release: in a code repository it is a second and blunter `prefer: code` applied
to every query, and it fights the real one. Do not reintroduce it, and do not
add a learned weight here. The per-repository learned profile is still a
roadmap item; the version it used to be numbered as was reassigned when the
roadmap was renumbered on 2026-09-16, so this sentence names no release.
- **An identifier-shaped query is answered by the definition, above the
fusion.** `search_preferring` reads the chunks that define the exact name
through `store::symbols_by_names` — the same filter every other list sees, so
it cannot return a chunk the filter excluded — and lifts them to the front of
the ranked list with a stable sort, badged `definition`. It is precedence, not
a weight: the list contributes 0.0 to the fusion score and exists in the fused
set only so the chunk is present and carries its badge. Reciprocal-rank fusion
is nearly flat across the first ranks, so one authoritative list placing a
definition first was outvoted by two vague lists placing something else fourth
and fifth plus a graph list derived from them — five identifier questions on
the pinned corpus had no satisfying span in the first eight results, two of
them definitions FTS5 ranked first by itself. Do not give this list a weight
instead: tuned high it drags a definition's neighbours up through the graph
list, tuned low it does nothing.
- **The query's shape is read in one place.** `shape_of` in `lib.rs`, from the
query text and nothing else, so every surface that prints it calls that
function and there is no second classifier. The portal draws the hint from
`/api/search`'s own `shape_label` and `weighting` fields for the same reason: a
classifier in the browser would be a second opinion about an answer the server
already ranked. An identifier-shaped query weights the keyword list twice; a
question-shaped one leaves the two level. `prefer` multiplies and never
filters, so `prefer: code` over a corpus of prose still answers.
- **`edges.line` is where the source text sat, not where anything is defined.**
Added by the same `ALTER TABLE` mechanism `hint` uses, so `FORMAT_VERSION` does
not move, and NULL on every row an older binary wrote. A renderer prints
nothing for a NULL rather than falling back to the enclosing definition's line:
those are different lines, often hundreds apart, and a wrong call site is worse
than no call site.
- **`aliases` is a dependency kind.** A re-export or an aliased import becomes a
symbol of its own plus an `aliases` edge to the bare name it stands for, and
the kind joins `DEPENDENCY_KINDS` so a walk can cross it — a walk that would
not answered "not connected" about code that is connected. A re-export that
does not rename emits nothing; there is no name change to cross.
- **`read` answers from chunks, never from disk.** `Semlith::read` returns a span
stitched from the store's own rows, by line number rather than by
concatenation, because chunks overlap by two lines and concatenating repeats
the seam. Reading the file off disk would answer for content semlith was never
allowed to index — an agent holding the agent key could name `~/.ssh/id_rsa` as
a span. A span path is resolved by suffix against what was indexed, because a
locate answer prints a store-relative path and `files.path` is absolute; two
files matching one suffix is an error naming both, never a guess.
- **There is one language table, and it is `filter::LANGUAGES`.** From 0.17.0 the
extractor has no list of its own: `graph::has_graph` answers "does this
language carry symbols and edges" by asking whether a grammar exists, and
`graph::language_of` dispatches through the same `filter::language_of_path`
the search filter uses. The six-language copy that used to live in `graph.rs`
drifted from the forty-six the filter advertised for four releases, so
`--language kotlin` narrowed a search perfectly well and `semlith symbol`
then answered nothing about the same corpus, with nothing saying why. Do not
add a second list; add a row to the one table and a grammar beside it.
`every_advertised_language_has_a_working_grammar` fails for a row that has neither a
grammar nor an entry in `graph::WITHOUT_GRAMMAR` giving the reason.
- **A grammar's licence is a gate, not a detail.** Every parser is compiled into
an Apache-2.0 binary, so a copyleft grammar would take the whole crate with
it. `tests/licences.rs` reads cargo's own metadata and refuses anything
outside a permissive allowlist, and it regenerates `THIRD-PARTY-NOTICES` so
the notice those licences require cannot go stale. A language whose only
grammar is copyleft ships without a graph and is named in
`graph::WITHOUT_GRAMMAR` — it is never taken anyway.
- **A grammar is a crates.io dependency, never a git one.** `cargo publish`
rejects git dependencies, so a grammar that is not published is either
replaced by another published crate or vendored with its licence. Two
languages already needed the first: `tree-sitter` sets `links =
"tree-sitter"`, so a grammar crate that depends on `tree-sitter` itself rather
than on `tree-sitter-language` cannot coexist with the version this crate
links. That is why clojure uses `tree-sitter-clojure-orchard` and perl uses
`ts-parser-perl`.
- **A language proves itself with a fixture, not with a grammar.** Every row in
`filter::LANGUAGES` has a file under `tests/fixtures/graph/<language>/` and a
row in `graph::tests::FIXTURES` declaring the symbols and edges it must yield.
A grammar that compiles and extracts nothing passes every other gate in the
crate; this is the one it fails. Write the row first and make the query meet
it — loosening a row to match what the query happened to produce is the move
the table exists to prevent.
- **Text predicates go inside the pattern's own parentheses.** `(#eq? @a "id")`
written after a pattern rather than inside it is a pattern of its own and
filters nothing, silently. That is how `rel="stylesheet"` briefly became an
HTML symbol.
- **`pattern` reads its grammar from `graph::language_grammar`.** The extractor's
own table, so a language `pattern` accepts is a language the graph accepts and
there is no second list to drift. Its text comes from the store's chunks too,
and a file whose first line the store does not hold is skipped rather than
parsed at line numbers that would be wrong.
- **The graph is extracted inside `index_set`, never separately.** It is written
where the file id and the new chunk ids are both already in hand, after
`delete_file` has taken the old rows away. That is the whole freshness claim:
the pass that re-embeds a file is the pass that re-extracts it, so there is no
artifact that can be stale. A `semlith graph build` command would be the one
change that breaks the release's central property — do not add one.
- **`edges.dst` is a symbol *name*, not an id.** `edges.src` is an id and
cascades with its file; the target is resolved through `symbols(name)` at
query time. With an id there, re-indexing `b.rs` would reissue its symbol ids
and silently orphan every edge pointing into it from `a.rs` — the graph would
rot on exactly the operation this design exists to make safe. It also lets an
edge to something unindexed (a standard-library call) still be recorded.
- **Confidence is four values, and only two of them are stored.** `extracted` (the
file said where the name came from) and `inferred` (a bare name match) are
written into `edges.confidence` at extraction time. `resolved` (the candidates
ranked down to one) and `ambiguous` (they did not) are computed by
`store::edges_out` when a query runs and are never written down — deliberately,
because re-indexing a target changes which definitions exist and a stored
ranking would go stale the moment it did. `edges_in` answers from the edge's own
`src` id, so callers still carry the stored value and there is nothing to
resolve. Nothing that renders an edge may present one value as another.
**What to trust, in order.** `extracted` and `resolved` are answers: the source
named the target, or the ranking left exactly one candidate standing. `inferred`
is a hint worth checking. `ambiguous` is not an answer at all — it means the
store holds several definitions of that name and cannot say which one this is,
and it carries the count so a renderer says "4 definitions" rather than printing
four calls. Never quote an ambiguous edge as a call site without opening the
file.
- **`edges.hint` is what the source said, not what the query decided.** The module
of a scoped call, the receiver of a method call, the object of a qualified
Python or TypeScript call. Nullable, NULL on every row an older binary wrote,
added by the `ALTER TABLE` in `store::add_columns` that runs on open — which is
why `FORMAT_VERSION` does not move. The ranking that reads it prefers, in order:
the calling file's own definition, a definition whose file the hint names, one in
a file the caller imports, then a unique definition in the corpus. An edge the
syntax tree already settled stays `extracted` and is not re-ranked: a fact
outranks a ranking. Rust's supplement must not emit the path segment of a scoped
call as a second `calls` edge — `store::edges_out()` is a call to `edges_out`,
not to `store`.
- **Traversal is bounded and reads one hop at a time.** No `petgraph`, no
in-memory whole-graph structure: `graph::neighbours` and
`graph::shortest_path` walk the indexed `edges(src)`/`edges(dst)` columns
under a depth limit and `graph::MAX_NODES`, which is what keeps peak RSS flat
as the corpus grows. `shortest_path` follows `DEPENDENCY_KINDS` only — the
structural edges are true and useless for reachability, since every symbol is
one hop from the file that defines it.
- **A path walks definitions, not names.** A node in the traversal is a name, a
file and a line, and a chain may only leave from the definition it arrived at.
Refusing ambiguous edges is not enough on its own: at 0.14.0 every hop of
`call_tool -> record_retrieval` on this repository's own store resolved to one
definition, and the chain was still false, because hop 3 arrived at `search` in
`lib.rs` and hop 4 left from `search` in `routes.rs`. Each hop true, the chain
not. (That pair is genuinely connected from 0.15.0, which is the release that
made `call_tool` record; `search_in -> record_retrieval` is the case that still
has no chain.) So `semlith path` refuses by default and says "not connected within N hops
by resolved edges"; `--all-edges` / `all_edges: true` walks the old way and
labels it — both endpoints per hop with file and line, a seam where the chain
changed subject, a trailer whose four confidence counts add up to the hop count,
and the line "A hypothesis, not a finding." `--strict` / `strict: true` states
the default out loud and wins over `--all-edges`. One renderer serves the CLI and
the MCP reply, so the two cannot describe one answer differently.
- **`semlith neighbors` collapses and hides, and `--all` undoes both.** Callees to
a name with several definitions are one row carrying the count, because four
rows saying `get` read as four calls. Targets the store holds no definition for
are counted rather than silently omitted, and `--all` / `all: true` lists them:
"semlith shows no callees" and "everything this calls is outside the index" are
different facts.
- **`semlith_search` answers where by default; the CLI still answers with text.**
Over MCP and `/api/search`, `format: locate` returns one line per hit — path,
span, enclosing symbol and kind, the lists that found it, provenance for a
graph-reached row, a freshness flag, one line of text — grouped by file and cut
to `max_tokens` (default 1500, floor 200) with `truncated: N of M`. Ask for
`format: "excerpt"` when you actually need the body; that is what 0.14.0
returned. `semlith search` in a terminal is unchanged. Every hit carries
`fresh`, from one `stat` per distinct path against the recorded size and mtime —
conservative on purpose, so a `touch` reads as stale, because a false "check
this" costs a reread and a false "this is current" costs a wrong quotation.
- **Reverse reachability left the free product in 0.13.0.** `semlith impact`,
`semlith_impact` and `/api/impact` are gone, with no shim: an agent carrying
`semlith_impact` in a saved prompt or a committed `.mcp.json` breaks on
upgrade, deliberately, because a stub answering a graph question wrongly is
worse than a tool that is not there. The `edges_dst` index stays — it is what
`graph::neighbours` reads to answer the callers half — and the traversal
returns in 0.14.0 as a paid surface. Do not reintroduce it here.
- **An image is embedded in the same `index_set` pass that embeds text**, inside
the same held lock, on the same changed-file path. There is no image build
step and there must not be one, for the reason there is no `graph build`: the
pass that re-reads a file is the pass that re-embeds it, so no artifact can go
stale. `forget` takes the row and the vector away together.
- **The query for the image half goes through CLIP's text encoder**, never the
store's own model. A granite vector and a CLIP vector are numbers of different
lengths about different things, and comparing them returns whatever the
arithmetic happens to produce — which is why the two CLIP repositories in
`image.rs` are fixed as a pair rather than selectable.
- **The images table and `images/` are additive, so `FORMAT_VERSION` does not
move.** Both are `IF NOT EXISTS`, and a binary that knows nothing about either
reads such a store as the text corpus it already was. Anything that would make
an older binary misread a store is a format bump instead; adding a table it
ignores is not.
- **The ledger records by default, locally, from every surface.** `src/ledger.rs`
is the one writer, and stdio MCP, the daemon's `/mcp` route, the CLI and the
portal all call it — a search from the CLI and the same search from an agent
must be counted the same way, which is why there is one module rather than four
call sites. A row carries the client's own name from the MCP `initialize`
handshake (`claude-code`, `cursor`, `cli`, `portal`) and a session id. Graph
tools record too, against the bytes a grep for that name would have made you
read; a zero-hit retrieval is recorded and credited nothing, because a ledger
that remembers only its successes is a marketing document. Rows are
hash-chained, each carrying the hash of the one before it, so
`store::ledger_break` finds an edited or deleted row — and the chain is
versioned *by the row*: `tool` is NULL on every pre-0.15.0 row and set on every
row since, which is what picks the formula, so a store holding both kinds
verifies end to end. Recording and `semlith ledger` are free on every tier,
permanently; do not put either behind a key.
- **On by default is the whole point, and the principle that makes it all right
is narrower than the one it replaced.** 0.12.0 said a local tool that starts
logging without being told is no different from one that phones home. What that
bought was a ledger nobody switched on: no denominator under the savings figure
and no rows in the audit trail. What holds instead is checkable — the rows never
leave the store they were written into, the daemon says on every start that it
is recording and names the flag that stops it, and erasing every row is one
`DELETE`. `semlith start --ledger` is removed rather than kept as a no-op, so a
script that passes it fails at parse time; `--no-ledger` stops a session and
`SEMLITH_LEDGER=0` stops a machine. Do not reintroduce an off-by-default.
- **Both sides of a ratio are counted on the same instrument.** The store's own
embedding tokenizer counts excerpt and whole-file tokens — the same
`tokenizer.json`, the same cache, the same pinned digest. Four characters per
token survives only as the fallback for a session that never loaded a model, and
`retrievals.tokenizer` records which counted the row, so rows counted two
different ways are never summed.
- **`mcp::tool_names` is the one tool list.** `routes::agents` reads it rather
than repeating it, because two hand-written copies is how a tool ends up served
by the server and invisible in the portal.
- **Two credentials, and their reach is what separates them.** The per-run
session token opens the portal and every `/api/*` route; the agent key opens
`http::MCP_PATH` and nothing else. That asymmetry is the whole design: a key
that sits in a client's config file on disk must not be able to rotate a
token, adopt a store or start an upgrade, so no route outside `/mcp` may ever
learn to accept one. `tests/endpoint.rs` asserts it.
- **The agent key is persisted, and never reminted implicitly.**
`home::agent_key` creates it on first read and returns it forever after;
`home::rotate_agent_key` is a separate function precisely so nothing can
rotate by accident from the state of the disk. A key that changed per run
would make every stanza stale on every restart, and `setup.rs` can only repair
Claude Code's config — everything else would need a human. Rotating the
session token therefore does not disconnect agents, and `/mcp` keeps serving
the previous key for `http::KEY_GRACE` — fifteen minutes, and never past the
daemon's exit — so an open session finishes. It used to be "until this process
exits", which on a daemon somebody leaves running is a second live credential
rather than a grace period.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
- **`add` is the only command that reaches the network besides `upgrade` and
the model download**, it is https-only, it refuses before opening a socket
under `--airgap`, and it writes only inside the store's own `downloads/`.
Every one of those is a tested case in `tests/add.rs`, not a comment. See
[#50](https://github.com/semlith/semlith/issues/50).
- `filter::LANGUAGES` has a `filenames` column as well as `extensions`, because
a Dockerfile and a Makefile have no extension. Both resolve into the same
pattern group, so there is still one code path behind the vector allowlist and
the FTS5 predicate.
Env overrides: `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_FILES`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`, `SEMLITH_LEDGER` (`0`, `off`
or `false` stops the ledger recording on this machine).
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- A count stated in the README is asserted against its source by
`tests/readme.rs`, which also holds the under-500-line ceiling and the grep
for release-specific prose. Change a count in the code and the README fails
the build; add a count to the README and add its row there.
- Public surface changes must land in `docs/compatibility.md`; the client
stanzas in `docs/clients.md` are executed by `tests/clients.rs`, so a renamed
flag breaks there. From 0.18.0 that file also carries the registration facts,
in fence info strings a renderer hides: `sh register` is the command
`semlith setup` runs and must carry the scope that client spells "every
project", `sh unregister` is what clears an existing entry first,
`<lang> config path=…` (with `os=` and double quotes where a path has a space)
is a client's user-level configuration file, and `scope=project` marks a
registration command that would register only the current directory — semlith
does not run those. `clients.rs` also parses that file's structure, not only
its fences: the stdio stanzas are read from under `### Setting it up in your
client` and grouped by `#### Terminal`, `#### Editors` and `#### Desktop
apps`, and the HTTP ones from under `### Connecting over HTTP`. Reword any of
those five headings and the portal's Agents page silently empties — change the
prose around them, not them. The stanzas lived in the README until 0.17.2;
they are in `docs/clients.md` now, and the README only links to it.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 Semlith does listen on a
port — `semlith start` serves the portal, and from 0.13.0 MCP at `/mcp` as well
— under rules that are tested rather than stated, and that are not up for
relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- The portal and every `/api/*` route need the per-run token, in a
`Semlith-Token` header. Without it: 401 and an empty body. `/mcp` takes the
agent key as a bearer instead, and is the only route that does.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, and no update check Semlith makes on its own.
`semlith upgrade` and `semlith upgrade --check` exist from 0.10.0 and reach
GitHub, but only in the second a user asks: there is no startup check, no
timer, and no banner that appears without a click. `semlith add` is the same
shape from 0.11.0 — one request, for one URL, because somebody asked for it,
with no crawling and no re-fetching. The other downloads are all model
weights: the embedding model, once, on first index; `semlith setup`'s
pre-fetch of the same file; and CLIP's two halves, on the first image a store
indexes and never at start — `--airgap` refuses all of it, naming the model it
would have fetched.
Discuss in an issue before building: any other bind address, hosted embedding
APIs, any outbound connection, or a user-editable config file.
`~/.semlith/registry.json` is not one: it is tool-written state, like
`store.db`, `agent.key` and the lock file. Semlith writes it, nothing documents
a way to hand-edit it, and `--store`/`SEMLITH_STORE` remain the only way a user
names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries.
A *view* is not always a page. `semlith languages` is the About page's table and
`semlith_read` is the Search page's second stage — both have a surface a person
can open, which is what the rule is for. What the rule forbids is a capability
with no surface at all, and the exemption list in `tests/portal.rs` is where a
deliberate absence is argued rather than assumed: `start` and `mcp` have no
state of their own to show, and `pattern` takes a tree-sitter query that nobody
writes into a browser box. Adding a row there is allowed; adding one without the
reasoning beside it is not. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar `start` and `mcp`, which have their reasons recorded
there — has no route. The Agents page is the one view whose content is not
computed: it renders the stanzas `src/clients.rs` parses out of
`docs/clients.md`, so a change to that page is usually a change to that file.
`docs/portal.md` documents every page, and a new one belongs there in the same
release.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
matches `Cargo.toml`'s version. GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.
The Linux jobs build with `--no-default-features --features dynamic-ort` and
pack Microsoft's own ONNX Runtime into the archive beside the binary. The runner
is not what decides the floor and never was: the prebuilt ONNX Runtime `ort`
downloads references `__isoc23_strtol@GLIBC_2.38`, so 0.13.0's attempt to build
on `ubuntu-22.04` failed to link, and no older runner or container could have
helped. Not linking it is what helped. The `glibc floor` step runs `objdump -T`
over the packaged binary *and* the library beside it — what a user runs is the
pair — and **fails above `GLIBC_2.35`**. It printed a number for three releases
and the floor rose to 2.39 without failing a build; a gate that does not exit
non-zero is not a gate.
The runner matters as much as the feature, and the two are easy to confuse. Not
linking ONNX Runtime is what makes `ubuntu-22.04` *possible* — 0.13.0 tried it
and the link failed. It is the runner that then sets the floor: a Rust binary
references whatever symbol versions the glibc it was built against provides, so
one built on 24.04 needs 2.39 no matter what else it links. The first tagged run
with `dynamic-ort` demonstrated exactly that — GLIBCXX fell from 3.4.32 to
3.4.22, which is Microsoft's library, and GLIBC stayed at 2.39, which was the
binary. Both Linux jobs build on 22.04 for that reason; moving either forward
moves the floor.
`cargo install`, macOS and Windows link ONNX Runtime as they always have. The
two Linux release binaries are the only build that differs, which is why
`link_runtime` is a no-op everywhere else.
## `unsafe` in this crate
Every site, with the reason it is sound. Adding one means adding a row: a
`SAFETY:` comment on the block is necessary and not sufficient, because what
makes this set reviewable is being able to read the whole of it at once.
| `embed::performance_cores` | `libc::sysctlbyname` | A NUL-terminated C string, and an `i32` whose size the call is told and will not exceed. macOS only. |
| `embed::check_cache_dir` | `libc::getuid` | Reads this process's own uid and cannot fail. |
| `system::sysctl` | `libc::sysctlbyname` | A NUL-terminated C string, and a `u64` whose size the call is told and will not exceed. macOS only. |
| `system::macos_available_mb` | `libc::host_statistics64`, `libc::mach_host_self`, `libc::sysconf`, `mem::zeroed` | The struct is zeroed and its size passed as a count the call will not exceed; the host port is this process's own and is not retained. `mach_host_self` is deprecated in libc 0.2 in favour of the `mach2` crate, which is a dependency this release does not take for three lines — the deprecation is allowed at the call with its reason. macOS only. |
| `system::platform` (windows) | `GlobalMemoryStatusEx` | A zeroed `MEMORYSTATUSEX` whose `dwLength` is set to its own size before the call, which is the documented contract. Windows only. |
| `daemon::owner_only` | `libc::getuid` | As above. |
| `daemon::alive` | `libc::kill(pid, 0)` | Signal 0 delivers nothing; the documented liveness probe. |
| `upgrade::writable` | `libc::access` | Two `CString`s that outlive the call; `access` reads them and returns. |
| `watch::on_signal` | `libc::signal` | Installs and restores a handler that only stores into an `AtomicBool`, which is what an async-signal-safe handler may do. |
| `main::quiet_on_a_closed_pipe` | `libc::signal` | Restores `SIGPIPE` to `SIG_DFL` — the disposition the process would have had if the Rust runtime had not changed it — once, at the top of `main`, before any thread is spawned and before anything is printed. unix only. |
| `main`'s `--airgap` | `std::env::set_var` | Before any thread is spawned. |
| Test helpers in `embed`, `home`, `setup`, `upgrade` | `std::env::set_var` | Each guarded by a mutex that every test reading the variable also takes, and each restores what was there. |
None of them holds a raw pointer across a call, and none is on a path that runs
per file or per request.