# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.89+, edition 2024, 64-bit only (turbovec refuses 32-bit). Nothing has to
be installed first: turbovec 1.0.0 dropped BLAS, so there is no OpenBLAS step on
Linux any more, and the TLS stack is rustls rather than the system OpenSSL. The
packaged Linux binary must keep needing nothing but glibc and libstdc++ —
`release.yml` reads its `NEEDED` entries and fails on `libssl`, `libcrypto` or
`libopenblas`. First build compiles bundled SQLite and downloads ONNX Runtime —
minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo test --test endpoint # the HTTP MCP endpoint; no model, so it is in that set
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --test image -- --ignored # images; downloads CLIP as well
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
```
`tests/endpoint.rs` is the exception in the other direction: it drives the
daemon's `/mcp` route and its credential, neither of which needs an embedding
model, so it runs in the default set and a break in the agent key's reach fails
CI rather than an `--ignored` run somebody remembered to do.
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
A store that has indexed an image holds a third, `images/`, in the same shard
layout at CLIP's 512 dimensions. It is opened lazily, so a store that never
meets an image never grows the directory.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched `4 * k` deep, then fused by reciprocal rank
(`1 / (60 + rank)`). Scores are fusion scores, not cosines — meaningful for
ordering within one result set, meaningless across queries. Query text reaches
FTS5 as bare terms, never as typed, because `MATCH` is a query language.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way. A store that holds
images loads CLIP's two encoders on top of that, on the first image it indexes
or is searched for — measured per store rather than folded into the number
above, which is about the text path.
Module responsibilities:
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/chunk.rs` | File bytes → text → chunks (800 chars, 2 overlap lines, 8 MiB cap) |
| `src/formats.rs` | Readers for the thirteen non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/graph.rs` | tree-sitter extraction, and the bounded traversals over the edges |
| `src/image.rs` | Image support: the five extensions, and the CLIP pair that makes a picture comparable with a sentence |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/add.rs` | `semlith add`: the one outbound fetch, its refusals, and where a fetched file lands |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | The README's client stanzas, parsed, so the portal shows the tested text |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the three halves can never disagree about eligibility — graph
expansion resolves its neighbours through `store::symbols_by_names`, which
takes the same predicate, so a chunk outside the filter cannot arrive through
the graph either.
- **The graph is extracted inside `index_set`, never separately.** It is written
where the file id and the new chunk ids are both already in hand, after
`delete_file` has taken the old rows away. That is the whole freshness claim:
the pass that re-embeds a file is the pass that re-extracts it, so there is no
artifact that can be stale. A `semlith graph build` command would be the one
change that breaks the release's central property — do not add one.
- **`edges.dst` is a symbol *name*, not an id.** `edges.src` is an id and
cascades with its file; the target is resolved through `symbols(name)` at
query time. With an id there, re-indexing `b.rs` would reissue its symbol ids
and silently orphan every edge pointing into it from `a.rs` — the graph would
rot on exactly the operation this design exists to make safe. It also lets an
edge to something unindexed (a standard-library call) still be recorded.
- **`extracted` versus `inferred` is load-bearing.** A call whose name the file
imports is extracted; a bare name match is inferred. Two functions called `new`
in different modules is the normal case, so nothing that renders an edge may
present the second as the first.
- **Traversal is bounded and reads one hop at a time.** No `petgraph`, no
in-memory whole-graph structure: `graph::neighbours` and
`graph::shortest_path` walk the indexed `edges(src)`/`edges(dst)` columns
under a depth limit and `graph::MAX_NODES`, which is what keeps peak RSS flat
as the corpus grows. `shortest_path` follows `DEPENDENCY_KINDS` only — the
structural edges are true and useless for reachability, since every symbol is
one hop from the file that defines it.
- **Reverse reachability left the free product in 0.13.0.** `semlith impact`,
`semlith_impact` and `/api/impact` are gone, with no shim: an agent carrying
`semlith_impact` in a saved prompt or a committed `.mcp.json` breaks on
upgrade, deliberately, because a stub answering a graph question wrongly is
worse than a tool that is not there. The `edges_dst` index stays — it is what
`graph::neighbours` reads to answer the callers half — and the traversal
returns in 0.14.0 as a paid surface. Do not reintroduce it here.
- **An image is embedded in the same `index_set` pass that embeds text**, inside
the same held lock, on the same changed-file path. There is no image build
step and there must not be one, for the reason there is no `graph build`: the
pass that re-reads a file is the pass that re-embeds it, so no artifact can go
stale. `forget` takes the row and the vector away together.
- **The query for the image half goes through CLIP's text encoder**, never the
store's own model. A granite vector and a CLIP vector are numbers of different
lengths about different things, and comparing them returns whatever the
arithmetic happens to produce — which is why the two CLIP repositories in
`image.rs` are fixed as a pair rather than selectable.
- **The images table and `images/` are additive, so `FORMAT_VERSION` does not
move.** Both are `IF NOT EXISTS`, and a binary that knows nothing about either
reads such a store as the text corpus it already was. Anything that would make
an older binary misread a store is a format bump instead; adding a table it
ignores is not.
- **The ledger is off unless asked for**, and its rows are hash-chained: each
carries the hash of the one before it, so `store::ledger_break` finds an edited
or deleted row. Recording and `semlith ledger` are free on every tier,
permanently; do not put either behind a key.
- **`mcp::tool_names` is the one tool list.** `routes::agents` reads it rather
than repeating it, because two hand-written copies is how a tool ends up served
by the server and invisible in the portal.
- **Two credentials, and their reach is what separates them.** The per-run
session token opens the portal and every `/api/*` route; the agent key opens
`http::MCP_PATH` and nothing else. That asymmetry is the whole design: a key
that sits in a client's config file on disk must not be able to rotate a
token, adopt a store or start an upgrade, so no route outside `/mcp` may ever
learn to accept one. `tests/endpoint.rs` asserts it.
- **The agent key is persisted, and never reminted implicitly.**
`home::agent_key` creates it on first read and returns it forever after;
`home::rotate_agent_key` is a separate function precisely so nothing can
rotate by accident from the state of the disk. A key that changed per run
would make every stanza stale on every restart, and `setup.rs` can only repair
Claude Code's config — everything else would need a human. Rotating the
session token therefore does not disconnect agents, and `/mcp` keeps serving
the previous key until the daemon that held it exits, so an open session
finishes.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
- **`add` is the only command that reaches the network besides `upgrade` and
the model download**, it is https-only, it refuses before opening a socket
under `--airgap`, and it writes only inside the store's own `downloads/`.
Every one of those is a tested case in `tests/add.rs`, not a comment. See
[#50](https://github.com/semlith/semlith/issues/50).
- `filter::LANGUAGES` has a `filenames` column as well as `extensions`, because
a Dockerfile and a Makefile have no extension. Both resolve into the same
pattern group, so there is still one code path behind the vector allowlist and
the FTS5 predicate.
Env overrides: `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_SECS`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`.
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- Public surface changes must land in `docs/compatibility.md`; README client
stanzas are executed by `tests/clients.rs`, so a renamed flag breaks there.
`clients.rs` also parses the README's structure, not only its fences: the
stdio stanzas are read from under `### Setting it up in your client` and
grouped by `#### Terminal`, `#### Editors` and `#### Desktop apps`, and the
HTTP ones from under `### Connecting over HTTP`. Reword any of those five
headings and the portal's Agents page silently empties — change the prose
around them, not them.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 Semlith does listen on a
port — `semlith start` serves the portal, and from 0.13.0 MCP at `/mcp` as well
— under rules that are tested rather than stated, and that are not up for
relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- The portal and every `/api/*` route need the per-run token, as a
`SameSite=Strict; HttpOnly` cookie. Without it: 401 and an empty body. `/mcp`
takes the agent key as a bearer instead, and is the only route that does.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, and no update check Semlith makes on its own.
`semlith upgrade` and `semlith upgrade --check` exist from 0.10.0 and reach
GitHub, but only in the second a user asks: there is no startup check, no
timer, and no banner that appears without a click. `semlith add` is the same
shape from 0.11.0 — one request, for one URL, because somebody asked for it,
with no crawling and no re-fetching. The other downloads are all model
weights: the embedding model, once, on first index; `semlith setup`'s
pre-fetch of the same file; and CLIP's two halves, on the first image a store
indexes and never at start — `--airgap` refuses all of it, naming the model it
would have fetched.
Discuss in an issue before building: any other bind address, hosted embedding
APIs, any outbound connection, or a user-editable config file.
`~/.semlith/registry.json` is not one: it is tool-written state, like
`store.db`, `agent.key` and the lock file. Semlith writes it, nothing documents
a way to hand-edit it, and `--store`/`SEMLITH_STORE` remain the only way a user
names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar `start` and `mcp`, which have their reasons recorded
there — has no route.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
matches `Cargo.toml`'s version. GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.
The Linux jobs pin `ubuntu-22.04`, not `-latest`. A binary built on 24.04
carries a glibc 2.39 floor and is dead on Debian 12 and Ubuntu 22.04 LTS; 22.04
is glibc 2.35, which is what the `glibc floor` step asserts. That assertion
reads the *versions* of the symbols, because the floor rose across three
releases without failing a single build when nothing checked them. Moving those
runners forward means moving the floor, so do not.