# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.89+, edition 2024, 64-bit only (turbovec refuses 32-bit). Nothing has to
be installed first: turbovec 1.0.0 dropped BLAS, so there is no OpenBLAS step on
Linux any more, and the TLS stack is rustls rather than the system OpenSSL. The
packaged Linux binary must keep needing nothing but glibc and libstdc++ —
`release.yml` reads its `NEEDED` entries and fails on `libssl`, `libcrypto` or
`libopenblas`. First build compiles bundled SQLite and downloads ONNX Runtime —
minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
```
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched `4 * k` deep, then fused by reciprocal rank
(`1 / (60 + rank)`). Scores are fusion scores, not cosines — meaningful for
ordering within one result set, meaningless across queries. Query text reaches
FTS5 as bare terms, never as typed, because `MATCH` is a query language.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way.
Module responsibilities:
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/chunk.rs` | File bytes → text → chunks (800 chars, 2 overlap lines, 8 MiB cap) |
| `src/formats.rs` | Readers for the thirteen non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/graph.rs` | tree-sitter extraction, and the bounded traversals over the edges |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/add.rs` | `semlith add`: the one outbound fetch, its refusals, and where a fetched file lands |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | The README's client stanzas, parsed, so the portal shows the tested text |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the three halves can never disagree about eligibility — graph
expansion resolves its neighbours through `store::symbols_by_names`, which
takes the same predicate, so a chunk outside the filter cannot arrive through
the graph either.
- **The graph is extracted inside `index_set`, never separately.** It is written
where the file id and the new chunk ids are both already in hand, after
`delete_file` has taken the old rows away. That is the whole freshness claim:
the pass that re-embeds a file is the pass that re-extracts it, so there is no
artifact that can be stale. A `semlith graph build` command would be the one
change that breaks the release's central property — do not add one.
- **`edges.dst` is a symbol *name*, not an id.** `edges.src` is an id and
cascades with its file; the target is resolved through `symbols(name)` at
query time. With an id there, re-indexing `b.rs` would reissue its symbol ids
and silently orphan every edge pointing into it from `a.rs` — the graph would
rot on exactly the operation this design exists to make safe. It also lets an
edge to something unindexed (a standard-library call) still be recorded.
- **`extracted` versus `inferred` is load-bearing.** A call whose name the file
imports is extracted; a bare name match is inferred. Two functions called `new`
in different modules is the normal case, so nothing that renders an edge may
present the second as the first.
- **Traversal is bounded and reads one hop at a time.** No `petgraph`, no
in-memory whole-graph structure: `graph::impact` and `graph::shortest_path`
walk the indexed `edges(src)`/`edges(dst)` columns under a depth limit and
`graph::MAX_NODES`, which is what keeps peak RSS flat as the corpus grows.
`impact` and `path` follow `DEPENDENCY_KINDS` only — the structural edges are
true and useless for a blast radius, since every symbol is one hop from the
file that defines it.
- **The ledger is off unless asked for**, and its rows are hash-chained: each
carries the hash of the one before it, so `store::ledger_break` finds an edited
or deleted row. Recording and `semlith ledger` are free on every tier,
permanently; do not put either behind a key.
- **`mcp::tool_names` is the one tool list.** `routes::agents` reads it rather
than repeating it, because two hand-written copies is how a tool ends up served
by the server and invisible in the portal.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
- **`add` is the only command that reaches the network besides `upgrade` and
the model download**, it is https-only, it refuses before opening a socket
under `--airgap`, and it writes only inside the store's own `downloads/`.
Every one of those is a tested case in `tests/add.rs`, not a comment. See
[#50](https://github.com/semlith/semlith/issues/50).
- `filter::LANGUAGES` has a `filenames` column as well as `extensions`, because
a Dockerfile and a Makefile have no extension. Both resolve into the same
pattern group, so there is still one code path behind the vector allowlist and
the FTS5 predicate.
Env overrides: `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_SECS`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`.
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- Public surface changes must land in `docs/compatibility.md`; README client
stanzas are executed by `tests/clients.rs`, so a renamed flag breaks there.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 semlith does listen on a
port — `semlith start` serves the portal — under rules that are tested rather
than stated, and that are not up for relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- Every request needs the per-run token, as a `SameSite=Strict; HttpOnly`
cookie. Without it: 401 and an empty body.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, and no update check semlith makes on its own.
`semlith upgrade` and `semlith upgrade --check` exist from 0.10.0 and reach
GitHub, but only in the second a user asks: there is no startup check, no
timer, and no banner that appears without a click. `semlith add` is the same
shape from 0.11.0 — one request, for one URL, because somebody asked for it,
with no crawling and no re-fetching. The other two downloads are the embedding
model, once, on first index, and `semlith setup`'s pre-fetch of the same file
— `--airgap` refuses all of it.
Discuss in an issue before building: any other bind address, MCP over HTTP as an
endpoint agents connect to directly, hosted embedding APIs, any outbound
connection, or a user-editable config file. `~/.semlith/registry.json` is not
one: it is tool-written state, like `store.db` and the lock file. semlith writes
it, nothing documents a way to hand-edit it, and `--store`/`SEMLITH_STORE`
remain the only way a user names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar `start` and `mcp`, which have their reasons recorded
there — has no route.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
matches `Cargo.toml`'s version. GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.