# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.89+, edition 2024, 64-bit only (turbovec refuses 32-bit). On Linux
install OpenBLAS first (`libopenblas-dev` / `openblas-devel` / `openblas`);
macOS uses Accelerate. First build compiles bundled SQLite and downloads ONNX
Runtime — minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
```
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched `4 * k` deep, then fused by reciprocal rank
(`1 / (60 + rank)`). Scores are fusion scores, not cosines — meaningful for
ordering within one result set, meaningless across queries. Query text reaches
FTS5 as bare terms, never as typed, because `MATCH` is a query language.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way.
Module responsibilities:
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/chunk.rs` | File bytes → text → chunks (800 chars, 2 overlap lines, 8 MiB cap) |
| `src/formats.rs` | Readers for the nine non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | The README's client stanzas, parsed, so the portal shows the tested text |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the two halves can never disagree about eligibility.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
Env overrides: `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_SECS`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`.
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- Public surface changes must land in `docs/compatibility.md`; README client
stanzas are executed by `tests/clients.rs`, so a renamed flag breaks there.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 semlith does listen on a
port — `semlith start` serves the portal — under rules that are tested rather
than stated, and that are not up for relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- Every request needs the per-run token, as a `SameSite=Strict; HttpOnly`
cookie. Without it: 401 and an empty body.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, no update check. The one download that exists is
the embedding model, once, on first index — and `--airgap` refuses even that.
Discuss in an issue before building: any other bind address, MCP over HTTP as an
endpoint agents connect to directly, hosted embedding APIs, any outbound
connection, or a user-editable config file. `~/.semlith/registry.json` is not
one: it is tool-written state, like `store.db` and the lock file. semlith writes
it, nothing documents a way to hand-edit it, and `--store`/`SEMLITH_STORE`
remain the only way a user names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar `start` and `mcp`, which have their reasons recorded
there — has no route.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
matches `Cargo.toml`'s version. GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.