# AGENTS.md
Guidance for coding agents working in this repository. This is the single source
of truth; `CLAUDE.md` points here.
## Build and test
Rust 1.89+, edition 2024, 64-bit only (turbovec refuses 32-bit). Nothing has to
be installed first: turbovec 1.0.0 dropped BLAS, so there is no OpenBLAS step on
Linux any more, and the TLS stack is rustls rather than the system OpenSSL. The
packaged Linux binary must keep needing nothing but glibc and libstdc++ —
`release.yml` reads its `NEEDED` entries and fails on `libssl`, `libcrypto` or
`libopenblas`. First build compiles bundled SQLite and downloads ONNX Runtime —
minutes.
```sh
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test # offline default set — this is what CI runs
cargo build --release
```
Most integration tests are `#[ignore]`d because they download the ~52 MB
embedding model on first run:
```sh
cargo test -- --ignored # everything, incl. round trip
cargo test --test mcp -- --ignored # one file
cargo test --test mcp -- --ignored some_test_name # one test
cargo test --release --test measure -- --ignored --nocapture # perf claims; needs --release
cargo test --release --test shards -- --ignored --nocapture
```
`tests/measure.rs` and `tests/shards.rs` are `#![cfg(unix)]` and assert real
thresholds — run them in `--release` or the numbers fail honestly.
Run `cargo test -- --ignored` (the `tests/roundtrip.rs` cycle) after touching
`lib.rs`, `store.rs`, or `chunk.rs`. It is the only test that catches chunk ids
in the vector index drifting out of sync with their rows in SQLite.
Document fixtures under `tests/fixtures/` are generated by real libraries via
`tests/fixtures/generate.py`, not hand-written XML. Regenerate rather than edit.
## Architecture
A store directory holds two pieces of state that must agree:
- **`index/`** — turbovec (TurboQuant) shards of 65536 quantized vectors, each
file named for the first chunk id it holds, zero-padded so sorted-by-name is
sorted-by-id. **The directory listing *is* the manifest** — nothing else
records shard boundaries, so nothing else can disagree with it. Format 1
stores (pre-0.7.0) keep a single `index.tv` and are never migrated.
- **`store.db`** — SQLite: chunk text, file path, line span, content hash.
`chunks.id` is `AUTOINCREMENT` specifically so SQLite cannot reissue a deleted
row's id — a reused id would fall inside two shards' ranges at once. Do not
change that.
Search is hybrid: the query embedding goes to the vector index and the bare
terms go to FTS5, each searched `4 * k` deep, then fused by reciprocal rank
(`1 / (60 + rank)`). Scores are fusion scores, not cosines — meaningful for
ordering within one result set, meaningless across queries. Query text reaches
FTS5 as bare terms, never as typed, because `MATCH` is a query language.
`index.rs` reads nothing from disk until a caller asks something only vectors
can answer, so `stats`, `files`, and an idle MCP server cost no vectors. Peak
RSS is ~600 MB regardless of corpus size; keep it that way.
Module responsibilities:
| `src/lib.rs` | `Semlith`: open, index, search, persist |
| `src/store.rs` | Every SQL statement. Nothing else touches the database |
| `src/index.rs` | Vector side: `Single` vs `Sharded` layouts, memory budget |
| `src/chunk.rs` | File bytes → text → chunks (800 chars, 2 overlap lines, 8 MiB cap) |
| `src/formats.rs` | Readers for the nine non-plain-text formats. Private on purpose |
| `src/embed.rs` | Model selection/loading, incl. the hand-assembled granite default |
| `src/filter.rs` | `--path`/`--ext`/`--lang` → GLOB patterns → one chunk id set |
| `src/fleet.rs` | Several stores, one query, merged ranking |
| `src/lock.rs` | One writer per store, OS advisory lock (not file existence) |
| `src/watch.rs` | Event source in front of the same indexer `index` runs |
| `src/mcp.rs` | Hand-rolled stdio JSON-RPC MCP server |
| `src/home.rs` | The store home, `registry.json`, and which store a command resolves to |
| `src/http.rs` | Hand-rolled synchronous HTTP/1.1: the token, `Host` and CSP rules live here |
| `src/daemon.rs` | `semlith start`: the locks, the watcher threads, the write queue |
| `src/routes.rs` | The daemon's routes, as adapters over `Fleet` and the queue |
| `src/portal/` | The page itself, `include_bytes!`d — HTML, CSS, JS, IBM Plex |
| `src/proxy.rs` | `semlith mcp` forwarding to a running daemon |
| `src/clients.rs` | The README's client stanzas, parsed, so the portal shows the tested text |
| `src/main.rs` | Clap parsing and human output formatting |
### Invariants worth knowing before editing
- **The model is fixed when a store is created.** Vectors from two models are
not comparable; switching means delete and re-index. Changing
`embed::GRANITE_NAME` orphans every store that recorded the old string.
- **One writer per store.** `watch` holds the lock for its whole life, so a
concurrent `index` on that store is refused by design. `semlith start` is the
same rule taken further: it is the writer for every store it opened, and the
portal and a forwarded `semlith_index` reach it through the store's queue —
`Semlith::index_within_held` and `forget_held` exist for exactly that caller.
- **`semlith mcp` forwards when a daemon is running**, found through the
discovery file beside the lock, and the daemon answers with the same
`mcp::answer` the stdio server runs. Every failure to reach one — no file, an
unreadable file, a dead port — falls back to opening the store directly.
- **`mcp.rs`: stdout is protocol, stderr is diagnostics.** Never print to
stdout outside the JSON-RPC framing, and keep `quiet` set on the Fleet there.
- The MCP server speaks two protocol eras at once, decided per message by the
presence of `_meta.io.modelcontextprotocol/protocolVersion`. Every entry in
`mcp::SUPPORTED` has a session in `tests/mcp.rs` proving it — adding a
revision means adding that session.
- `filter.rs` derives both the vector allowlist and the FTS5 predicate from one
id set, so the two halves can never disagree about eligibility.
- Extraction dispatches on extension *before* looking at bytes — `.docx` and
friends are ZIP archives and the binary check would reject them all.
Env overrides: `SEMLITH_STORE` (PATH-style separated), `SEMLITH_HOME`,
`SEMLITH_PORT`, `SEMLITH_AIRGAP`, `SEMLITH_EMBED_THREADS`,
`SEMLITH_INDEX_MEMORY`, `SEMLITH_SHARD_VECTORS`, `SEMLITH_CHECKPOINT_SECS`,
`SEMLITH_MODEL_CACHE`, `SEMLITH_MCP_INDEX_BUDGET`.
Deeper rationale lives in `docs/architecture.md`; what is and isn't a stability
contract lives in `docs/compatibility.md`.
## Conventions
- Comments explain *why*, not *what*. The ones worth writing save the next
person an hour — why a batch size is 32, why the hash is written after the
index.
- Deliberate shortcuts get a `ponytail:` comment naming the ceiling and the
upgrade path.
- Errors a user can act on: `bail!("store was built with X, not Y")`, not
`bail!("model mismatch")`.
- Conventional Commits. One logical change per PR. Update `CHANGELOG.md` under
`## [Unreleased]`.
- Behaviour changes to indexing or search need measured numbers, not adjectives.
The 100k benchmark store is a fixture kept at `~/.cache/semlith/bench/100k-store`
— build it once (recipe in `CONTRIBUTING.md`), not per release.
- Public surface changes must land in `docs/compatibility.md`; README client
stanzas are executed by `tests/clients.rs`, so a renamed flag breaks there.
## Deliberately out of scope
**Local means local, and it is checkable.** Since 0.9.0 semlith does listen on a
port — `semlith start` serves the portal — under rules that are tested rather
than stated, and that are not up for relaxation without an issue like
[#41](https://github.com/semlith/semlith/issues/41):
- `127.0.0.1` is the only bind address, and there is no flag to change it.
- Every request needs the per-run token, as a `SameSite=Strict; HttpOnly`
cookie. Without it: 401 and an empty body.
- The `Host` header must be `localhost`, `127.0.0.1` or `::1`. Otherwise: 400.
- Every response carries a `Content-Security-Policy` allowing only `'self'`, and
no CORS header is emitted anywhere.
- Every byte the portal loads is `include_bytes!`d into the binary. No CDN, no
build step, no npm. The page loads with the cable unplugged.
- No telemetry, no analytics, and no update check semlith makes on its own.
`semlith upgrade` and `semlith upgrade --check` exist from 0.10.0 and reach
GitHub, but only in the second a user asks: there is no startup check, no
timer, and no banner that appears without a click. The other two downloads are
the embedding model, once, on first index, and `semlith setup`'s pre-fetch of
the same file — `--airgap` refuses all of it.
Discuss in an issue before building: any other bind address, MCP over HTTP as an
endpoint agents connect to directly, hosted embedding APIs, any outbound
connection, or a user-editable config file. `~/.semlith/registry.json` is not
one: it is tool-written state, like `store.db` and the lock file. semlith writes
it, nothing documents a way to hand-edit it, and `--store`/`SEMLITH_STORE`
remain the only way a user names a store.
## Portal parity
**A CLI command or an MCP tool is not done until its portal view exists.** Every
release that adds one adds the view in the same release; parity debt is not a
thing this repository carries. `tests/portal.rs` is the gate: it reads the
subcommand list out of `--help` and the tool list off a running daemon, and
fails if any of them — bar `start` and `mcp`, which have their reasons recorded
there — has no route.
## Release
Tags `v*` trigger `.github/workflows/release.yml`, which first verifies the tag
matches `Cargo.toml`'s version. GitHub Actions must stay SHA-pinned with a
trailing `# vN` comment for Dependabot. No Intel macOS target — ONNX Runtime
stopped publishing `osx-x86_64`.