semlith

A fast local vector store for AI agents — index files once, keep it current as you save, and answer questions across all of it in milliseconds without leaving the machine.

Point it at your docs, PDFs, notes and code. semlith chunks them, embeds them, and keeps a quantized vector index next to a SQLite database — all on your machine, nothing sent anywhere. An agent then asks a question in plain English and gets back the handful of excerpts that actually matter, with file paths and line numbers, instead of reading whole files and burning tokens on the way.

Think of it as a semantic cache for everything your agent needs to know.

Local. No API keys, no network at query time. The embedding model is downloaded once and cached.
Fast. Vector search runs on turbovec (Google Research's TurboQuant, 4 bits per coordinate, SIMD scan).
Hybrid. Every query searches meaning and literal terms, so retry backoff and EMBED_BATCH both land on the right chunk.
Incremental. Re-running index only re-embeds files whose contents changed, and drops files that disappeared.
Agent-native. Ships an MCP server, so any MCP-capable agent can call it as a tool.

Install

Requires a 64-bit machine. Prebuilt binaries cover Linux (x86_64, aarch64), Apple silicon macOS and Windows x86_64. Intel macOS is not supported: ONNX Runtime no longer publishes x86_64 macOS builds, so the embedding backend cannot link there.

On Linux, install OpenBLAS. turbovec links against a system BLAS; macOS uses Apple's Accelerate framework, which ships with the OS, and Windows falls back to a pure-Rust implementation, so this step is Linux-only.

sudo apt-get install libopenblas-dev     # Debian/Ubuntu
sudo dnf install openblas-devel          # Fedora/RHEL
sudo pacman -S openblas                  # Arch

From crates.io

cargo install semlith

From a release

Grab the archive for your platform from the latest release and put the binary on your PATH:

VERSION=v0.1.0
TARGET=aarch64-apple-darwin        # or x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu
curl -LO "https://github.com/semlith/semlith/releases/download/$VERSION/semlith-$VERSION-$TARGET.tar.gz"
tar xzf "semlith-$VERSION-$TARGET.tar.gz"
sudo install "semlith-$VERSION-$TARGET/semlith" /usr/local/bin/
semlith --version

Each release ships a SHA256SUMS file; check your download against it before running it. On Windows, unpack the .zip and move semlith.exe somewhere on your PATH.

From source

Needs a Rust toolchain (1.89+):

cargo install --git https://github.com/semlith/semlith

If the build fails with unable to find library -lopenblas, that is the OpenBLAS step above you missed.

Quick start

# Index a directory. Creates ./.semlith and downloads the embedding model
# (~52 MB) the first time.
semlith index ~/notes ~/papers ./src

# Ask it something.
semlith search "how does the retry backoff work"

# What's in there?
semlith stats

Output looks like this:

1. 0.812  src/client.rs:120-158
   /// Retries use full jitter: the delay is uniform in [0, base * 2^attempt],
   /// capped at MAX_BACKOFF. ...

2. 0.744  docs/reliability.md:44-71
   ...

The path:start-end locator is directly usable — hand it to an editor, or read just those lines instead of the whole file.

Commands

Command	What it does
`semlith index [PATHS...]`	Index files and directories (defaults to `.`). Re-run to update.
`semlith watch [PATHS...]`	Stay running and re-embed files as they are saved. `--debounce MS` to tune.
`semlith search <QUERY>`	Search. `-k N` for result count, `--json` for machine output, `--path`/`--ext`/`--lang` to narrow it.
`semlith stats`	File count, chunk count, model, index size.
`semlith files`	List indexed files.
`semlith forget <PATH>`	Drop one file from the store.
`semlith mcp`	Run as an MCP server over stdio.
`semlith models`	List available embedding models.
`semlith languages`	List the language names `--lang` accepts.

Global: --store <DIR> picks the store directory (default .semlith, or the SEMLITH_STORE environment variable). search, stats, files and mcp read, so the flag is repeatable and they cover every store named; index, watch and forget write, so they take exactly one.

Keeping the store current

semlith index is a snapshot of the moment it ran. Leave semlith watch running instead and a file is re-embedded when you save it:

semlith watch ~/notes ./src

watching notes, src — 412 files, 2183 chunks (0 indexed at startup, 412 unchanged)
  ~ src/client.rs
  1 re-embedded, 0 removed, 6 chunks in 0.3s

It starts with the same incremental pass index does, so anything that changed while it was not running is caught up first, and then it waits on filesystem events — no polling, no rescanning. Measured on a 1000-file corpus: no measurable CPU over a 60-second idle window, and about a second from saving a file to that file's new text being searchable.

New files are indexed, deleted files lose their chunks, and a rename moves the file rather than duplicating it. An editor that saves by writing a temp file and renaming it over the original is handled as an edit. .gitignore, hidden files and the store directory itself are skipped, exactly as index skips them.

Saves are batched: events are collected until things go quiet for --debounce milliseconds (500 by default), so one save — or a formatter rewriting a file three times — costs one re-embed and one index write.

Two things worth knowing:

watch holds the store's write lock for as long as it runs. A store has one writer. While it is running, semlith index against that store exits non-zero and names the watcher's process; searching is unaffected.
An MCP server already running picks the changes up. It reloads the vector index when the watcher replaces it, so an agent that connected an hour ago searches what you saved a second ago without restarting anything.

What it does not cover: changes made while it was not running are caught by its next startup pass, not retroactively; network filesystems do not deliver reliable events and are not supported; and on Linux a very large tree can exhaust the per-user inotify watch limit, which is reported on stderr rather than leaving a watcher that is running but watching nothing.

Searching part of a corpus

One store per repository, and then ask it about one subsystem:

semlith search "how does retry backoff work" --path 'src/http/**'
semlith search "how does retry backoff work" --ext rs --ext toml
semlith search "how does retry backoff work" --lang rust

Each flag is repeatable. Repeats union, kinds intersect — so --ext rs --ext toml means "Rust or TOML", while --path 'src/**' --ext md means "Markdown, under src".

The filter is applied before either half of the search picks its results, not after. Ask for eight hits inside a subdirectory and you get the eight best hits in that subdirectory, not whatever survives filtering the eight best hits in the repository — which, for a subdirectory that is a small part of the corpus, is usually nothing.

Three things about the globs are worth knowing:

A relative pattern matches anywhere in the tree. Paths are stored absolute, so --path 'src/**' is matched as */src/** and finds /home/me/proj/src/lib.rs from any working directory. Start a pattern with / to mean exactly that path and nothing else.
* crosses /. This is SQLite's GLOB, which has no separate **, so --path 'src/*' already reaches the whole subtree. Writing src/** is allowed and means the same thing.
Matching ignores case, so --ext md finds README.MD.

A filter that selects no indexed file says so, rather than reporting that nothing in the corpus matched the query:

$ semlith search "retry backoff" --path 'srv/**'
no files match the filter (store has 6527 chunks)

--lang is a fixed table of extensions, not content sniffing. Run semlith languages to see it; an unrecognised name is an error, not a silent empty result.

Searching several stores at once

Your work is not one repository. Name several stores and one query covers all of them:

semlith search "how is the store lock taken" -s ../api/.semlith -s ../cli/.semlith

1. 0.033  [api] src/lock.rs:14-31
   ...
2. 0.032  [cli] src/main.rs:96-104
   ...
2 hits in 4.1ms across 2 stores: api 1, cli 1

Every hit says which store it came from — in the text output, in --json as a store field, and over MCP. With one store there is nothing to tell apart, so nothing is labelled and the output is exactly what it always was.
A store is named after the directory holding it, so ../api/.semlith is api. Two stores that would end up with the same name get their paths instead.
-k is global. Ten results over three stores is ten results, not thirty.
Filters apply everywhere. --ext rs over two stores searches the Rust in both, and a filter that matches files in only one of them returns that one's hits rather than reporting that nothing matched.
Stores may disagree about the model. Each one embeds the query with its own, which is the only way to query it. Merging happens on rank, not on distance, so nothing compares two models' numbers.

SEMLITH_STORE takes a list, split like PATH, which is what an MCP server definition wants:

export SEMLITH_STORE=~/work/api/.semlith:~/work/cli/.semlith
semlith stats

Two limits worth knowing:

Writes stay single-store. index, watch and forget take exactly one --store; a store has one writer, and four locks with four failure modes is not an improvement. Run a watch per store if you want several kept current — a watched store is searchable inside a multi-store query, freshness included.
A store path that is not already a store is an error. Read commands refuse it instead of creating an empty one, because a mistyped store answers every question with nothing and the other stores hide it.

Adding a store is cheap. Measured on three 300-file stores that share a model, M1: one query embed per search rather than one per store, a median 3.4ms for one store rising to 4.0ms for three, and an MCP server that has answered a query holding 137 MB on one store and 137 MB on three — one loaded model, not three.

Using it from an agent

semlith mcp speaks MCP over stdio and exposes five tools:

Tool	What it does
`semlith_search`	Ranked excerpts with file and line range, with the same `path`/`ext`/`lang`/`store` narrowing as the CLI.
`semlith_stats`	What each open store holds, and the names the other tools accept.
`semlith_files`	Which files are indexed — so "not indexed" and "not discussed" stop looking the same.
`semlith_index`	Index a path into an open store, so a corpus becomes searchable mid-conversation.
`semlith_forget`	Drop one file from a store. The file on disk is untouched.

The two write tools take the store's lock for the call and give it back. A store another process is writing — semlith watch, say — comes back as a tool error naming the holder rather than a corrupted index. When more than one store is open they need a store argument, because a store takes one writer and there is no "the" store to guess at.

Indexing a large tree takes longer than a client will wait for one tool call, so semlith_index works to a time budget, reports what it did not reach, and continues where it left off when it is called again. SEMLITH_MCP_INDEX_BUDGET sets that budget in seconds if your client's tool timeout is unusual.

The server loads the embedding model at startup, so the first tool call is as fast as the rest. It also notices when the store has been rewritten underneath it — run semlith watch alongside and the agent's answers track your working tree, with no restart of the server or the agent.

semlith_search takes the same filters as the CLI, as optional path, ext and lang arrays, so an agent working on one subsystem can ask about that subsystem instead of the whole repository:

{ "query": "how does retry backoff work", "path": ["src/http/**"], "lang": ["rust"] }

One server can hold several stores, which is how an agent working across repositories asks one question instead of one per repository:

{
  "mcpServers": {
    "semlith": {
      "command": "/path/to/semlith",
      "args": ["--store", "/work/api/.semlith", "--store", "/work/cli/.semlith", "mcp"]
    }
  }
}

Every excerpt then names its store, and the tool description lists the stores that are open so the agent knows what it may narrow to. An optional store array restricts one query — useful only when the agent already knows which corpus holds the answer:

{ "query": "how does retry backoff work", "store": ["api"] }

A name that is not open comes back as a tool error listing the ones that are, rather than as an empty result an agent would read as "the corpus does not discuss this". semlith_stats reports one line per store, which is where those names come from.

Which protocol revisions

semlith implements MCP 2026-07-28, 2025-11-25, 2025-06-18 and 2024-11-05, and every one of them has a session in tests/mcp.rs proving it. Clients built on 2026-07-28 — the revision that removed the initialize handshake — get server/discover and per-request versions; every client shipping today gets the handshake it expects. A revision semlith does not implement is answered with one it does, rather than echoed back.

2025-03-26 is deliberately not advertised. It is the one revision that required JSON-RPC batching, and a client pinned to it is answered with 2025-11-25.

Setting it up in your client

Every snippet below runs semlith --store /path/to/.semlith mcp. Repeat --store to open several stores, or set SEMLITH_STORE to a path-separator-delimited list instead. cargo install semlith puts the binary at ~/.cargo/bin/semlith, which is on your PATH in a shell but often not in an editor launched from a desktop icon — those entries use the absolute path.

Claude Code — claude mcp add, or a committed .mcp.json in the project root. The -- matters: without it Claude Code reads --store as one of its own flags.

claude mcp add semlith -- semlith --store /path/to/.semlith mcp

{
  "mcpServers": {
    "semlith": {
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

Claude Desktop — ~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows. Settings → Developer → Edit Config opens it.

{
  "mcpServers": {
    "semlith": {
      "command": "/Users/you/.cargo/bin/semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

OpenAI Codex — ~/.codex/config.toml, shared by the CLI, the IDE extension and the desktop app. TOML, and the table is mcp_servers with an underscore.

[mcp_servers.semlith]
command = "semlith"
args = ["--store", "/path/to/.semlith", "mcp"]

codex mcp add semlith -- semlith --store /path/to/.semlith mcp writes the same table.

GitHub Copilot in VS Code — .vscode/mcp.json for a workspace, or the profile copy that MCP: Open User Configuration opens. The root key is servers, not mcpServers.

{
  "servers": {
    "semlith": {
      "type": "stdio",
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

GitHub Copilot CLI — ~/.copilot/mcp-config.json, or /mcp add in a session. Its name for a stdio server is local, not stdio.

{
  "mcpServers": {
    "semlith": {
      "type": "local",
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"],
      "tools": ["*"]
    }
  }
}

Cursor — ~/.cursor/mcp.json everywhere, or .cursor/mcp.json in one repo.

{
  "mcpServers": {
    "semlith": {
      "type": "stdio",
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

Windsurf — ~/.codeium/windsurf/mcp_config.json.

{
  "mcpServers": {
    "semlith": {
      "command": "/Users/you/.cargo/bin/semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

Zed — the zed: open settings file command. Zed calls MCP servers context servers, and keys them under context_servers.

{
  "context_servers": {
    "semlith": {
      "command": "/Users/you/.cargo/bin/semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

Gemini CLI — ~/.gemini/settings.json, or gemini mcp add semlith semlith --store /path/to/.semlith mcp.

{
  "mcpServers": {
    "semlith": {
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

JetBrains — Junie reads ~/.junie/mcp/mcp.json, or .junie/mcp/mcp.json per project; AI Assistant takes the same JSON under Settings → Tools → AI Assistant → Model Context Protocol.

{
  "mcpServers": {
    "semlith": {
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

Cline — the MCP Servers panel, Configure. Cline's own documentation gives two different paths for the file it writes (~/.cline/mcp.json and ~/.cline/data/settings/cline_mcp_settings.json), so let the panel open it rather than guessing.

{
  "mcpServers": {
    "semlith": {
      "command": "semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"],
      "disabled": false,
      "autoApprove": []
    }
  }
}

Goose — goose configure → Add Extension → Command-line Extension, or ~/.config/goose/config.yaml. Goose calls them extensions and spells the command cmd.

extensions:
  semlith:
    type: stdio
    name: semlith
    enabled: true
    cmd: semlith
    args: ["--store", "/path/to/.semlith", "mcp"]
    timeout: 300

What gets indexed

Everything under the given paths, except:

files ignored by .gitignore (and hidden files)
binaries — detected by a NUL byte in the first 8 KiB
files larger than 8 MiB
anything under a .semlith directory

PDFs are extracted to text automatically. Everything else is read as UTF-8. Files are split into line-aligned chunks of up to 800 characters with two lines of overlap, so a chunk boundary rarely cuts a match in half.

How it works

files ──chunk──> text ──embed──> vectors ──quantize──> index.tv   (turbovec)
                  │
                  └──────────────────────────────────> store.db   (SQLite)

query ──embed──> vector ──search index.tv──> chunk ids ──lookup store.db──> excerpts

Two files live in the store directory:

index.tv — the turbovec index. Holds only quantized vectors keyed by chunk id. TurboQuant is data-oblivious, so there is no training step and no rebuild as the corpus grows: add vectors, they are searchable.
store.db — SQLite. Holds the chunk text, its file, and its line span, plus the content hash that makes re-indexing incremental.

A search embeds the query, gets ids from the index, and resolves them with one SQLite lookup each. The index only ever needs to hold vectors, so it stays small enough to sit in memory even for large corpora.

Embeddings default to granite-embedding-small-english-r2, quantized to int8 (384 dimensions, ~52 MB), which runs on CPU via ONNX Runtime. Pick another with semlith index --model <NAME>; see semlith models. The model is fixed when the store is created, since vectors from two models are not comparable — to switch, delete the store and re-index.

A store built by an earlier version keeps the model it was built with, so upgrading semlith never silently re-embeds a corpus.

Searches consult the vector index and SQLite's FTS5 keyword index together, fusing the two rankings by position. Dense vectors alone are weak at exact identifiers — every constant in a codebase embeds to roughly the same place — and keywords alone cannot answer a question phrased as a sentence.

--path, --ext and --lang resolve to one set of chunk ids by a single SQLite query against the stored file paths. That set becomes an allowlist the vector index scans inside, and the same predicate goes into the FTS5 query, so both halves rank within the subset and fusion never sees a chunk one half was forbidden to return. Nothing is stored for it: the file path has been in the database since 0.1.0, so filtering works on an existing store with no re-indexing.

Performance

Measured on a 4P+4E Apple Silicon laptop with 8 GB of RAM, over three corpora of mixed Rust, Markdown and TypeScript:

store	warm query, p50	p95	indexing	peak RSS
1.2k chunks	2.7 ms	3.5 ms	27.6 chunks/sec	600 MB
9.9k chunks	5.4 ms	11.0 ms	24.3 chunks/sec	637 MB
105k chunks	22.7 ms	67.4 ms	23.5 chunks/sec	595 MB

Re-indexing 8334 unchanged files takes 1.7 seconds, because content hashes match and the model is never loaded. Cold CLI start on the 105k store is about 430 ms, most of it loading the model and the index.

Two things are worth reading off that table. Peak memory does not grow with the corpus — 105k chunks is 85 times the work of 1.2k for slightly less memory, so the number to plan for is roughly 600 MB whatever you point it at. Query latency does grow, because the index scan is linear: budget a few milliseconds for a repository and a few tens for a very large corpus.

The numbers that matter for an agent are the query row and the re-index figure. semlith mcp loads the model once at startup, so every tool call costs the warm figure; and keeping a store current is nearly free.

Indexing is the slow half, and that cost is the embedding model, not the index — a transformer on CPU is simply not fast. If you have a large corpus and can trade some retrieval quality for throughput, --model AllMiniLML6V2 is about 1.8x faster (6 transformer layers instead of 12).

Quantization is worth testing rather than assuming. The int8 build of the default model is both smaller and faster than its fp32 build on ARM, while BGE's quantized variants measured no faster than fp32 on the same machine — whether int8 wins depends on the model's graph, not on the architecture alone.

Thread count is chosen rather than left to ONNX Runtime. Its threads synchronise at every operator, so on a CPU with performance and efficiency cores a thread on a slow core paces the whole batch; semlith uses the performance-core count on Apple silicon and the full count elsewhere. Override with SEMLITH_EMBED_THREADS if your machine disagrees.

Compatibility

docs/compatibility.md says which parts of semlith are a contract — the CLI commands and flags, the MCP tool names and schemas, the protocol revisions, the store on disk, the Rust API — and which parts are free to change under you, such as ranking scores and the text printed for a person to read. It also says plainly what a 0.x version number does and does not promise.

Stores carry a format_version from 0.6.0 on. A store written before that is read as format 1 and never rewritten, and a binary that meets a store from a newer format refuses it naming both numbers rather than misreading it.

Contributing

Contributions are welcome. Start with CONTRIBUTING.md — it covers the checks CI runs, how the code is laid out, and what is deliberately out of scope.

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test                  # unit tests, offline
cargo test -- --ignored     # end-to-end round trip; downloads the model

For how the pieces fit together and why, see docs/architecture.md.

Everyone participating is expected to follow the Code of Conduct. Found a security problem? Please read SECURITY.md rather than opening an issue.

Known limits

One writer per store. A second index run against a store already being indexed exits with an error naming the process that holds it, rather than waiting. Searching during an index run is fine. semlith watch is a writer for as long as it runs, so it blocks index on that store the whole time.
watch misses nothing while it is running, but it is not a journal: changes made while it was down are picked up by its next startup pass, not reconstructed. It needs local filesystem events, so network mounts are out. On Windows, Ctrl-C terminates it immediately rather than at a batch boundary, which can leave a file to be re-indexed on the next run.
First-time indexing is bound by transformer speed on CPU, at roughly 23 chunks/sec — a 100k-chunk corpus takes over an hour. Subsequent runs only touch what changed, and cost seconds.
Query latency grows with corpus size, from under 3 ms at a thousand chunks to low tens of milliseconds at a hundred thousand. The index scan is linear.
The default model is English-only. semlith models lists multilingual alternatives, which must be chosen when the store is created.
Search filters are SQLite GLOB patterns, so * crosses / and there is no distinct **, no regex, and no way to express "not this path". --lang maps a fixed table of extensions and never reads file contents, so a Perl script named build is not Perl as far as semlith is concerned.
Results are not reranked. A cross-encoder over the top results would improve ordering, at a cost per query that a local tool should not pay by default.
Multi-store search is a merge, not a joint ranking. Each store ranks its own chunks and the merge compares fused rank scores across them, so a store with nothing to say still offers its best hit. Ties go to the closer vector, which is measured against the store's own model — across two models that comparison is approximate, and it only ever decides between hits the rank evidence has already called equal.
Stores are named by path; there is no registry of named stores, and no discovery. A store is searched because it was named, and its label comes from the directory holding it.

License

Apache-2.0. See LICENSE and NOTICE.

Note that semlith downloads embedding model weights at runtime; those are covered by their own licenses. The default, ibm-granite/granite-embedding-small-english-r2, is Apache-2.0.

semlith 0.6.0