semlith

A fast local vector store for AI agents.

Point it at your docs, PDFs, notes and code. semlith chunks them, embeds them, and keeps a quantized vector index next to a SQLite database — all on your machine, nothing sent anywhere. An agent then asks a question in plain English and gets back the handful of excerpts that actually matter, with file paths and line numbers, instead of reading whole files and burning tokens on the way.

Think of it as a semantic cache for everything your agent needs to know.

• Local. No API keys, no network at query time. The embedding model is downloaded once and cached.
• Fast. Vector search runs on turbovec (Google Research's TurboQuant, 4 bits per coordinate, SIMD scan).
• Incremental. Re-running index only re-embeds files whose contents changed, and drops files that disappeared.
• Agent-native. Ships an MCP server, so any MCP-capable agent can call it as a tool.

Install

Requires a 64-bit machine. Prebuilt binaries cover Linux (x86_64, aarch64), Apple silicon macOS and Windows x86_64. Intel macOS is not supported: ONNX Runtime no longer publishes x86_64 macOS builds, so the embedding backend cannot link there.

On Linux, install OpenBLAS. turbovec links against a system BLAS; macOS uses Apple's Accelerate framework, which ships with the OS, and Windows falls back to a pure-Rust implementation, so this step is Linux-only.

sudo apt-get install libopenblas-dev     # Debian/Ubuntu
sudo dnf install openblas-devel          # Fedora/RHEL
sudo pacman -S openblas                  # Arch

From a release

Grab the archive for your platform from the latest release and put the binary on your PATH:

VERSION=v0.1.0
TARGET=aarch64-apple-darwin        # or x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu
curl -LO "https://github.com/semlith/semlith/releases/download/$VERSION/semlith-$VERSION-$TARGET.tar.gz"
tar xzf "semlith-$VERSION-$TARGET.tar.gz"
sudo install "semlith-$VERSION-$TARGET/semlith" /usr/local/bin/
semlith --version

Each release ships a SHA256SUMS file; check your download against it before running it. On Windows, unpack the .zip and move semlith.exe somewhere on your PATH.

From source

Needs a Rust toolchain (1.85+):

cargo install --git https://github.com/semlith/semlith

If the build fails with unable to find library -lopenblas, that is the OpenBLAS step above you missed.

Quick start

# Index a directory. Creates ./.semlith and downloads the embedding model
# (~130 MB) the first time.
semlith index ~/notes ~/papers ./src

# Ask it something.
semlith search "how does the retry backoff work"

# What's in there?
semlith stats

Output looks like this:

1. 0.812  src/client.rs:120-158
   /// Retries use full jitter: the delay is uniform in [0, base * 2^attempt],
   /// capped at MAX_BACKOFF. ...

2. 0.744  docs/reliability.md:44-71
   ...

The path:start-end locator is directly usable — hand it to an editor, or read just those lines instead of the whole file.

Commands

Global: --store <DIR> picks the store directory (default .semlith, or the SEMLITH_STORE environment variable).

Using it from an agent

semlith mcp speaks MCP over stdio and exposes two tools, semlith_search and semlith_stats. For Claude Code:

claude mcp add semlith -- /path/to/semlith --store /path/to/.semlith mcp

Or in an MCP client config:

{
  "mcpServers": {
    "semlith": {
      "command": "/path/to/semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

The server loads the embedding model at startup, so the first tool call is as fast as the rest.

What gets indexed

Everything under the given paths, except:

• files ignored by .gitignore (and hidden files)
• binaries — detected by a NUL byte in the first 8 KiB
• files larger than 8 MiB
• anything under a .semlith directory

PDFs are extracted to text automatically. Everything else is read as UTF-8. Files are split into line-aligned chunks of up to 800 characters with two lines of overlap, so a chunk boundary rarely cuts a match in half.

How it works

files ──chunk──> text ──embed──> vectors ──quantize──> index.tv   (turbovec)
                  │
                  └──────────────────────────────────> store.db   (SQLite)

query ──embed──> vector ──search index.tv──> chunk ids ──lookup store.db──> excerpts

Two files live in the store directory:

• index.tv — the turbovec index. Holds only quantized vectors keyed by chunk id. TurboQuant is data-oblivious, so there is no training step and no rebuild as the corpus grows: add vectors, they are searchable.
• store.db — SQLite. Holds the chunk text, its file, and its line span, plus the content hash that makes re-indexing incremental.

A search embeds the query, gets ids from the index, and resolves them with one SQLite lookup each. The index only ever needs to hold vectors, so it stays small enough to sit in memory even for large corpora.

Embeddings default to BGESmallENV15 (384 dimensions), which runs on CPU via ONNX Runtime. Pick another with semlith index --model <NAME>; see semlith models. The model is fixed when the store is created, since vectors from two models are not comparable — to switch, delete the store and re-index.

Performance

Measured on an 8-core Apple Silicon laptop with 8 GB of RAM, indexing 79 Rust source files (1.5 MB, 2375 chunks):

The numbers that matter for an agent are the first and the last. semlith mcp loads the model once at startup, so every tool call costs the warm figure; and keeping a store current is nearly free, because unchanged files are skipped before anything is embedded.

Indexing is the slow half, and that cost is the embedding model, not the index — a transformer on CPU is simply not fast. If you have a large corpus and can trade some retrieval quality for throughput, --model AllMiniLML6V2 is about 1.8x faster (6 transformer layers instead of 12). Quantized variants such as BGESmallENV15Q were not faster in testing on ARM, though they do use less memory.

Contributing

Contributions are welcome. Start with CONTRIBUTING.md — it covers the checks CI runs, how the code is laid out, and what is deliberately out of scope.

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test                  # unit tests, offline
cargo test -- --ignored     # end-to-end round trip; downloads the model

For how the pieces fit together and why, see docs/architecture.md.

Everyone participating is expected to follow the Code of Conduct. Found a security problem? Please read SECURITY.md rather than opening an issue.

Known limits

• Search is dense-vector only. Exact identifier lookup (grep-style) is still better served by grep; hybrid keyword + vector search is not implemented.
• One process at a time per store. Concurrent index runs will fight over index.tv.
• First-time indexing of a large corpus takes a while — see Performance. Subsequent runs only touch what changed.
• Peak memory during indexing is around 1.7 GB. On a memory-tight machine, that is the number to watch.

License

Apache-2.0. See LICENSE and NOTICE.

Note that semlith downloads embedding model weights at runtime; those are covered by their own licenses. The default, BAAI/bge-small-en-v1.5, is MIT.