semlith

A fast local vector store for AI agents.

Point it at your docs, PDFs, notes and code. semlith chunks them, embeds them, and keeps a quantized vector index next to a SQLite database — all on your machine, nothing sent anywhere. An agent then asks a question in plain English and gets back the handful of excerpts that actually matter, with file paths and line numbers, instead of reading whole files and burning tokens on the way.

Think of it as a semantic cache for everything your agent needs to know.

• Local. No API keys, no network at query time. The embedding model is downloaded once and cached.
• Fast. Vector search runs on turbovec (Google Research's TurboQuant, 4 bits per coordinate, SIMD scan).
• Hybrid. Every query searches meaning and literal terms, so retry backoff and EMBED_BATCH both land on the right chunk.
• Incremental. Re-running index only re-embeds files whose contents changed, and drops files that disappeared.
• Agent-native. Ships an MCP server, so any MCP-capable agent can call it as a tool.

Install

Requires a 64-bit machine. Prebuilt binaries cover Linux (x86_64, aarch64), Apple silicon macOS and Windows x86_64. Intel macOS is not supported: ONNX Runtime no longer publishes x86_64 macOS builds, so the embedding backend cannot link there.

On Linux, install OpenBLAS. turbovec links against a system BLAS; macOS uses Apple's Accelerate framework, which ships with the OS, and Windows falls back to a pure-Rust implementation, so this step is Linux-only.

sudo apt-get install libopenblas-dev     # Debian/Ubuntu
sudo dnf install openblas-devel          # Fedora/RHEL
sudo pacman -S openblas                  # Arch

From crates.io

cargo install semlith

From a release

Grab the archive for your platform from the latest release and put the binary on your PATH:

VERSION=v0.1.0
TARGET=aarch64-apple-darwin        # or x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu
curl -LO "https://github.com/semlith/semlith/releases/download/$VERSION/semlith-$VERSION-$TARGET.tar.gz"
tar xzf "semlith-$VERSION-$TARGET.tar.gz"
sudo install "semlith-$VERSION-$TARGET/semlith" /usr/local/bin/
semlith --version

Each release ships a SHA256SUMS file; check your download against it before running it. On Windows, unpack the .zip and move semlith.exe somewhere on your PATH.

From source

Needs a Rust toolchain (1.89+):

cargo install --git https://github.com/semlith/semlith

If the build fails with unable to find library -lopenblas, that is the OpenBLAS step above you missed.

Quick start

# Index a directory. Creates ./.semlith and downloads the embedding model
# (~52 MB) the first time.
semlith index ~/notes ~/papers ./src

# Ask it something.
semlith search "how does the retry backoff work"

# What's in there?
semlith stats

Output looks like this:

1. 0.812  src/client.rs:120-158
   /// Retries use full jitter: the delay is uniform in [0, base * 2^attempt],
   /// capped at MAX_BACKOFF. ...

2. 0.744  docs/reliability.md:44-71
   ...

The path:start-end locator is directly usable — hand it to an editor, or read just those lines instead of the whole file.

Commands

Global: --store <DIR> picks the store directory (default .semlith, or the SEMLITH_STORE environment variable).

Using it from an agent

semlith mcp speaks MCP over stdio and exposes two tools, semlith_search and semlith_stats. For Claude Code:

claude mcp add semlith -- /path/to/semlith --store /path/to/.semlith mcp

Or in an MCP client config:

{
  "mcpServers": {
    "semlith": {
      "command": "/path/to/semlith",
      "args": ["--store", "/path/to/.semlith", "mcp"]
    }
  }
}

The server loads the embedding model at startup, so the first tool call is as fast as the rest.

What gets indexed

Everything under the given paths, except:

• files ignored by .gitignore (and hidden files)
• binaries — detected by a NUL byte in the first 8 KiB
• files larger than 8 MiB
• anything under a .semlith directory

PDFs are extracted to text automatically. Everything else is read as UTF-8. Files are split into line-aligned chunks of up to 800 characters with two lines of overlap, so a chunk boundary rarely cuts a match in half.

How it works

files ──chunk──> text ──embed──> vectors ──quantize──> index.tv   (turbovec)
                  │
                  └──────────────────────────────────> store.db   (SQLite)

query ──embed──> vector ──search index.tv──> chunk ids ──lookup store.db──> excerpts

Two files live in the store directory:

• index.tv — the turbovec index. Holds only quantized vectors keyed by chunk id. TurboQuant is data-oblivious, so there is no training step and no rebuild as the corpus grows: add vectors, they are searchable.
• store.db — SQLite. Holds the chunk text, its file, and its line span, plus the content hash that makes re-indexing incremental.

A search embeds the query, gets ids from the index, and resolves them with one SQLite lookup each. The index only ever needs to hold vectors, so it stays small enough to sit in memory even for large corpora.

Embeddings default to granite-embedding-small-english-r2, quantized to int8 (384 dimensions, ~52 MB), which runs on CPU via ONNX Runtime. Pick another with semlith index --model <NAME>; see semlith models. The model is fixed when the store is created, since vectors from two models are not comparable — to switch, delete the store and re-index.

A store built by an earlier version keeps the model it was built with, so upgrading semlith never silently re-embeds a corpus.

Searches consult the vector index and SQLite's FTS5 keyword index together, fusing the two rankings by position. Dense vectors alone are weak at exact identifiers — every constant in a codebase embeds to roughly the same place — and keywords alone cannot answer a question phrased as a sentence.

Performance

Measured on a 4P+4E Apple Silicon laptop with 8 GB of RAM, over three corpora of mixed Rust, Markdown and TypeScript:

Re-indexing 8334 unchanged files takes 1.7 seconds, because content hashes match and the model is never loaded. Cold CLI start on the 105k store is about 430 ms, most of it loading the model and the index.

Two things are worth reading off that table. Peak memory does not grow with the corpus — 105k chunks is 85 times the work of 1.2k for slightly less memory, so the number to plan for is roughly 600 MB whatever you point it at. Query latency does grow, because the index scan is linear: budget a few milliseconds for a repository and a few tens for a very large corpus.

The numbers that matter for an agent are the query row and the re-index figure. semlith mcp loads the model once at startup, so every tool call costs the warm figure; and keeping a store current is nearly free.

Indexing is the slow half, and that cost is the embedding model, not the index — a transformer on CPU is simply not fast. If you have a large corpus and can trade some retrieval quality for throughput, --model AllMiniLML6V2 is about 1.8x faster (6 transformer layers instead of 12).

Quantization is worth testing rather than assuming. The int8 build of the default model is both smaller and faster than its fp32 build on ARM, while BGE's quantized variants measured no faster than fp32 on the same machine — whether int8 wins depends on the model's graph, not on the architecture alone.

Thread count is chosen rather than left to ONNX Runtime. Its threads synchronise at every operator, so on a CPU with performance and efficiency cores a thread on a slow core paces the whole batch; semlith uses the performance-core count on Apple silicon and the full count elsewhere. Override with SEMLITH_EMBED_THREADS if your machine disagrees.

Contributing

Contributions are welcome. Start with CONTRIBUTING.md — it covers the checks CI runs, how the code is laid out, and what is deliberately out of scope.

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test                  # unit tests, offline
cargo test -- --ignored     # end-to-end round trip; downloads the model

For how the pieces fit together and why, see docs/architecture.md.

Everyone participating is expected to follow the Code of Conduct. Found a security problem? Please read SECURITY.md rather than opening an issue.

Known limits

• One writer per store. A second index run against a store already being indexed exits with an error naming the process that holds it, rather than waiting. Searching during an index run is fine.
• First-time indexing is bound by transformer speed on CPU, at roughly 23 chunks/sec — a 100k-chunk corpus takes over an hour. Subsequent runs only touch what changed, and cost seconds.
• Query latency grows with corpus size, from under 3 ms at a thousand chunks to low tens of milliseconds at a hundred thousand. The index scan is linear.
• The default model is English-only. semlith models lists multilingual alternatives, which must be chosen when the store is created.
• Results are not reranked. A cross-encoder over the top results would improve ordering, at a cost per query that a local tool should not pay by default.

License

Apache-2.0. See LICENSE and NOTICE.

Note that semlith downloads embedding model weights at runtime; those are covered by their own licenses. The default, ibm-granite/granite-embedding-small-english-r2, is Apache-2.0.