# Pith — PRD (crate: pithdb, binary: pith)
Status: Draft v0.1
Owner: Vanja
Language: Rust (stable, edition 2021+)
## 1. Summary
Pith is a lightweight, embeddable vector database engine with built-in ONNX
embedding inference, written in Rust. One static binary / one library crate
that can index text and code, embed it locally, and serve semantic + keyword
search. It must run identically on-device (laptop, phone-class ARM, edge) and
on a small server, with no external services and no Python.
Primary first use case: local context retrieval for AI coding tools (Claude
Code, Hermes, etc.) over my own projects, exposed through MCP and CLI.
## 2. Goals
- G1. Single-binary, zero-dependency install (no Docker, no Python, no daemon required).
- G2. Embedded-first: usable as a Rust library, a CLI, an HTTP server, or an MCP server from the same core.
- G3. Fully local embeddings via ONNX Runtime; no network calls at query or index time.
- G4. Small footprint: idle RSS < 50 MB (excl. loaded model), index overhead low via quantization.
- G5. Good code search: tree-sitter chunking + hybrid (vector + BM25) retrieval.
- G6. Durable and crash-safe: WAL + immutable segments, safe on power loss.
- G7. Portable index files: one directory, copyable between machines of the same format version.
## 3. Non-goals (v1)
- Distributed clustering / sharding / replication.
- Multi-tenant auth and billing.
- Billion-scale datasets (target is up to ~10M vectors per node).
- Training models or fine-tuning.
- GPU requirement (GPU is an optional execution provider only).
## 4. Users and use cases
1. Me, daily: index all projects under `~/Desktop/CODING PROJECTS`, query from Claude Code via MCP.
2. Local apps: embed Pith as a crate for in-app semantic search (notes, files, docs).
3. Small server: self-hosted search/RAG backend for a side project on a cheap VPS (1 vCPU / 1 GB).
4. On-device: macOS/Linux/Windows, aarch64 and x86_64; later iOS/Android via static lib.
## 5. Functional requirements
### 5.1 Collections and data model
- A database is a directory. It contains named collections.
- Collection: fixed dimension, distance metric (cosine default, dot, L2), embedding model id.
- Record: `id` (string/u64), `vector` (f32 on input), `text` (optional), `metadata` (JSON key/values), `source` (file path + byte/line range for code).
### 5.2 Embeddings (ONNX)
- Embedded inference via ONNX Runtime (`ort` crate) with tokenizers (`tokenizers` crate, HF tokenizer.json).
- Default model: small, quantized sentence embedding model (e.g. bge-small-en-v1.5 / all-MiniLM-L6 int8, 384-d). Code-oriented model selectable (e.g. jina-embeddings-v2-base-code, nomic-embed-text).
- Models are plain files in a `models/` dir; pluggable by config (model path, tokenizer path, pooling, normalize, max tokens, prefixes).
- Execution providers: CPU default; optional CoreML (macOS/iOS), CUDA/DirectML via feature flags.
- Batched inference with configurable batch size and thread count.
- Model download is an explicit command (`pith model pull <name>`); never implicit at runtime.
- "Bring your own vectors" mode: ONNX can be compiled out (feature flag) for a minimal build.
### 5.3 Indexing
- v0.1: flat (exact) index, SIMD distance kernels.
- v0.3: HNSW (default above ~50k vectors), persistent, with tuneable M / ef_construction / ef_search.
- Optional later: IVF-PQ or DiskANN-style for beyond-RAM datasets.
- Quantization: f32 -> f16 -> int8 scalar; binary quantization with rescoring for large sets.
- Deletes via tombstones; background compaction merges segments.
### 5.4 Search
- `search(query_text | query_vector, k, filter, mode)`.
- Modes: `vector`, `keyword` (BM25), `hybrid` (reciprocal rank fusion, weights configurable).
- Metadata filters: equality, in, range, and/or/not; evaluated via bitmap pre-filter.
- Returns id, score, text snippet, metadata, source location.
### 5.5 Code ingestion
- Walk directory respecting `.gitignore` (`ignore` crate).
- Chunk by syntax with tree-sitter (function / class / impl / block) for Rust, TS/JS, Python, Go, Swift, C/C++, Java, JSON/MD fallback to window chunking.
- Incremental re-index: per-file content hash; only changed files re-embedded; removed files tombstoned.
- File watcher mode (`pith watch <dir>`) using `notify`.
- Chunk context header (path, language, symbol name) prepended before embedding.
### 5.6 Interfaces
- Rust crate API (`pith-core`).
- CLI:
- `pith init <db>`
- `pith model pull|list`
- `pith index <path> [--collection c]`
- `pith watch <path>`
- `pith search "query" [-k 10] [--mode hybrid] [--filter ...]`
- `pith stats | compact | export | import`
- HTTP server (`pith serve`): JSON REST (axum), OpenAPI spec.
- MCP server (`pith mcp`, stdio transport): tools `search_code`, `index_path`, `get_chunk`, `stats`.
- C ABI + Swift/Kotlin bindings: stretch goal, post v1.
### 5.7 Storage and durability
- Layout:
```
db/
MANIFEST (format version, collections, segment list)
wal/ (append-only log)
collections/<name>/
segments/ (immutable: vectors.bin, graph.bin, meta, text)
bm25/ (inverted index)
models/ (optional, symlink-able)
```
- Vectors memory-mapped (`memmap2`); graph and metadata loaded lazily.
- WAL fsync policy configurable (every write / interval / none).
- Format versioning with forward migration tooling.
- Single-writer, many-reader concurrency; readers never block on writer.
## 6. Non-functional requirements
| Binary size | < 25 MB core + ONNX Runtime (static/dynamic), < 8 MB without ONNX |
| Idle memory | < 50 MB excl. model; model int8 small ~ 30-130 MB |
| Query latency (flat, 100k x 384d) | p95 < 15 ms excl. embedding |
| Query latency (HNSW, 1M x 384d) | p95 < 10 ms at recall@10 >= 0.95 |
| Embedding throughput | >= 200 chunks/s on M-series CPU (small model, batch 32) |
| Index time (100k LOC repo) | < 60 s cold, < 2 s incremental |
| Recall (HNSW vs exact) | >= 0.95 recall@10 default params |
| Platforms | macOS arm64/x86_64, Linux x86_64/aarch64 (musl where possible), Windows x86_64 |
| Crash safety | No corruption after kill -9 / power loss during write |
| Min server | 1 vCPU, 1 GB RAM with small model |
## 7. Architecture
```
+-------------------------------------------+
|
+--------v---------+
| pith-core |
| collections, |
| query planner |
+--+----+-----+----+
| | |
+---------------+ | +----------------+
| | |
+-------v------+ +--------v-------+ +--------v-------+
| pith-index | | pith-storage | | pith-embed |
| flat / HNSW | | WAL, segments, | | ONNX (ort) + |
| quant, SIMD | | mmap, manifest | | tokenizers |
+--------------+ +----------------+ +----------------+
|
+---------v---------+
| pith-ingest |
| tree-sitter, walk |
| hash, watch |
+-------------------+
```
Cargo workspace crates: `pith-core`, `pith-index`, `pith-storage`,
`pith-embed`, `pith-ingest`, `pith-cli`, `pith-server`, `pith-mcp`.
Heavy deps behind features: `onnx`, `treesitter`, `server`, `mcp`, `coreml`, `cuda`.
Key crates: `ort`, `tokenizers`, `memmap2`, `tree-sitter` (+ grammars),
`tantivy` or custom BM25, `ignore`, `notify`, `axum`, `tokio`, `serde`,
`rmcp`, `clap`, `rayon`, `half`, `criterion`.
## 8. Milestones
| v0.1 | Workspace, flat index, SIMD cosine/dot/L2, in-memory + save/load, bench harness | Exact search correct vs NumPy reference; criterion benches in CI |
| v0.2 | ONNX embedding (ort + tokenizers), `model pull`, CLI `index`/`search` on text files | Embed + search a folder end to end offline |
| v0.3 | WAL + segments + manifest, crash-recovery tests, tombstones | kill -9 fuzz test passes |
| v0.4 | tree-sitter chunking, gitignore walk, incremental hashing, `watch` | Re-index of changed file < 2 s |
| v0.5 | BM25 + hybrid RRF, metadata filters | Hybrid beats vector-only on code query eval set |
| v0.6 | MCP server (stdio) | Claude Code can call `search_code` on my projects |
| v0.7 | HNSW persistent, ef tuning, recall harness | recall@10 >= 0.95 at p95 < 10 ms on 1M |
| v0.8 | int8/binary quantization + rescoring | 4x+ memory reduction, recall loss < 2% |
| v0.9 | HTTP server, OpenAPI, Docker-less deploy docs, cross-compile releases | Runs on 1 vCPU/1 GB VPS |
| v1.0 | Format freeze, docs, install script/Homebrew, release binaries | Daily-driven for 30 days without data loss |
## 9. Evaluation
- Recall: HNSW vs flat ground truth on ANN-Benchmarks style sets (SIFT/GloVe) and on my own embeddings.
- Retrieval quality: hand-built eval set of ~100 queries over my repos (query -> expected file/symbol); metrics MRR, recall@5, nDCG.
- Performance: criterion microbenchmarks + end-to-end index/query timings tracked per release.
- Reliability: crash-injection tests, property tests (`proptest`) on storage and index invariants.
## 10. Risks and mitigations
| ONNX Runtime bloats binary / hard to cross-compile | Feature-flag ONNX; offer dynamic-load of the ORT shared lib; `tract` as pure-Rust fallback to evaluate |
| Persistent HNSW complexity (updates, deletes) | Ship flat first; HNSW built per immutable segment, merge in compaction |
| Embedding model licensing | Only default to permissively licensed models; record license in model manifest |
| Scope creep | Milestone gating; each milestone must be usable on its own |
| Learning Rust while shipping | Keep core simple, avoid unsafe except SIMD/mmap, wrap and test those |
| Index format churn | Version field from day one; no freeze until v1.0 |
## 11. Open questions
1. Name: "Pith" chosen. Crate `pithdb`, binary `pith`. Free on crates.io, npm, PyPI, .dev/.io/.com and Homebrew as of research date.
2. `ort` (ONNX Runtime) vs `tract` vs `candle` for default backend: benchmark in v0.2.
3. Custom BM25 vs `tantivy` dependency (size vs effort).
4. Default code embedding model: size/quality tradeoff on my eval set.
5. License (MIT/Apache-2.0 dual is the default plan).
## 12. Success criteria
- I use it daily as my coding-context backend through MCP for 30 days.
- Cold start to first result < 1 s on laptop; works fully offline.
- Runs on a 1 GB VPS and on a laptop from the same release binary family.
- Someone else can install and index a repo in under 5 minutes using the README.