Skip to main content

Crate pithdb

Crate pithdb 

Source
Expand description

Pith: a local single-file vector database with built-in ONNX embeddings, for searching a codebase by meaning.

Re-exports§

pub use commands::index_project;
pub use commands::index_project_with_progress;
pub use commands::is_no_model;
pub use commands::is_yes;
pub use commands::load_default_embedder;
pub use commands::load_optional_embedder;
pub use commands::open_project;
pub use commands::render_error;
pub use commands::render_stats;
pub use commands::search_project;
pub use commands::search_with;
pub use commands::skip_summary;
pub use commands::stats_project;
pub use commands::IndexReport;
pub use commands::Progress;
pub use commands::SearchHit;
pub use commands::SearchMode;
pub use commands::Stats;
pub use commands::KEYWORD_ONLY;
pub use error::Error;
pub use error::Result;

Modules§

commands
Library behind the pith subcommands: index a project, search it, report stats. Network access never happens here (only embed::pull touches the network).
embed
Text embeddings: the Embedder trait, the ONNX implementation (feature onnx), a deterministic [FakeEmbedder] for tests, and verified model delivery (pull).
error
index
Cosine index: an int8 SIMD scan selects the top max(CANDIDATE_MULT * k, 32) candidates, which are then rescored with f32 dot products. Not strictly exact: a true top-k vector could be lost in the int8 pass (recall is checked in tests/index_reference.rs).
ingest
Ingest: walk a project, drop secrets and junk, window files into chunks.
mcp
pith mcp: a read-only stdio MCP server for one project. Tools: search_code, get_chunk, stats. It never indexes, never writes, and never takes the writer lock. Stdout carries only the protocol; nothing else in this process may print there.
storage
Single-file database (layout in format.rs). Writes replace the whole file atomically (unique temp file, fsync, rename, fsync dir) under the writer lock; Pith never modifies a database file in place, so a reader sees a complete old or new file. Vectors stay in a shared read-only mapping for the life of the index.
tui
pith ui: a terminal explorer for one project’s index. Stack (directory -> file -> chunk), Map (2-D PCA of the vectors), RAG (what search_code would send Claude) and Index (a live reindex). Reads the database like pith mcp: read-only, no writer lock. Only the Index view writes, through index_project_with_progress, which takes the lock for its final write. A reindex from elsewhere is picked up on r, not automatically.