Skip to main content

Crate ripindex

Crate ripindex 

Source
Expand description

ripindex — a persistent, memory-mapped inverted index over a directory tree.

Pipeline:

crawler ──► Vec<FileEntry> ──► index::builder ──► Index<MemTermStore>
                                 (tokenizer)          │
                                                       ▼
                      query::parse ──► query::exec ──► Vec<Hit> ──► snippet

Storage boundary. The query engine sees an index only through index::IndexReader / index::SegmentReader and the posting-list traits in index::postings. index::MemIndex is the in-RAM segment the builder fills; store serialises it into the on-disk format in docs/FORMAT.md and reads it back through memory maps behind the same traits. Nothing in query/ or snippet knows which backend it is on.

File contents are never stored. Snippets re-read the file at query time.

Beyond a fresh build, store also covers reconciling an index against a changed filesystem without rebuilding it (store::update_index, backed by store::state’s staged, cheapest-first change detection), merging small or heavily-tombstoned segments (store::merge_index), a refreshable handle safe to hold across concurrent updates and merges (store::LiveIndex), and a hint-driven watcher with deterministically testable scheduling (store::watcher). The watcher is a hint, never a source of truth — store::state’s module docs say why.

Re-exports§

pub use error::Error;
pub use error::QueryError;
pub use error::Result;
pub use index::DocId;
pub use index::Index;

Modules§

bench
ripindex bench: build, measure open (cold, best-effort, and warm), time a set of queries, and watch RSS while they run.
crawler
Directory crawler: turns a root directory into the ordered list of files the indexer will consider.
daemon
M4: the daemon. Every mutation to any root — watch, reconcile, update, merge — happens in exactly one process, reached by every client (the CLI, future plugins) over a local, permission-restricted transport. See protocol for the wire format, transport for the socket/pipe layer, actor for the per-root single-writer task, server for the accept loop, and client for the autostart-and-connect logic the CLI uses.
error
format
On-disk encodings — pure functions over byte slices, no I/O. The layout is specified field-by-field in docs/FORMAT.md.
fs
File-system access behind a trait, so the commit protocol can run under injected faults.
index
Index abstractions and the in-memory segment.
query
Query parsing, evaluation and ranking.
report
Human-readable formatting shared by index and bench.
snippet
Result snippets. The index holds no content, so the file is re-read here.
store
The persistent index: .ripindex/ under the indexed root, committed through an atomic manifest protocol. Layout and encodings are specified in docs/FORMAT.md; writer implements §7 (commit), reader §8 (open).
tokenizer
Unicode-aware tokenizer with code-identifier splitting.