sift-core

Core engine for indexed code search. Build on-disk indexes over a codebase, then run regex or fixed-string queries with automatic candidate narrowing.

Index Architecture

The engine is built around composable indexes. Each index type independently narrows candidates for a query; the Indexes registry combines their results via set intersection. Today the shipped index type is a trigram index; the IndexKind and Index enums provide static dispatch for adding future index kinds (AST indexes, dependency graphs, vector indexes, etc.) without changing the query planner or search pipeline.

IndexKind::Trigram  ──build/open/update──>  Index::Trigram(TrigramIndex)
IndexKind::???      ──build/open/update──>  Index::???(...)
                                                 │
                                          Indexes registry
                                                 │
                              intersect candidate sets at query time

Modules

API

use sift_core::{
    IndexStore, IndexKind, IndexConfig, Indexes,
    SearchQuery, SearchOptions, GrepRequest, CandidateFilter,
};

// Build indexes (currently trigram; extensible to multiple kinds)
let mut store = IndexStore::open_or_create(&sift_dir, &meta)?;
store.build(&[IndexKind::Trigram], &config, &[])?;

// Open all indexes in the store
let indexes = Indexes::open(&sift_dir)?;

// Search via the grep pipeline
let query = SearchQuery::new(&patterns, SearchOptions::default())?;
let run = GrepRequest {
    indexes: &indexes,
    filter: &filter,
    output,
    separators: &separators,
    collect: SearchCollection::none(),
    store_meta: None,
    walk_unindexed: false,
}
.run(&query)?;

SearchQuery compiles the regex once; repeated run calls reuse the compiled matcher and searcher cache.

Features

Testing

Unit tests are co-located with implementation files in #[cfg(test)] mod tests blocks. Integration tests live in tests/.

cargo test -p sift-core
cargo bench -p sift-core --bench query
cargo bench -p sift-core --bench index
cargo bench -p sift-core --bench grep

See benches/README.md for the full benchmark and profiling workflow.