sift-core

Indexed grep-style search engine. Build on-disk indexes, then run regex or fixed-string queries with automatic candidate narrowing. The shipped index type is a trigram index; the SearchIndex trait allows plugging in additional index kinds.

Modules

API

use sift_core::{SearchOptions, SearchQuery, TrigramIndex, TrigramIndexBuilder, Indexes, CandidateFilter};

// Build (using the shipped trigram index)
let index = TrigramIndexBuilder::new(&corpus_root).with_dir(&index_dir).build()?;

// Open
let index = TrigramIndex::open(&index_dir)?;

// Search (via grep pipeline)
let indexes = Indexes::open(&sift_dir)?;
let query = SearchQuery::new(&patterns, SearchOptions::default())?;
let run = GrepRequest {
    indexes: &indexes,
    filter: &filter,
    output,
    separators: &separators,
    collect: SearchCollection::none(),
    store_meta: None,
    walk_unindexed: false,
}
.run(&query)?;

SearchQuery compiles the regex once; repeated run calls reuse the compiled matcher and searcher cache.

Features

Testing

Unit tests are co-located with implementation files in #[cfg(test)] mod tests blocks. Integration tests live in tests/.

cargo test -p sift-core
cargo bench -p sift-core --bench query
cargo bench -p sift-core --bench index
cargo bench -p sift-core --bench grep

See benches/README.md for the full benchmark and profiling workflow.