sift-core 0.4.0

Indexed regex search over codebases (library + grep-like CLI)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

# sift-core

Indexed grep-style search engine. Build on-disk indexes, then run regex or fixed-string queries with automatic candidate narrowing. The shipped index type is a trigram index; the `SearchIndex` trait allows plugging in additional index kinds.

## Modules

| Module | Description |
|--------|-------------|
| [`query/`](src/query/) | Query description (`QuerySpec`), planning |
| [`index/`](src/index/) | `SearchIndex` trait, `Indexes` registry, shared types (`FileId`, `IndexId`, `IndexMeta`) |
| [`index/trigram/`](src/index/trigram/) | Trigram index: build, load, search, and persistence |
| [`index/trigram/storage/`](src/index/trigram/storage/) | Binary persistence format (lexicon, postings, file tables) |
| [`grep/`](src/grep/) | Pipeline orchestration: `GrepRequest`, `run()` |
| [`search/`](src/search/) | Regex execution, scanning, output formatting, parallelism |
| [`lib.rs`](src/lib.rs) | Public API re-exports, error types, constants |

## API

```rust
use sift_core::{SearchOptions, SearchQuery, TrigramIndex, TrigramIndexBuilder, Indexes, CandidateFilter};

// Build (using the shipped trigram index)
let index = TrigramIndexBuilder::new(&corpus_root).with_dir(&index_dir).build()?;

// Open
let index = TrigramIndex::open(&index_dir)?;

// Search (via grep pipeline)
let indexes = Indexes::open(&sift_dir)?;
let query = SearchQuery::new(&patterns, SearchOptions::default())?;
let run = GrepRequest {
    indexes: &indexes,
    filter: &filter,
    output,
    separators: &separators,
    collect: SearchCollection::none(),
    store_meta: None,
    walk_unindexed: false,
}
.run(&query)?;
```

`SearchQuery` compiles the regex once; repeated `run` calls reuse the compiled matcher and searcher cache.

## Features

| Feature | Effect |
|---------|--------|
| `profile` | Enables `sift-profile` binary and `tempfile` dependency |

## Testing

Unit tests are co-located with implementation files in `#[cfg(test)] mod tests` blocks. Integration tests live in `tests/`.

```bash
cargo test -p sift-core
cargo bench -p sift-core --bench query
cargo bench -p sift-core --bench index
cargo bench -p sift-core --bench grep
```

See [`benches/README.md`](benches/README.md) for the full benchmark and profiling workflow.