sift-core 0.4.0

Indexed regex search over codebases (library + grep-like CLI)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# index/trigram/

Trigram index construction and in-memory index handle. Walks the corpus, extracts trigrams, writes persistence files, and provides zero-copy access for queries.

## Modules

| File | Description |
|------|-------------|
| [`mod.rs`](mod.rs) | `TrigramIndex` struct, posting list intersection, `Index` impl |
| [`builder.rs`](builder.rs) | `IndexTableBuilder`: corpus walk, trigram extraction, incremental table construction |
| [`file_table.rs`](file_table.rs) | `MappedFilesView`, `FileFingerprint`: file ID to relative path + fingerprint mapping |
| [`storage/`](storage/) | Binary persistence format (lexicon, postings, trigram sets, mmap, format constants) |

## API

```rust
use sift_core::{TrigramIndex, TrigramIndexBuilder};

// Build
let index = TrigramIndexBuilder::new(&corpus_root).with_dir(&index_dir).build()?;

// Open
let index = TrigramIndex::open(&index_dir, &root, corpus_kind)?;

// Query
let path = index.file_path(FileId::new(0));
```

## Format

Each table file starts with an 8-byte magic header:

| File | Magic | Contents |
|------|-------|----------|
| `files.bin` | `SIFTFIL1` | Offset table + length-prefixed UTF-8 paths with fingerprints (mtime, size) |
| `lexicon.bin` | `SIFTLEX1` | Sorted trigram entries with postings offsets |
| `postings.bin` | `SIFTPST1` | Flat array of `u32` file IDs referenced by lexicon |
| `trigrams.bin` | `SIFTTRI1` | Per-file sorted unique trigram sets for incremental rebuild |

All integers are little-endian.