sift-core 0.6.0

Indexed regex search over codebases (library + grep-like CLI)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# index/trigram/storage/

On-disk binary format for the trigram index tables. All access is zero-copy via memory-mapped files.

## Modules

| File | Description |
|------|-------------|
| [`mod.rs`](mod.rs) | Module re-exports |
| [`format.rs`](format.rs) | Magic bytes (`SIFTFIL1`, `SIFTLEX1`, `SIFTPST1`, `SIFTTRI1`) |
| [`lexicon.rs`](lexicon.rs) | `LexiconEntry`, `MappedLexicon`: sorted trigram to postings slice descriptor |
| [`postings.rs`](postings.rs) | `MappedPostings`: contiguous `u32` LE file-ID payloads |
| [`trigram_sets.rs`](trigram_sets.rs) | `MappedTrigramSets`: per-file trigram sets for incremental updates |
| [`mmap.rs`](mmap.rs) | `open_mmap`: minimal memory-map wrapper (contains the only `unsafe` in the crate) |

## Format Overview

Each table file starts with an 8-byte magic header:

| File | Magic | Contents |
|------|-------|----------|
| `files.bin` | `SIFTFIL1` | Offset table + length-prefixed UTF-8 paths with fingerprints (mtime, size) |
| `lexicon.bin` | `SIFTLEX1` | Sorted trigram entries with postings offsets |
| `postings.bin` | `SIFTPST1` | Flat array of `u32` file IDs referenced by lexicon |
| `trigrams.bin` | `SIFTTRI1` | Per-file sorted unique trigram sets for incremental rebuild |

All integers are little-endian. Lexicon entries are sorted by trigram bytes for binary search.