annex 0.1.0

ANNex: in-memory ANN (HNSW) vector search engine with payload filtering, snapshot persistence, and WAL replay.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

# Recall/Filtering test config cheat sheet

Tests are `#[ignore]` and driven entirely by env vars. Defaults favor loading prebuilt snapshots.

Snapshot defaults (shared):
- `VECTORDB_PERSIST_PATH` (or dataset-specific overrides below)
- `VECTORDB_USE_SNAPSHOT` (default `1`)
- `VECTORDB_ALLOW_BUILD` (default `0` — set to `1` to build if snapshot is missing)
- `VECTORDB_SAVE_SNAPSHOT` (default `0`)

- Core recall controls (shared):
  - `VECTORDB_M` / `VECTORDB_M0` (graph connectivity; M0 defaults to M)
  - `VECTORDB_EF_SEARCH_LIST` (or dataset-specific overrides below)
  - `VECTORDB_EARLY_EXIT_PATIENCE`
  - `VECTORDB_DISABLE_EARLY_EXIT` (set to `1` to turn it off)
  - `VECTORDB_SEARCH_EXPANSION_MULT` (default 1)
  - `VECTORDB_SEARCH_EXPANSION_CAP` (0 => uncapped for unfiltered search)
  - `VECTORDB_FILTER_EXPANSION_CAP` (0 => uncapped for filtered search; otherwise max expansions)
  - `VECTORDB_DISABLE_FILTER_SEEDS` (set to `1` to disable seeding; default seeds enabled)
  - `VECTORDB_FILTER_SEARCH_LOG` path to JSONL for per-query traversal stats
  - `VECTORDB_FILTER_PASSING_BUDGET` / `VECTORDB_FILTER_FAILING_BUDGET` (override routing budgets; defaults derive from M: passing≈max(8, 2*M), failing=1)
  - `VECTORDB_NEIGHBOR_SCAN_CAP_LEVEL0` (per-node window, default 64; 0 disables)
  - `VECTORDB_NEIGHBOR_SCAN_ROTATE` (1 to shift the window start, default 0/off)
  - `VECTORDB_NEIGHBOR_SCAN_STRIDE` (1 to stride through the neighbor list, default 0/off)
  - `VECTORDB_NEIGHBOR_SCAN_PATIENCE` (consecutive non-improving neighbors before stopping the scan; 0 disables, defaults to 0)

- Logging/analysis:
  - `VECTORDB_TEST_LOG` -> `quiet|info|debug` (default `info`)
  - `VECTORDB_PROGRESS_EVERY` -> progress interval for debug logs (default `100`)
  - `VECTORDB_INSERT_PROGRESS_EVERY` -> print insert progress every N vectors during builds (default `1000`; set to `0` to disable)
  - `VECTORDB_QUERY_LOG` -> JSONL path for per-query stats
  - `VECTORDB_QUERY_LOG_EVERY` -> sample interval for per-query stats (default `1`)
  - `VECTORDB_SEARCH_TRACE_LOG` -> JSONL path for unfiltered traversal trace
  - `VECTORDB_INSERT_TRACE_LOG` -> JSONL path for insert traversal trace
  - `VECTORDB_TRACE_EVERY` -> sample interval for trace logs (default `100`)
  - `VECTORDB_FILTER_SEARCH_LOG` -> JSONL with per-query expansions/visited/filter hit rates/seeds/stop_reason.

- WAL:
  - `VECTORDB_WAL_PATH` (auto-enable WAL for new segments when set)
  - `VECTORDB_WAL_DIR` (auto-enable WAL; uses `<dir>/segment.wal`)
  - `VECTORDB_WAL_AUTO_REPLAY` (default `1`)
  - `VECTORDB_WAL_FSYNC` (default `1`)
  - `VECTORDB_WAL_FSYNC_EVERY` (default `100`)
  - `VECTORDB_WAL_FSYNC_MS` (default `100`)
  - `VECTORDB_MAX_RSS_MB` (0/empty disables memory cap)
  - `VECTORDB_OOM_SNAPSHOT_PATH` (path to write snapshot before unloading)

- NYTimes (unfiltered):
  - `VECTORDB_NYT_DATA_DIR` (default `data/nytimes-256-angular`)
  - `VECTORDB_NYT_PERSIST_PATH` (default `data/nytimes-256-angular/index_m16_m0_32_efc100.bin`)
  - `VECTORDB_NYT_USE_SNAPSHOT` (override `VECTORDB_USE_SNAPSHOT`)
  - `VECTORDB_NYT_ALLOW_BUILD` / `VECTORDB_NYT_SAVE_SNAPSHOT`
  - `VECTORDB_NYT_M` / `VECTORDB_NYT_M0`
  - `VECTORDB_NYT_EF_SEARCH_LIST` (falls back to `VECTORDB_EF_SEARCH_LIST`)
  - `VECTORDB_EF_CONSTRUCT` / `VECTORDB_NYT_EF_CONSTRUCT`

- H&M (filtered):
  - `VECTORDB_HNM_DATA_DIR` (default `data/hnm`)
  - `VECTORDB_HNM_PERSIST_PATH` (default `data/hnm/index_filtered.bin`)
  - `VECTORDB_HNM_USE_SNAPSHOT` (override `VECTORDB_USE_SNAPSHOT`)
  - `VECTORDB_HNM_ALLOW_BUILD` / `VECTORDB_HNM_SAVE_SNAPSHOT`
  - `VECTORDB_HNM_M` / `VECTORDB_HNM_M0` (override HNSW M/M0 for rebuilds; M0 defaults to M)
  - `VECTORDB_HNM_EF_SEARCH_LIST` (falls back to `VECTORDB_EF_SEARCH_LIST`)
  - `VECTORDB_HNM_EF_CONSTRUCT` (build-time)
  - `VECTORDB_HNM_TOPK` (default derived from test cases)
  - `VECTORDB_DISABLE_FILTER_SEEDS` (set to `1` for unseeded runs)

Example commands:

- H&M seeded, EF=64, patience=3, uncapped filtered expansions:
```
VECTORDB_HNM_PERSIST_PATH=data/hnm/index_filtered.bin \
VECTORDB_HNM_EF_SEARCH_LIST=64 \
VECTORDB_EARLY_EXIT_PATIENCE=3 \
VECTORDB_SEARCH_EXPANSION_MULT=4 \
VECTORDB_FILTER_EXPANSION_CAP=0 \
VECTORDB_FILTER_SEARCH_LOG=logs/hnm_seeded_ef64.jsonl \
cargo test --release hnm_recall_from_snapshot -- --ignored --nocapture
```

- H&M unseeded, EF=64, patience=3, uncapped filtered expansions:
```
VECTORDB_HNM_PERSIST_PATH=data/hnm/index_filtered.bin \
VECTORDB_HNM_EF_SEARCH_LIST=64 \
VECTORDB_EARLY_EXIT_PATIENCE=3 \
VECTORDB_SEARCH_EXPANSION_MULT=4 \
VECTORDB_FILTER_EXPANSION_CAP=0 \
VECTORDB_DISABLE_FILTER_SEEDS=1 \
VECTORDB_FILTER_SEARCH_LOG=logs/hnm_noseed_ef64.jsonl \
cargo test --release hnm_recall_from_snapshot -- --ignored --nocapture
```

- NYT unfiltered, EF list example:
```
VECTORDB_NYT_PERSIST_PATH=data/nytimes-256-angular/index.bin \
VECTORDB_EF_SEARCH_LIST=64,128,256,512 \
VECTORDB_EARLY_EXIT_PATIENCE=3 \
VECTORDB_SEARCH_EXPANSION_MULT=4 \
VECTORDB_SEARCH_EXPANSION_CAP=0 \
cargo test --release nytimes_recall_from_snapshot -- --ignored --nocapture
```