# SIFS benchmark report
These measurements were collected on May 7, 2026, on the development machine.
Results vary by hardware, so use them as a relative reference rather than an
absolute performance contract.
## Summary
SIFS was evaluated against 63 pinned open-source repositories, 19 languages,
and 1,251 annotated search tasks. The benchmark reports NDCG@10 for ranking
quality and separate timing fields for cold indexing, semantic first use,
warm queries, and cached repeats:
```text
cold_index_ms
cold_semantic_build_or_load_ms
cold_first_search_ms
warm_uncached_query_ms
warm_cached_repeat_query_ms
```
`warm_uncached_query_ms` bypasses the in-process query-result cache and is the
right field to compare for normal searches after an index exists.
`warm_cached_repeat_query_ms` measures identical repeated queries after one
warm-up. `cold_index_ms` covers sparse and chunk index construction only;
semantic and hybrid first-use cost is reported separately as
`cold_semantic_build_or_load_ms` and included in `cold_first_search_ms`.
| CodeRankEmbed Hybrid | 0.8617 | 57.3 s | 16.9 ms | n/a |
| Semble | 0.8544 | 439.4 ms | 1.3 ms | n/a |
| **SIFS** | **0.8471** | **167.0 ms** | **2.7 ms** | **0.0049 ms** |
| CodeRankEmbed | 0.7648 | 57.3 s | 13.3 ms | n/a |
| ColGREP | 0.6925 | 3.9 s | 979.3 ms | n/a |
| grepai | 0.5606 | 35.0 s | 47.7 ms | n/a |
| probe | 0.3872 | 0.0000 ms | 207.1 ms | n/a |
| ripgrep | 0.1257 | 0.0000 ms | 8.8 ms | n/a |
SIFS builds and queries faster than the neural embedding baselines and lands
just behind CodeRankEmbed Hybrid and Semble on NDCG@10 in this run. The
warm-query figure to compare against other tools is `2.7 ms`, not the
`0.0049 ms` cached-repeat figure.
## Figures
## Methodology
The SIFS result was generated with the Rust benchmark binary against the
annotated pinned-repository corpus:
```bash
cargo build --release --features diagnostics --bins
target/release/sifs-benchmark \
--benchmarks-dir /path/to/benchmark-corpus \
--bench-root /path/to/pinned-checkouts \
--output benchmarks/results/sifs-full.json \
--no-download \
--no-cache
```
For failure analysis, add `--include-tasks --candidate-diagnostics`. This
emits per-target final rank, BM25 rank, semantic rank, candidate-union
presence, and a coarse failure stage, so candidate-generation misses can be
separated from reranking misses. The output also includes a repo-level
`candidate_diagnostic_summary` with aggregated counts for top-10, reranking,
depth, and candidate-generation failure stages.
Comparison baselines come from result JSON files in the adjacent Python tool
checkout. The Semble row is included as a direct comparison to that tool.
| Semble | `semble-hybrid-0332378809c5.json` |
| CodeRankEmbed Hybrid | `coderankembed-0332378809c5.json` |
| CodeRankEmbed | `coderankembed-0332378809c5.json` |
| ColGREP | `colgrep-c8a40fab2235.json` |
| grepai | `grepai-715563a812c3.json` |
| probe | `probe-715563a812c3.json` |
| ripgrep | `ripgrep-fixed-strings-0332378809c5.json` |
Cold latency in the figures is cold index time plus warm uncached query p50.
Warm latency is warm uncached query p50 with an existing index. Some baseline
files only carry precomputed summary timing fields, which are kept as-is.
The full SIFS payload is checked in at
[benchmarks/results/sifs-full.json](../benchmarks/results/sifs-full.json). It
contains per-repository NDCG, latency, index time, memory, file count, chunk
count, and category-level scores.
The checked-in result JSON also includes per-repository `reproducibility`,
`cold_semantic_build_or_load_ms`, and `cold_first_search_ms` fields.
Regenerate with the command above on the target machine before quoting fresh
release claims.
## SIFS by language
| bash | 3 | 60 | 0.8994 | 1.037 ms | 0.0022 ms |
| c | 3 | 60 | 0.7370 | 4.690 ms | 0.0061 ms |
| cpp | 3 | 60 | 0.8521 | 2.929 ms | 0.0056 ms |
| csharp | 3 | 60 | 0.8586 | 3.233 ms | 0.0038 ms |
| elixir | 3 | 58 | 0.9202 | 1.456 ms | 0.0048 ms |
| go | 3 | 58 | 0.8378 | 1.152 ms | 0.0046 ms |
| haskell | 3 | 60 | 0.7777 | 3.040 ms | 0.0041 ms |
| java | 3 | 61 | 0.7948 | 5.606 ms | 0.0048 ms |
| javascript | 3 | 60 | 0.8919 | 0.492 ms | 0.0053 ms |
| kotlin | 3 | 60 | 0.8055 | 2.723 ms | 0.0045 ms |
| lua | 3 | 60 | 0.8504 | 2.795 ms | 0.0051 ms |
| php | 3 | 60 | 0.8073 | 4.063 ms | 0.0048 ms |
| python | 9 | 184 | 0.8609 | 1.046 ms | 0.0049 ms |
| ruby | 3 | 58 | 0.8774 | 0.890 ms | 0.0040 ms |
| rust | 3 | 60 | 0.8852 | 2.984 ms | 0.0051 ms |
| scala | 3 | 59 | 0.8991 | 2.697 ms | 0.0065 ms |
| swift | 3 | 53 | 0.8451 | 2.161 ms | 0.0046 ms |
| typescript | 3 | 60 | 0.7397 | 3.071 ms | 0.0062 ms |
| zig | 3 | 60 | 0.9318 | 7.442 ms | 0.0059 ms |
## SIFS by query category
| architecture | 0.7857 |
| semantic | 0.8412 |
| symbol | 0.9711 |
Symbol lookup is the strongest category: BM25 and query-aware boosts handle
exact identifiers while semantic retrieval covers natural-language discovery.
## Language relevance work
C is the weakest language slice in the full benchmark at `NDCG@10=0.7370`
across 60 tasks. TypeScript is next at `NDCG@10=0.7397`. A checked-in mini
corpus covers React components, hooks, type definitions, barrel exports,
`.d.ts` declarations, route files, and test/spec files:
- [tests/fixtures/ts-mini-corpus](../tests/fixtures/ts-mini-corpus)
- [tests/typescript_relevance.rs](../tests/typescript_relevance.rs)
The test/spec-file query uses a looser rank threshold because the current
ranking penalizes test files. That keeps the weakness visible before any
global ranking change.
## Large repository smoke test
A separate smoke benchmark runs against a shallow clone of
`https://github.com/facebook/react`. It's a scale and latency check, not an
annotated relevance test.
```bash
cargo build --release --example bench
target/release/examples/bench \
/path/to/react \
"how React schedules updates and work loops" \
100
```
Current checked-in result:
```text
cold_index_ms=2137.435 warm_uncached_query_ms=2.053 warm_uncached_query_p90_ms=2.322 warm_cached_repeat_query_ms=0.001 warm_cached_repeat_query_p90_ms=0.001 peak_rss_mb=461.9 files=4370 chunks=21096
```
The captured output is checked in at
[benchmarks/results/react-smoke.txt](../benchmarks/results/react-smoke.txt).
## Reproducing the graphs
The plotting script used for these graphs is checked in at
[benchmarks/plot_sifs_comparison.py](../benchmarks/plot_sifs_comparison.py). It
was run with `uv`:
```bash
uv run --with matplotlib \
benchmarks/plot_sifs_comparison.py \
--sifs-result benchmarks/results/sifs-full.json
```
The generated PNGs are written into [assets/images](../assets/images), and a
compact generated table is written to
[benchmarks/README.generated.md](../benchmarks/README.generated.md).
The query-type figure uses the current `--sifs-result` payload only.
Regenerate historical mode-ablation JSON files under
`benchmarks/results/sifs-mode-*.json` with the current benchmark binary and
`--no-cache` before using them for fresh comparison claims.
The context-efficiency figure is generated from
[benchmarks/results/sifs-context-curves.json](../benchmarks/results/sifs-context-curves.json),
a compact summary of context-mode benchmark runs for SIFS hybrid, BM25,
semantic search, and `ripgrep + read`. The ripgrep curve is generated from
[benchmarks/ripgrep_context_curve.py](../benchmarks/ripgrep_context_curve.py).
That helper writes the ignored raw payload `benchmarks/results/ripgrep-context.json`,
then the plotting script compacts the curve into
`benchmarks/results/sifs-context-curves.json`. It splits each query into
keywords, drops stopwords and short words, runs fixed-string ripgrep per
keyword, ranks files by distinct keyword coverage, and charges the prompt
budget for reading the full matched files in rank order. Token counts for the
ripgrep curve use the `cl100k_base` tokenizer when regenerated with
`uv run --with tiktoken`.