roust

Recall-first code retrieval for coding agents.

Point an agent at grep and it has to iterate on search terms, reading through a lot of matches to find what it needs. Point it at roust and it gets a single, ranked, token-budgeted bundle of the relevant code back in one call, with no embeddings, no LLM calls, no API keys, and no training. roust is a ranking-and-packing pipeline over plain lexical, structural, and version-control signals — it reads like a very disciplined grep session, compressed into one process call.

Eight languages, packed as real functions and classes rather than blind line windows: Python, JavaScript, TypeScript, Java, Go, Rust, C, and C++. Each one's structural support was gated on its own benchmark slice before shipping — the per-language numbers, including where roust is weak, are in Multi-language localization. Any other language still indexes and ranks; it just packs windows instead of syntactic units.

Validated on 407 held-out SWE-bench Verified instances (92.1% all-gold-files, never tuned on), on the archex head-to-head benchmark (40/40 tasks at recall 1.00), and per-language on Multi-SWE-bench. Python is the strongest slice by a wide margin, and the scoreboard says so.

Install

roust is a single Rust binary (~15 MB release build; ~9.3 MB of that is the exactly-pinned tree-sitter grammars for the eight languages whose structural packing they power — grammar bumps are gated dependency changes). Every install path below builds the same roust-rs engine — there is no separate Python implementation.

v0.3.0 is published. Install it one of three ways:

# npm — downloads the prebuilt binary for your platform (no Node runtime cost;
# the launcher just execs it). The package is `roust-cli`; the command it
# installs is `roust`. Works with npx too.
npm install -g roust-cli
npx roust-cli "connection pooling" ~/code/httpx

# crates.io — builds from source
cargo install roust

# GitHub Releases — raw per-platform binaries with .sha256 checksums
# https://github.com/narehart/roust/releases

Or from a checkout:

git clone https://github.com/narehart/roust && cd roust
cargo install --path roust-rs

Prebuilt binaries cover darwin-x64, darwin-arm64, linux-x64, linux-arm64, and win32-x64. Any other platform builds from source via cargo install roust.

git should be on PATH if you want the commit-history signal (roust degrades gracefully without it).

Developing against roust-rs/: uv run roust does not rebuild automatically when roust-rs/src changes -- after any Rust edit, run uv sync --reinstall-package roust before relying on uv run roust again (roust --version embeds a git SHA + dirty flag so a stale build is identifiable; see lab/tokenbench/README.md's engine-provenance guard for the automated version of this check).

The first roust call against a repo builds an index — a few hundred milliseconds to a few seconds depending on repo size. The index is cached under <repo>/.roust/ (add that directory to your .gitignore) and refreshes automatically whenever indexed files change, so every call after the first is a cache hit.

Usage

roust QUERY [PATH]

QUERY can be a natural-language question or raw issue/error text; PATH defaults to .. Default output on stdout is a token-budgeted, packed bundle of the most relevant code regions; a one-line stats summary always goes to stderr, so stdout stays clean for piping.

A real session, run against encode/httpx:

$ roust "connection pooling" ~/code/httpx
[... ~8.4k tokens of packed file regions on stdout ...]
roust: 25 files, 8366 tokens (indexed 57 files, index 9ms, query 160ms, cache hit)

Other flags:

# Ranked file paths only, one per line -- for fast localization
roust "connection pooling" ~/code/httpx --files-only

# Machine-readable output: files, packed regions, bundle text, timing stats
roust "connection pooling" ~/code/httpx --json

# Cap the file count (0 = no cap, the default)
roust "connection pooling" ~/code/httpx --k 5

# Change the token budget for the packed bundle (default: 8192).
# The budget is a target, not a hard cap: the packer seats a minimum core of
# regions first, so small budgets can overshoot (measured 1.04-1.22x at 2048-8192
# on requests/flask). Check stats.bundle_tokens in --json for the actual size.
roust "connection pooling" ~/code/httpx --budget 4096

# Force a fresh index build even if a cache entry exists
roust "connection pooling" ~/code/httpx --reindex

# Skip the on-disk cache entirely (neither reads nor writes .roust/)
roust "connection pooling" ~/code/httpx --no-cache

# Disable individual signal channels (all on by default)
roust "connection pooling" ~/code/httpx --no-history     # git commit-message field + co-change frontier
roust "connection pooling" ~/code/httpx --no-docs        # *.rst/*.txt/*.md docs-bridge
roust "connection pooling" ~/code/httpx --no-anchors     # definition-symbol anchor channel
roust "connection pooling" ~/code/httpx --no-testbridge  # test-file lexical bridge

# Region-packing knobs (the shipped defaults, adopted from the #4 campaign):
# pad every packed span by N context lines (guarded padding, default 5)
roust "connection pooling" ~/code/httpx --pad-lines 5
# length-normalization exponent in region selection, gain/tokens^len_exp
# (default 0.85; `--pad-lines 0 --len-exp 1.0` reproduces the pre-adoption packing)
roust "connection pooling" ~/code/httpx --len-exp 0.85

# Dump the full diagnostic record (the engine's Explain struct) as JSON to stderr
roust "connection pooling" ~/code/httpx --explain

Exit codes: 0 = results found (this includes low-confidence matches, see below -- roust still returns its best guess), 1 = no query term matched anything in the indexed corpus vocabulary at all, 2 = usage error.

Low-confidence matches

roust always returns a budget-filled bundle for any query that matches at least one term somewhere in the repo -- it doesn't refuse to answer just because the match is weak. To make that weak-match case visible instead of silent, --json output's stats includes:

• top_score: the strongest candidate file's raw (pre-normalization) BM25F score for this query -- comparable across queries and repos, unlike the 0-1 normalized scores used for ranking.
• matched_query_terms / total_query_terms: how many of the query's terms exist anywhere in the indexed corpus vocabulary (body text, comments, docs pages, commit messages, or path components).
• low_confidence: true, present only when the calibrated criterion trips (top_score below a fixed threshold, or fewer than 45% of query terms found in the corpus vocabulary) -- also appended as [low-confidence match] to the stderr summary line.

The thresholds were calibrated empirically against all 300 SWE-bench Lite (query, repo) pairs -- 0 false trips on that real-query population is the hard constraint -- checked against ~30 gibberish/off-topic queries across 3 repos. Because real BM25F scores scale with query length and repo size, this signal is calibrated for realistic-size repositories; a tiny few-file toy repo can legitimately score below the threshold even on a genuinely on-topic query.

Output size: agents vs humans

The default --budget 8192 is sized for LLM context windows, not for a human scrolling a terminal. A coding agent reads the bundle selectively, and roust's recall-first packing is measured against that use case: 93.3% agent-loop solve rate. A human reading the same bundle top-to-bottom will find it broad by design -- region precision is intentionally traded for recall, so the bundle covers as many candidate edit sites as fit in the budget rather than just the single best match.

For hand use, shrink the bundle instead of reading past it:

# Quarter-size bundle, same latency, best-ranked content first
roust "connection pooling" ~/code/httpx --budget 2048

# Cap the file count directly
roust "connection pooling" ~/code/httpx --k 8

# Scannable list instead of packed code
roust "connection pooling" ~/code/httpx --files-only

One honest caveat: shrinking the budget trades away recall roughly linearly (measured -- see issue #4's tail-cut experiment log), so leave the default alone for agent use.

Using with coding agents

This is the point of the tool: an agent that reaches for roust before grep gets the files it needs in one shot, without needing to iterate on search terms across a much larger result set. Don't lower --budget in agent configs -- the breadth is the product, since the agent reads selectively rather than top-to-bottom; a smaller budget just trades away measured recall for no benefit to the agent.

Claude Code

Add to your project's CLAUDE.md:

## Code search

Before using grep/find/glob to explore this repo, run roust first:

- `roust "<question or issue text>" --files-only` to localize which files
  are relevant.
- `roust "<question or issue text>"` to get a packed bundle of the actual
  relevant code, ready to read.

Pass the raw question or issue text as the query -- don't summarize or
clean it up first. Include error messages, stack traces, file paths, and
backtick-quoted symbol/function names verbatim; roust uses those as
high-precision anchors. Only fall back to grep for a literal string match
roust's bundle doesn't cover.

And allowlist it in .claude/settings.json so it runs without a permission prompt:

{
  "permissions": {
    "allow": ["Bash(roust *)"]
  }
}

Cursor

Add to .cursorrules:

Before grepping this repo, run `roust "<question or issue text>" --files-only`
(or without --files-only for a packed code bundle) in the terminal to find
relevant files. Pass the raw question/issue text as the query, including
error strings and backtick-quoted symbol names -- don't paraphrase it first.

Aider

Invoke it from chat with /run:

/run roust "TypeError in connection pool cleanup" --files-only

And add a line to CONVENTIONS.md:

Search this repo with `roust "<raw question or error text>"` before grep --
it returns a token-budgeted bundle of the relevant code directly.

OpenAI Codex CLI / generic agents

Add to AGENTS.md:

## Code search

Run `roust "<question or issue text>" --files-only` to localize relevant
files, or `roust "<question or issue text>"` for a ready-to-read code
bundle. Pass the raw question/issue text verbatim as the query (error
messages, paths, and backtick-quoted symbols included) rather than a
cleaned-up paraphrase.

MCP

No MCP server yet (it's on the roadmap) -- roust is shell-first by design today, since every agent already has a shell and roust is a single subprocess call with structured --json output when you need it.

Query tips for agents

• Pass the raw issue/question text verbatim as the query.
• Include error messages, stack traces, and symbol names -- don't strip them out.
• Don't summarize the question into clean prose first: measured on adversarial paraphrases that drop key terms, mean task recall falls from 1.00 to 0.833, and only 14 of 19 tasks still retrieve every gold file -- summarization removes the anchors roust relies on. See "Known limits" in lab/README.md.

How it works

• BM25F over identifier subtokens (camelCase/snake_case split, Porter-lite stemming), with path tokens as a separate weighted field and an implementation-file prior (tests/docs/examples down-weighted).
• 1-hop structural expansion over the import/same-package graph, with RM3 pseudo-relevance feedback carrying evidence from lexical hits to their quiet neighbors.
• Commit-message channel: git history text folded in as a monotone addition-only signal (never reorders the lexical head).
• Definition-symbol anchors: rarity-gated (symbol defined in ≤3 impl files), tiered promotion so only strong anchors can enter the top ranks.
• Test/docs bridges: tests and docs are treated as developer-written natural-language-to-code mappings, appended tail-only.
• Greedy weighted-coverage region packing under a token budget, so the final bundle is code regions, not whole files, chosen to maximize coverage per token.

Every component above was added to fix a concrete, measured miss, and every number in this README is reproduced in the pipeline's research log, including negative results and a pre-registered held-out validation run: see lab/README.md.

Scoreboard

Given the same task and the same agent (tokenbench v2, live Sonnet 4.5; the grep and roust arms get their method as the agent's only search tool, the embedding-RAG arm gets rag_search plus grep, and every arm also has a read_file tool), roust solves 93.3% of tasks, grep 26.7%, embedding-RAG 80.0% (9-trial mean) — n=15, a partial run (see below). roust is not the most accurate retriever available: trained retrievers (see Localization accuracy below) score higher on published localization benchmarks. What roust offers is the best result you can get for free — no model, no embeddings, no API key, no training.

Agent-loop outcomes (our protocol)

• roust costs more per attempt than grep (308k vs 240k tokens) and wins on solve rate anyway. grep is cheap because it gives up: 73.3% of its runs hit the turn cap and produce nothing.
• $ / successful run remains a lower bound on cost-to-answer for a single attempt. The full repeat-run campaign (#16, results_repeats.jsonl) measured the rest: for roust and grep, failures are stable across trials (p≈0 — the retry term is meaningless; roust's one miss failed 10/10), while embedding-RAG's failures are genuinely stochastic. Its per-instance E[cost to first success] — (1−p̂)/p̂ × mean failed-attempt cost + mean successful-attempt cost, p̂ over the 10 trials (trial 0 + 9 repeats) — aggregates over its solvable set (all 15 instances, per-instance p̂ 0.10–1.00) to a median of $2.42 per instance; the mean is $4.90, dominated by django-16400 (p̂ = 0.10, E ≈ $31). An earlier revision stated "$2.50" here without naming the aggregation; the stated-convention numbers above replace it.
• Giving the agent grep alongside roust makes it worse (93.3% → 57.1%): replace grep, don't supplement it (#5).
• embedding-RAG's Solves cell is a 9-trial mean, lab/tokenbench/results_repeats.jsonl; its other columns (median turns, tokens, $) are the trial-0 measurement, lab/tokenbench/results.jsonl.
• The two + stopping prompt rows are the forced-stopping steelman arms (grep_forced/roust_forced, hard stopping directive + 12-turn cap): lab/tokenbench/results_forced.jsonl.
• Outcome volatility (measured across 9 identical trial repeats, lab/tokenbench/results_repeats.jsonl): embedding-RAG bounces 73.3–86.7% across 9 identical runs (mean 80.0% ± 4.4pp); roust reproduced 93.3% exactly with 0 outcome flips across all repeats, and its single failure (django-16400) failed 10/10 trials — a capability gap, not variance (p < 0.30 at 95%, rule of three). grep's failures were stable across both its trials.

Localization accuracy (published protocol)

— = not measured by us (see gaps below). archex has two rows: its default retrieval mode (BM25+graph, no embeddings) and its optional vector/hybrid mode (FastEmbed/ONNX + graph) — both are now measured, see #1.

The File-level column mixes several different metrics (Acc@10 / Top-1 / file-match / Agentless-metric FILE) and is not comparable straight down the column — each row names its own. The two metrics in roust's cell differ in both directions: Acc@10 counts an instance correct if any gold file appears in the top 10, while the Agentless-metric FILE score counts it correct only if all gold files appear anywhere in the returned set (~35 files for roust, range 22–38, measured from lab/results_regions/full300_v11.jsonl) — stricter on completeness, looser on depth, so neither subsumes the other. File@10 83.3 (all gold files within the top 10 — the FROZEN v7 ablation row of lab/README.md measured 82.7 = 248/300, and the adopted trace-frame boost adds +2 gains / 0 losses over the 46 trace-bearing instances, remeasured in lab/research/wave5/e20-e11b-results.md; all other instances are byte-identical) is the depth-aligned number to rank roust against the Acc@10 rows — and on that aligned metric roust sits below the trained retrievers, including SweRankEmbed-Small's 90.9; the comparison is conservative, since File@10 demands all gold files in the top 10 where Acc@10 needs one. The 92.3 all-gold figure is the one whose FUNCTION/LINE companions follow: roust's Agentless-metric scores on Lite are FILE 92.3% / FUNCTION 54.7% (exact) / LINE 43.3% (lab/results_regions/agentless_metric_e20_traceboost.json) — training-free roust now exceeds Agentless GPT-4o at function level (54.7 vs 52.0) and line level (43.3 vs 35.3), closing what was this table's weakest cell; Agentless (GPT-4o) for comparison is 69.7 / 52.0 / 35.3; archex (BM25 default) is 56.0 / 38.3 / 25.7 (lab/results_regions/agentless_metric_archex_bm25.json; 2 of 300 instances timed out — they count as wrong at FILE and LINE but are excluded from the FUNCTION denominator in that artifact, a baseline-favorable convention: 38.3 = 114/298, counting them wrong would give 38.0); archex (vector/hybrid) is 57.3 / 40.7 / 27.7 (lab/results_regions/agentless_metric_archex_vector.json, same 2 timeouts and convention, plus one git-show exclusion at FUNCTION) — a single-digit gain over BM25 that leaves the ~35-point FILE gap to roust unchanged. LINE mean-fraction-covered (a continuity metric with prior reporting, distinct from the strict all-or-nothing LINE % above) rose 0.4564 → 0.5168 → 0.5251 across the same changes. Region precision (gold lines returned / total lines returned, i.e. "how much of the packed context is actually the fix") rose from 0.4486% to 0.5522% mean (+23% relative) — roust still trades precision for recall by design, packing ~1,123 lines of surrounding context per instance under the 8192-token budget (down slightly from ~1,150 pre-adoption). These gains are the additive stack of three measured changes from the #4 campaign (autopsy-driven), now the shipped engine defaults: guarded span padding (--pad-lines, default 5), sub-linear length normalization (--len-exp, default 0.85), and the trace-frame FILE boost (E11b, PR #52: files named in a traceback in the query get a rank-decayed file-score boost, raise-site first, query text untouched; Verified held-out confirmed non-negative in every cell — FILE 92.14→92.38, LINE 35.38→35.63; --no-trace-boost disables) — run roust --help for the exact flags that reproduce the pre-adoption engine, or see #4.

The Multi-SWE JS/TS row is roust's first non-Python scoreboard entry (every other roust cell above is Python SWE-bench Lite/Verified): on the 580-instance Multi-SWE-bench JS/TS slice, FILE 46.4 (269/580) / FUNCTION 31.0 (exact) / LINE 13.3 / LINE mean-fraction .258 (lab/results_regions/agentless_metric_mswe_e23_tsblocks.json), measured with the now-default tree-sitter structural blocks for .js/.jsx/.ts/.tsx (E23, PR #55 — step one of the language-agnostic campaign, #56). Two corrections against prior reporting: (1) the previously published MSWE FUNCTION 99.83 is retired as vacuous — the gold-function scorer was Python-AST-only, so every JS/TS instance had n_gold_functions: 0 and passed the subset condition vacuously; with the fixed tree-sitter scorer the true pre-adoption baseline is 21.2 (lab/results_regions/agentless_metric_mswe_e23_baseline.json), lifted to 31.0 by the structural blocks (+68/−11 paired, p=3.5e-11). (2) FILE 46.4 sits under a measured ~76.7 ceiling: 135/580 instances have at least one gold file outside the indexed extension set (.json — 316 gold files, .md — 158, .svelte, .mjs, …), so no ranking change can lift FILE past ~76.7 on this corpus walk — universal indexing is workstream 1 of #56.

Multi-language localization (Agentless metric, all levels)

roust's per-language scoreboard across all eight benchmarked language slices — Python (SWE-bench Lite + held-out Verified) and the seven Multi-SWE-bench languages (#56 campaign; JS/TS via E23/PR #55, Java/Go/Rust/C/C++ via the WS2 grammar batch, PR #60). Every FUNCTION number is from the corrected language-aware scorer (the Python-AST-only scorer's vacuous non-Python FUNCTION numbers are retired — see lab/research/langagnostic/ws2-grammar-batch.md). All rows are the shipped engine defaults: since WS2c (lab/research/langagnostic/ws2c-vendor-guard.md) C-family indexing is default-ON behind a vendored-C guard, so the C and C++ rows no longer need an opt-in flag (see note below the table).

Notes: (1) The two Python rows are the current post-WS2c defaults (C-family indexing ON behind the vendored-C guard). Relative to the WS2b references they move by exactly two single instances, both itemized in lab/research/langagnostic/ws2c-vendor-guard.md: Lite LINE 43.67→44.00 (one gain) and Verified LINE 35.38→35.14 (one loss — two gold lines on astropy-14508, from the guard excluding astropy's vendored extern/ Python, not from C indexing; both sign tests p=1). The Verified row had already retired the stale pre-WS2b 35.63/.478 reference. (2) .c/.h/.cc/.cpp/.cxx/.hpp/.hh are indexed by default since WS2c; --no-cfamily-ext reverts to the pre-WS2c walk (C/C++ rows become FILE 0 — nothing indexable). The WS2b gate had deferred the flip after vendored libsvm displaced gold on one Lite instance; the WS2c VENDOR_RE guard (cextern/, extern/, libsvm/, liblinear/ path components) cured exactly that instance and left the MSWE C/C++ arms payload-identical (0/257 diffs). (3) Cross-language FILE differences are dominated by corpus shape (e.g. JS/TS's ~76.7 extension ceiling above, Go's single-repo skew — cli/cli is 397 of 428 instances); compare within a row's own slice, not down the column. (4) WS3b (lab/research/langagnostic/ws3b-trace-formats.md, PR #66): the Java FUNCTION cell (33.59→34.38, +1/−0) comes from the now-default multi-format trace-frame boost (Java/Node/Go/Rust frame parsing; Python byte-identical, 91/91 proven); the C++ row moves to the fresh baseline under the unconditional thirdparty vendor guard (65.89/18.60/7.75/.297 → 65.12/17.83/6.98/.295 — all 54 changed instances are nlohmann, whose checkouts vendor Google Benchmark under benchmarks/thirdparty/; no thirdparty file was ever packed by either engine, the shift is BM25 index-statistics reshuffle, itemized in the WS3b doc); the C row reproduces its prior reference digit-exact under the same fix. (5) WS3c (lab/research/langagnostic/ws3c-symbols.md, PR #67, adopted 2026-08-26 under the standing language-agnostic directive): the def/anchor channel is now structural for every grammar-covered language (tree-sitter-sourced def_index + anchor-forced region seating un-gated from .py). JS/TS, Java, and Rust rows move to the WS3c arms; the superseded post-WS3b jsts base was 46.21/30.86/13.45/.258 (itself a restatement of the pre-WS3b 46.38/31.03/13.28/.258 reference after the WS3b default flip's two documented jsts instance moves). Rust caveat, stated inline: FILE/fraction gain (+1/-0 FILE) but FUNCTION 20.50→19.67 (+0/−2 — two displacement losses where a new non-gold anchor squeezed the gold region's budget, itemized in the WS3c doc). Python rows are unchanged: all four metrics digit-identical per instance on Lite and Verified under the new default (zero FUNCTION flips; 79 instances repack non-gold content only). (6) WS3d (lab/research/langagnostic/ws3d-displacement-guard.md, PR #68, adopted 2026-08-26 under the standing directive): the JS/TS LINE/fraction cells (13.97→14.14, .260→.262; FILE/FUNCTION invariant with zero flips) come from the now-default fixture-dir anchor displacement guard — files under *.test//*.spec/ DIRECTORY components (the jscodeshift codemod fixture convention) no longer compete for symbol anchors; --no-displacement-guard reverts. Every other row is proven untouched: java/rust have zero fixture-dir paths in any evaluated tree (per-instance git ls-tree census), and the entire Lite/Verified exposure (31 pytest instances, all carrying the single path extra/setup-py.test/setup.py) is byte-identical under the guard. The general anchor/trace displacement guard the WS3c note queued was investigated and closed NO-GO by fire-level mining (culprit fires are shape-identical to the adoption wins' gold fires; see the WS3d doc): the rust FUNCTION caveat and the svelte-11104/jackson-4219-class losses remain live, with the consequence-side mechanisms named for future work.

Latency (measured, lab/latency/latency_v1.json)

Cold index (median of 3, .roust/ removed each time), warm index (median of 5, cache hit), and query time (p50/p95 of 20 queries cycling 10 problem-statement-like phrases, warm cache) — index_ms/query_ms from --json output, plus end-to-end subprocess wall time, the number that matches what an agent actually experiences (#15):

Measured on an Apple M3 Max (arm64), engine roust 0.2.0 (418212b, clean). Roughly a third of the wall-clock time at this repo size is fixed subprocess startup overhead, not indexing or query work — visible as the gap between index_ms/query_ms and the wall-time column above. Full samples, machine info, and per-repo files_indexed/disk-size in lab/latency/latency_v1.json; methodology in lab/latency/bench_latency.py.

Competitor latency: archex (BM25 default mode) query wall time on the SWE-bench Lite corpora, lab/results_regions/archex300_bm25_v1.jsonl — index mean 5.69s, query median 9.68s (2 of 300 queries hit the 300s timeout); archex (vector/hybrid mode), lab/results_regions/archex300_vector_v1.jsonl — index mean 0.92s, query median 12.98s (same 2 timeouts), worse than BM25 despite the faster index; vs roust's 0.1–0.4s wall time above on comparable repos (#1).

Historical note: an earlier claim (never backed by a committed artifact) compared the (now-deleted) Python engine against the Rust port directly — "Rust 3.6–4.2× faster than Python engine (httpx 145ms vs 522ms, django 1.8s vs 7.6s)". The Python engine was removed in #12, so that comparison is no longer reproducible; it's kept here only as a historical data point, not a current claim.

ContextBench (human-annotated gold context, their evaluator)

ContextBench (arXiv:2602.05892) scores retrieved context against human-annotated "necessary context" line regions. roust was run one-shot (--json --budget 8192, single call, no model) on the Python subset of their curated 500-instance Verified benchmark (266 tasks, 19 repos, 266/266 evaluated, 0 skipped) and scored with ContextBench's own evaluator, unmodified (#3):

Protocols differ and the comparison is not apples-to-apples: the published baselines are multi-turn LLM agents (read, navigate, then select context) on the full 500-task 8-language set; roust is a single sub-2-second call with no model, no API key, and no training, on the Python 266. Read it as: one free one-shot call recovers ~94% of the file-level recall of the best agent, and its precision is ~10x lower because roust deliberately packs a full 8192-token recall-first bundle rather than a minimal answer — the same recall-over-precision trade documented in #4. ContextBench's efficiency metrics (AUC-Coverage/Redundancy) are N/A for a one-step trajectory. Adapter + protocol: lab/contextbench/; aggregate: lab/contextbench/results_python.json.

What still needs work

• Line-level 35.7% and function-level 44.3% (a proxy, not the exact metric) measured exactly (lab/results_regions/agentless_metric_v2.json): FUNCTION 39.7% (exact, was a 44.3% proxy) and LINE 29.3% (was 35.7%) from a fresh 300-instance run of the shipped engine; a w_name sweep on the exact harness (#4) then showed the symbol-name weighting itself caused the LINE drop — reverting it (w_name=0.0) restores FUNCTION 41.0% (exact) and LINE 35.7% (lab/results_regions/agentless_metric_v3.json). FUNCTION is still the weakest cell vs Agentless GPT-4o's 52.0 closed (#4): the campaign's autopsy on the FUNCTION/LINE misses found the padding/length-normalization mechanism (comboA — guarded span padding + sub-linear length normalization) and it's now adopted as the shipped engine defaults (--pad-lines 5 --len-exp 0.85), raising FUNCTION 41.0→53.3% and LINE 35.7→42.7% (fraction 0.4564→0.5168) — roust now exceeds Agentless GPT-4o at both FUNCTION (53.3 vs 52.0) and LINE (42.7 vs 35.3) (lab/results_regions/agentless_metric_v5.json); the region-packing gains REPLICATED out-of-sample on the 407-instance held-out SWE-bench Verified set, never used for any tuning decision (commit 2f7d324, lab/results_regions/agentless_metric_verified_{old,new}.json): FUNCTION +12.9pp (34.2→47.0%), LINE +9.1pp (26.3→35.4%), fraction +0.053, FILE unchanged (lab/results_regions/agentless_metric_verified_{old,new}.json)
• archex has never been measured by us on any of our benches both Agentless-metric arms measured (#1): archex 0.19.2 BM25 default mode is FILE 56.0 / FUNCTION 38.3 / LINE 25.7 (lab/results_regions/agentless_metric_archex_bm25.json); vector/hybrid mode is FILE 57.3 / FUNCTION 40.7 / LINE 27.7 (lab/results_regions/agentless_metric_archex_vector.json), a single-digit gain over BM25 with worse latency (12.98s vs 9.68s query median) that leaves the ~35-point FILE gap to roust unchanged — steelman complete; the tokenbench agent-loop arm is not justified at current quality
• True cost-per-success measured via repeat runs (#16): roust solves 14/15 deterministically at ~$1/answer with one real capability gap (django-16400, 0/10); embedding-RAG reaches everything eventually at a median $2.42 (mean $4.90) per first success — see results_repeats.jsonl and the aggregation convention stated in the scoreboard notes above
• Latency has no committed benchmark artifact measured (#15): cold/warm index + query p50/p95 across four repo sizes (66–2,131 files indexed), lab/latency/latency_v1.json

How these were measured

Agent-loop outcomes is our agent-loop harness (live Sonnet 4.5, measured to task completion; the grep and roust arms use their method as the agent's only search tool, while the embedding-RAG arm had grep alongside rag_search, and every arm has a read_file tool) — a partial run, 58 of 120 planned pairs, stopped at an $80 spend cap, so n=15 (14 for embedding-RAG). Localization accuracy is published Acc@k-style numbers from each system's own paper, on its own harness — a different protocol, not comparable to the agent-loop numbers. Full artifacts and the research log (including the retracted "95% fewer tokens than grep" claim, which came from a v1 one-shot protocol and does not hold in the agent loop — #6) are in lab/README.md.

Limits

• File-level, not line-level. roust localizes to files and packs regions within them; it doesn't point at a specific line or diff hunk.
• @1 precision is the measured weak spot. Top-1 file accuracy on the held-out SWE-bench Verified set is .354 -- if you need "the one file", read further down the ranked list, don't trust rank 1 alone.
• Region-level gains replicate out-of-sample (commit 2f7d324, lab/results_regions/agentless_metric_verified_{old,new}.json). The held-out SWE-bench Verified FILE numbers (79.4 File@10 / 92.1 all-gold, lab/README.md's held-out validation section) are unaffected by the guarded-padding + length-normalization adoption above -- file-selection code is untouched by padding/length-normalization, which only reshape the region spans within already-selected files, and the 300/300 file-level parity gate (parity/rust_gate_300_v5.json) confirms file ranking is unchanged. Region- level metrics (FUNCTION/LINE/fraction), previously Lite-only evidence, are now measured on the held-out set too: on the same 407 held-out Verified instances, never used for any tuning decision, FUNCTION rose 34.2%→47.0% (+12.9pp) and LINE rose 26.3%→35.4% (+9.1pp, mean-fraction-covered +0.053), FILE essentially unchanged (92.14%→91.89%, one 180s engine timeout counted as wrong in the new arm). The absolute numbers are lower than Lite's (FUNCTION 53.3%, LINE 42.7%) because held-out Verified is a harder set (lower baseline FILE accuracy, more gold hunks per instance on average) -- what needed to replicate was the delta from the padding/length-norm change, and it did: 104% of the Lite FUNCTION delta, 130% of the Lite LINE delta, 88% of the Lite fraction delta (lab/results_regions/agentless_metric_verified_{old,new}.json, parity/region_eval_verified.py).
• Natural-language issues with no identifiers are the hard class. Every non-semantic retrieval method (roust included) leans on identifiers, paths, and error strings as anchors; a vague prose description with none of those gives the pipeline little to grab onto.
• Python gets the full signal set (import graph, definition-symbol index). JS/TS/TSX now additionally get tree-sitter structural region packing by default (E23, PR #55; --no-structural-blocks restores the old fixed windows). Other languages (Go, Rust, Java, etc.) still get a best-effort subset -- lexical/BM25F, paths, and history apply, but fixed-window packing and no import-graph or def-index expansion. Closing this gap across languages is the language-agnostic campaign, #56.

Roadmap

• Rust port complete and shipped as the only engine: roust-rs/ was brought to feature-parity with the (now-deleted) Python v0.2 engine (channel-aware packing, on-disk cache with incremental updates, deterministic seed) — bundle-level parity gate PASSED 300/300 exact on SWE-bench Lite (report in parity/bundle_parity_300.json: 300 EXACT, 0 region-level differences; parity/rust_gate_300_v3.json is the file-ranking-only gate) before the Python engine was removed. Measured absolute latency (cold/warm index, query p50/p95) is in the Scoreboard's Latency block above (lab/latency/latency_v1.json, #15); the old cold-index Rust-vs-Python ratio is no longer reproducible and is kept there only as a historical note. Build from source: cd roust-rs && cargo build --release.
• Language-agnostic roust (#56, user-directed campaign): universal indexing (binary sniffing + size caps instead of the extension allowlist — lifts the Multi-SWE FILE ceiling from ~76.7 toward ~100), the grammar batch (Java/Go/Rust/C/C++ via the E23 mechanism, gated per-language on Multi-SWE slices), and a Python-assumption audit (tokenizer, test-path heuristics, history mining, query construction) with Multi-SWE slices as first-class gates. Step one (JS/TS/TSX structural packing) shipped in PR #55.
• First release shipped: v0.3.0 is on npm (roust-cli plus five platform binary packages), crates.io (roust), and GitHub Releases, with docs at https://narehart.github.io/roust/ — see RELEASE.md.
• MCP server.
• Incremental index updates (avoid full reindex on every change).
• Homebrew tap.

Research artifacts -- benchmark JSONLs, diagnostics, and pre-registered held-out predictions -- live in lab/. lab/ is a frozen Python research sandbox (including lab/lanes2.py, the oracle the parity gates were built against) -- it is never the source of truth for shipped behavior, which is roust-rs/ end to end.

License: MIT.

History

Formerly bgrep; renamed to avoid collision with the binary-grep tool of that name.