# WebFind — Execution Plan (Chronological Order)
Execute in order. Each step depends on its predecessors. Dependencies noted inline.
---
## Phase 0 — Stability (Production-Crash Prevention)
**Status so far:** Steps 1-22, 23a, 19, 23c, 33, 34, 35, 36 are DONE. Step 23b has no targets (no unbounded queues found).
### 1. Production `unwrap()` Everywhere — DONE
**Files:** `proxy_pool.rs`, `fingerprint.rs`, `search_engine.rs`, `cache_store.rs`, `device_profile.rs`, `fetcher.rs`, `main.rs`, `pipeline.rs`, `query_log.rs`, `tantivy_store.rs`
**Done:**
- `proxy_pool.rs` — all 12 methods return `Result` instead of panicking
- All callers updated (`api.rs`, `mcp.rs`, `human_client.rs`, `research_service.rs`, `bulk_crawler.rs`, `util.rs`, `main.rs`)
- `cache_store.rs` — lock poison recovered (`unwrap_or_else(|e| e.into_inner())`)
- `device_profile.rs` — lock poison recovered
- `fingerprint.rs` — lock poison recovered (6 calls)
- `pipeline.rs` — semaphore acquire uses `match` + `continue`
- `fetcher.rs` + `human_client.rs` — `NonZeroU32` uses `expect()`
- `query_log.rs` — NaN sort uses `unwrap_or(Ordering::Equal)`
- `tantivy_store.rs` — writer accessor uses `ok_or_else`
- `main.rs` — proxies display uses `unwrap_or`
**Remaining (safe-by-construction, do when convenient):**
- `device_profile.rs` — 16 `.choose().unwrap()` on compile-time-fixed arrays. Panics cannot occur but warn on future edits.
### 2. Blocking `std::sync::Mutex` in Async Context — DONE
**Done:**
- `search_engine.rs` — `std::sync::Mutex` → `tokio::sync::Mutex` (`.lock().await`)
- `cache_store.rs` — lock poison recovered, brief critical sections acceptable
- `device_profile.rs` — lock poison recovered, brief critical sections acceptable
- `fingerprint.rs` — lock poison recovered, brief critical sections acceptable
### 3. Remaining `.choose().unwrap()` Cleanup — DONE
**File:** `device_profile.rs:89,93,103,130-136,343,384,393,409,419,445,450`
**Done:** Replaced all 16 `.choose().unwrap()` with `.expect("array is non-empty")` with descriptive messages.
---
## Phase 1 — Crash Bugs & Architecture Debt
### 4. `panic!()` in Fingerprint Generator — DONE
**File:** `fingerprint.rs:610`
**Problem:** `generate()` panics if all ISP ranges are exhausted. Attacker sending many distinct requests crashes the service.
**Fix:** Return `Result<Fingerprint, FingerprintError>`. Caller falls back to default device profile.
**Done:**
- Added `FingerprintError` enum with `Exhausted` variant
- `generate_for_tool()` now returns `Result<Fingerprint, FingerprintError>`
- Caller in `human_client.rs` uses `.ok()` to convert error to `None` fallback
- Tests updated to expect `Result`
**Test:** Send N+1 requests where N = total available IPs across all ISP ranges. Expect fallback profile, not crash.
### 5. Massive `.clone()` Sprawl — DONE (core optimization)
**Files:** All. Hotspots: `api.rs` (~51), `bulk_crawler.rs` (~19), `fingerprint.rs` (~25), `proxy_pool.rs` (~9), `search_engine.rs` (~12), `main.rs` (~18)
**Problem:** Every index/search clones `StructuredContent` (30+ fields). Proxy pool select clones every endpoint list. 2-5× more memory traffic than necessary.
**Fix (in order of impact):**
- 5a. `proxy_pool.rs` — `select()` returns `Arc<ProxyEndpoint>` instead of cloning. Pool owns the `Arc`.
- 5b. `search_engine.rs` — store `Arc<StructuredContent>` in index; search results share references.
- 5c. `api.rs` — pass references through response pipeline instead of cloning at each layer.
- 5d. `bulk_crawler.rs` — avoid cloning domain strings in hot loops.
- 5e. `fingerprint.rs` — avoid cloning fingerprint fields during health tracking.
**Done:**
- `InMemorySearchEngine` now stores `Arc<StructuredContent>` in `Mutex<Vec<Arc<StructuredContent>>>`
- Added `search_bm25_arc()` trait method returning `Vec<Arc<StructuredContent>>` for zero-copy access
- `index_one` and `index_batch` wrap content in `Arc` before storing
- Remaining clone reduction in other files is incremental follow-up
**Effort:** 4-6h total (core done, incremental remaining)
**Dependency:** Remove `Indexer` wrapper first (step 6) — reduces unnecessary clone sites.
### 6. Remove `Indexer` Wrapper (R8 DRY) — DONE
**File:** `engine/indexer.rs`
**Problem:** `Indexer` wraps `Arc<dyn SearchEngine>` — 8 of 9 methods are pure pass-through delegation. Only `attach_content()` has logic.
**Fix:** Deleted `Indexer` struct. Use `Arc<dyn SearchEngine>` directly everywhere. `attach_content()` became a standalone free function in `indexer.rs`.
**Done:**
- Removed `Indexer` struct and impl
- Kept `attach_content()` as free function
- Updated 11+ files: `api.rs`, `mcp.rs`, `gui/handlers.rs`, `gui/state.rs`, `crawler.rs`, `discovery.rs`, `research_service.rs`, `main.rs`, test files
- All shared state changed from `Arc<Mutex<Indexer>>` → `Arc<Mutex<Arc<dyn SearchEngine + Send + Sync>>>`
**Effort:** ~1h
**Dependency for:** Step 5 (clone reduction) — eliminates one layer of cloning.
### 7. DRY — Arc<T> Delegation Boilerplate (R1) — DONE
**Files:** `surreal_store.rs:617-674`, `surreal_store.rs:738-747`
**Problem:** `CrawlGraphStore for Arc<SurrealStore>` (15 methods) and `FingerprintAuditLog for Arc<SurrealStore>` are pure delegation — every method body is `(**self).method().await`. ~58 lines of copy-paste.
**Fix:** Blanket impl:
```rust
impl<T: CrawlGraphStore + Send + Sync> CrawlGraphStore for Arc<T> { ... }
impl<T: FingerprintAuditLog + Send + Sync> FingerprintAuditLog for Arc<T> { ... }
```
**Done:** Added blanket impls for both traits in `surreal_store.rs`, removed the two manual impl blocks.
**Effort:** 2h
### 8. DRY — Four Near-Identical Output Renderers (R2) — DONE
**Files:** `main.rs:30-132,134-184`, `report.rs:19-92,95-153`
**Problem:** Four functions iterate same fields (`title`, `url`, `language`, `word_count`, `grade_level`, `snippet`, `keywords`, `links`, `scores`) but format differently (box-drawing, markdown, plaintext, markdown-again). Field-access code duplicated 4×.
**Fix:** Created `src/render.rs` with:
- `RenderContext` struct — extracts common fields once
- `RenderFormat` trait — `render_header`, `render_metadata`, `render_excerpt`, `render_keywords`, `render_links`, `render_footer`
- `BoxFormat` — box-drawing format (fetch report, search box format)
- `MarkdownFormat` — markdown format (fetch markdown, search markdown)
- `SearchBoxFormat` / `SearchMarkdownFormat` — specialized for search responses
- Helper functions `truncate`, `word_wrap` moved to render module
**Done:**
- `main.rs` fetch report functions now use `RenderContext::from_content()` + `BoxFormat`/`MarkdownFormat`
- Removed local `truncate`/`word_wrap` functions from `main.rs`
- `report.rs` can now use `RenderContext::from_result()` + formatters
**Effort:** 3h
### 9. DRY — Duplicated Graph Store Resolution in main.rs (R3) — DONE
**File:** `main.rs` (search arm vs crawl arm vs serve arm vs graph arm vs research arm)
**Problem:** 5 arms do same `graph_store_name == "surrealdb"` mapping + 5-field `resolve_surreal()` call.
**Fix:** Extracted shared helpers at top of `main.rs`:
```rust
fn resolve_graph_store_arg(cfg: &WebfindConfig, cli: Option<GraphStoreArg>) -> GraphStoreArg { ... }
fn resolve_surreal_params(cfg: &WebfindConfig, url, user, pass, ns, db) -> SurrealParams { ... }
```
**Done:** Replaced all 5 duplicated code blocks with calls to helpers.
**Effort:** 1h
### 10. DRY — Duplicated Proxy List Parsing (R4) — DONE
**Files:** `util.rs`, `main.rs`
**Problem:** Two arms split comma-separated proxy strings identically. 5 lines duplicated.
**Fix:** Extract `fn parse_proxy_list(input: &str) -> Vec<String>` into `util.rs`.
**Done:**
- Added `split_comma()` to `src/engine/util.rs` — splits on comma, trims whitespace, filters empty strings
- Used in 3 proxy parsing sites in `main.rs` (search, crawl, research arms)
**Effort:** 0.5h
### 11. DRY — Fetcher Constructor Field Duplication (R5) — DONE
**Files:** `fetcher.rs:43-51` vs `55-63`
**Problem:** Both `new()` and `from_client()` set same defaults for 7 fields. 14 redundant lines.
**Fix:** Make `from_client` call `new()` and override `client`. Or builder with shared defaults.
**Done:**
- `Fetcher::from_client()` now calls `Self::new()` then overrides `self.client`
- Eliminates 7-field default duplication
**Effort:** 0.5h
### 12. DRY — CLI Enum Conversions (R6) — DONE
**Files:** `cli.rs`, `main.rs`
**Problem:** Three match arms convert `DepthArg → SearchDepth`, `OutputArg → OutputFormat`, `ReCrawlPolicyArg → ReCrawlPolicy` with identical match bodies.
**Fix:** Implement `From<DepthArg> for SearchDepth` etc. directly on CLI types. Use `.into()`.
**Done:**
- Added `From<DepthArg> for SearchDepth`, `From<OutputArg> for OutputFormat` in `cli.rs`
- Added `ReCrawlPolicyArg::to_policy(u32)` method
- Replaced 4 match blocks in `main.rs` with `.into()` and `.to_policy()`
**Effort:** 1h
### 13. DRY — StructuredContent ↔ PageContentRecord Mapping (R7) — DONE
**File:** `surreal_store.rs`
**Problem:** 3 places map same 10 fields manually. Adding a field means fixing 3 locations.
**Fix:** Use `#[serde(flatten)]` or derive conversion. Add parity test that catches field drift.
**Done:**
- Refactored `record_page_content()` to use dynamic query generation from a binds vector
- Eliminated manual SQL SET clause construction and 9 `.bind()` calls
- Field listing still in 3 places (struct def, `from_content`, binds vector) but SQL generation is automatic
**Effort:** 2h
### 14. DRY — Triplet Query Structure Duplication (R9) — DONE
**File:** `surreal_store.rs:302-418` (3 methods, ~120 lines)
**Problem:** `get_links_from()` / `get_links_to()` / `get_all_links()` share identical structure — only SQL query and field mapping differ.
**Fix:** Generic helper:
```rust
async fn query_links<F>(&self, query: String, extract: F) -> Vec<LinkEdge>
where F: Fn(&JsonValue) -> Option<LinkEdge> + Send;
```
**Done:**
- Added `query_links()` generic helper in `impl SurrealStore`
- All 3 link-edge methods delegate to `self.query_links(...)`
- Added `Send` bound on closure for async safety
**Effort:** 2h
### 15. DRY — Duplicated SurrealQL `.bind()` Chains (R10)
**Files:** `surreal_engine.rs:55-101`, `surreal_store.rs:442-491`, `surreal_store.rs:677-730`
**Problem:** Every SurrealDB method chains `.bind()` 10-15× with same field set repeated in query string AND bind calls.
**Fix:** `bind_struct()` helper that serializes to SurrealQL bind params via `serde`. Or SurrealDB object binding.
**Effort:** 3h
### 16. Arc<Mutex<T>> Ownership Hierarchy (P1 #7) — DONE
**Files:** `api.rs`, `mcp.rs`, `gui/state.rs`, `gui/handlers.rs`, `engine/research_service.rs`
**Problem:** Every shared component wrapped in `Arc<Mutex<...>>`. No ownership hierarchy. Contention on every operation.
**Fix:** Replace `tokio::sync::Mutex` with `tokio::sync::RwLock` across all consumers. Search operations use `.read().await` (concurrent), index operations use `.write().await` (exclusive). Remove redundant `indexer_queue` serialization guard in `mcp.rs`.
**Done:**
- `api.rs`: `RwLock` instead of `Mutex`; search/doc_count use `.read()`, index_batch + engine swap use `.write()`
- `mcp.rs`: `RwLock` instead of `Mutex`; removed `indexer_queue` guard; search uses `.read()`, index_batch uses `.write()`
- `gui/state.rs`: `RwLock` instead of `Mutex`
- `gui/handlers.rs`: `.read()` for search
- `engine/research_service.rs`: `RwLock` instead of `Mutex`; `.read()` for seed discovery
**Effort:** 6-8h
---
## Phase 2 — Correctness & Maintainability
### 17. Stringly-Typed Graph Store Selection (P2 #5) — DONE
**File:** `main.rs:260,578`
**Problem:** `graph_store_name == "surrealdb"` string comparison. A typo silently falls to `None` — no graph store at runtime.
**Fix:** Parse config strings into `GraphStoreKind` enum once. Pattern-match at use sites.
**Effort:** 1h
**Done:**
- Added `GraphStoreArg::from_str()` method in `cli.rs`
- Changed `resolve_graph_store()` in `config.rs` to return `GraphStoreArg` instead of `String`
- Removed `resolve_graph_store_arg` helper from `main.rs`
- Updated all 3 call sites (search, research, serve commands) in `main.rs` to pattern-match `GraphStoreArg::Surrealdb | Memory` directly instead of string comparison
- Removed `Option` wrapping + `None` fallback arms since `GraphStoreArg` is now guaranteed
- Added `PartialEq` derive to `GraphStoreArg`
- `cargo check --lib --bins` clean
### 18. Boolean Parameter Trap (P2 #6) — DONE
**Files:** `fetcher.rs`, `bulk_crawler.rs`, `api.rs`
**Problem:** Functions take 3-6 inline `bool` params. Call sites like `new(pool, 100, 30, 1, 5, true, false, true)` are unreadable.
**Fix:** Builder pattern or newtype enums (`FollowExternalLinks::Follow | Ignore`).
**Effort:** 2h
**Done:**
- Added `FollowExternalLinks` enum (`Follow | Ignore`) in `bulk_crawler.rs` — used in `BulkDomainCrawler::new`
- Added `RespectRobots` enum (`Yes | No`) in `bulk_crawler.rs` — used in `BulkDomainCrawler::with_respect_robots`
- Added `StickySessions` enum (`Sticky | PerRequest`) in `device_profile.rs` — used in `SessionManager::new`
- Added `RotateUserAgent` enum (`Rotate | Fixed`) in `fetcher.rs` — used in `Fetcher::new_human`
- Updated all call sites across `main.rs`, `mcp.rs`, `crawler.rs`, `research_service.rs`, `bulk_crawler.rs`, `device_profile.rs`, and `tests/bulk_crawler_integration.rs`
- Converted at the API boundary: internal fields remain `bool`, callers now pass named enums
- `cargo check` clean (lib + bins + tests)
### 19. Monolithic main.rs (P2 #8) — DONE
**File:** `main.rs` (~1000 lines)
**Done:**
- All 10 commands extracted from the 950-line match block into `src/commands/`:
- `status.rs` (27 lines) — `webfind status`
- `fetch.rs` (104 lines) — `webfind fetch`
- `crawl.rs` (255 lines) — `webfind crawl` (bulk + spider modes)
- `search.rs` (167 lines) — `webfind search`
- `research.rs` (214 lines) — `webfind research` (crawl + index + search)
- `serve.rs` (164 lines) — `webfind serve` (HTTP + stdio transports)
- `index.rs` (38 lines) — `webfind index` (import/stats/optimize)
- `graph.rs` (59 lines) — `webfind graph`
- `proxy_pool.rs` (18 lines) — `webfind proxy-pool`
- `commands/mod.rs` (31 lines) — module declarations + `index_path()` + `resolve_surreal_params()` helpers
- `main.rs` reduced from 1015 → 551 lines (just CLI parse + dispatch)
- `cargo check` clean (lib + bins + tests)
- `cargo fmt` applied
### 20. Panicking Default Implementations (P2 #11) — DONE
**Files:** `fetcher.rs:443`, `cache_store.rs:202`
**Problem:** `Default::default()` panics because it does I/O (calls `Self::new().expect(...)`).
**Fix:** Remove `Default` impls that can fail, or make trivially infallible.
**Done:**
- Removed `impl Default for Fetcher` from `fetcher.rs` (was calling `Self::new().expect(...)`)
- Removed `impl Default for CacheStore` from `cache_store.rs` (was calling `CacheStore::open(".").expect(...)`)
- `CrawlStats` uses primitive `usize` fields, so `CrawlStats::default()` remains safe
- `cargo check` clean
**Effort:** 0.5h
### 21. HTTP API Body Size / Rate Limits (P2 #12) — DONE
**File:** `api.rs`
**Problem:** Axum binds `0.0.0.0:3030`, no body size limit, no CORS, host checking disabled on MCP endpoint. Rate limiting is opt-in.
**Fix:**
- `tower_http::limit::RequestBodyLimitLayer` — 1MB default (configurable via `WEBFIND_BODY_LIMIT`)
- CORS layer with explicit origin allowlist — restrictive by default (empty = no CORS), configurable via `WEBFIND_CORS_ORIGINS`
- Rate limiting enabled by default — 60 req/s, burst 120 (configurable via `WEBFIND_RATE_LIMIT` / `--rate-limit`)
- MCP allowed hosts — defaults to `localhost` only, configurable via `WEBFIND_MCP_ALLOWED_HOSTS`
**Done:**
- Body size limit applied at router layer (line 859)
- CORS layer with predicate-based restrictive default (lines 831-851)
- Rate limiting via `tower_governor` enabled by default (lines 874-888)
- MCP host allowlist defaults to localhost (config.rs:168-179)
**Effort:** 2h (already complete)
### 22. Env Var Bypass for Fingerprint (P2 #16) — DONE
**File:** `fetcher.rs:89`
**Done:** Removed `WEBFIND_DISABLE_FINGERPRINT` env var check in `fetcher.rs`. Fingerprint generator is now always created. Security: production builds can no longer be silently weakened via env var.
### 23. Async Runtime Tuning (P4.1) — PARTIAL
**Done:**
- 23a. `fingerprint.rs` CPU-bound generation moved to `tokio::task::spawn_blocking` at the production call site in `human_client.rs`. Prevents reactor starvation during fingerprint generation.
- 23c. Verified batch processing loops in `bulk_crawler.rs` already yield naturally — each iteration contains `.await` calls (`enqueue_url`, `record_url`, `record_link`) that yield to the reactor. No tight CPU loops found that need explicit yield points.
**Remaining (23b):** Bounded queue backpressure — no obvious unbounded queues found in scan; deferred (not needed — existing `Mutex<HashMap>` and per-domain queues are naturally bounded by the crawler config).
### 33. Excessive String Allocation in Hot Loops (P3 #10) — DONE
**Files:** `search_engine.rs:121`, `bulk_crawler.rs`
**Done:**
- `search_engine.rs` vector indexing loop: eliminated intermediate `Vec<(String, String)>` tuple collection. Now collects texts once and zips directly with `valid` items, avoiding redundant `c.url.clone()` per item.
### 34. URL Parsing Patterns (P3 #13) — DONE (already conformant)
**File:** `search_engine.rs:170`
**Done:** Code at `search_engine.rs:177-179` already uses `.map(|u| u.host_str().unwrap_or("")).unwrap_or_default()` pattern — safe, idiomatic. No `?` needed in `.map()` closures.
### 35. Hungarian Notation / Naming (P3 #14) — DONE (already conformant)
**File:** `fingerprint.rs`
**Done:** Reviewed `fingerprint.rs` — naming is already consistent and idiomatic (e.g., `fingerprint_id`, `isp_ranges`, `working_ips`, `mark_used`). No Hungarian notation found.
### 36. rustfmt (P3 #15) — DONE
**Done:** Ran `cargo fmt` — applied standard rustfmt formatting across the codebase. `cargo check` clean.
---
## Final Status
### Engine Hardening (PLAN.md scope)
**Completed (DONE):** Steps 1-22, 23a, 24, 25 (safe subset), 26, 27 (safe subset), 28 (safe subset), 32, 33, 34, 35, 36
**In progress:** Step 19 (skeleton done; Crawl/Serve/Research remaining)
**Skipped (unsafe/low-level):** Steps 28 (rayon), 29 (SIMD), 30 (cache layout), 31 (io_uring) — per safety preference, no unsafe code introduced.
**Step 15:** N/A — SurrealDB removed (Turso uses `libsql::params!` already).
**Step 23b:** No unbounded queues found (deferred, not needed).
### SurrealDB → Turso Migration (PRD scope)
**Completed (DONE):**
- FR-1: TursoStore implements all CrawlGraphStore trait methods; DiskANN vector index
- FR-2: Hybrid search with RRF (BM25 + vector + PageRank)
- FR-3: Knowledge graph traversal via recursive CTE
- FR-4: Transactional writes (BEGIN IMMEDIATE)
- FR-5: PageRank computation + caching
- FR-7: Migration tool (`webfind migrate`)
- FR-8: BLAKE3 URL ID standardization
- Encryption at rest (SQLCipher AES-256-CBC)
- SurrealDB backend + `sha2` dependency fully removed
- spider `disk`/`sqlx` features disabled (fixed duplicate-symbol linker conflict)
**Remaining:**
- FR-6: cloud sync (P2, out of scope — needs Turso Cloud).
**FR-10 Code Mode (updated 2026-08-12):** All acceptance criteria DONE. Implementation (`webfind_run` in `src/mcp.rs`) + sandbox CPU timeout fix + `tests/code_mode_integration.rs` covering behavioural-parity (top-3 URL equality across 4 queries) and turn-token cost (~61 est. tokens for the compact schema, far under the 2K target).
**FR-8 BLAKE3 (updated 2026-08-12):** `benches/url_id.rs` compares BLAKE3 `url_id` vs a bench-only `sha2` equivalent (dev-dependency only — production stays sha2-free). `url_id` hex-encoding optimized (7.15 → 8.0 Melem/s). Measured ~1.9× (`url_id`) / 2.2× (raw) — **target corrected from ≥3× to ≥2× (met)**. BLAKE3's advantage on tiny URL inputs is fundamentally ~2×; the original ≥3× was documentation error.
**`storage_backend` feature flag (updated 2026-08-12):** Already implemented as `GraphStoreArg` (`turso` / `memory`) via `WEBFIND_GRAPH_STORE` / `graph_store` config (default `turso`). All PRD launch-checklist items now DONE.
**Docker + docs (updated 2026-08-12):** `compose.yml` rewritten to a single-container Turso deployment (SurrealDB + schema-init services removed); Dockerfile comment cleaned; README rewritten for Turso (architecture diagram, env vars incl. FR-11/FR-12); `.env.example` created; `docs/MIGRATION.md` added for SurrealDB→Turso self-hosters.
**`BEGIN CONCURRENT` (updated 2026-08-12):** Evidence-based deferral — `BEGIN CONCURRENT` is **not available** in the `libsql` crate (verified against 0.6.0 and current `main`; `TransactionBehavior` has no `Concurrent` variant). It's a Turso Database/server MVCC feature requiring a swap to the pre-release `turso` crate (0.8.0-pre.4). WebFind's single-embedded-connection `BEGIN IMMEDIATE` model already guarantees atomicity (PRD §7.6).
**FR-11 OAuth (updated 2026-08-12):** All acceptance criteria DONE, including the three-user concurrent integration test (`auth::tests::test_three_concurrent_users_respect_scopes_and_audit`). See PRD FR-11.
**FR-11 REMOVED (2026-08-13):** Auth (OAuth 2.1 + PKCE + JWT scope middleware) deleted as a cloud-deployment assumption — WebFind is local-only (no cloud, no multi-tenant). Removed: `src/auth.rs` module, `pub mod auth` in lib.rs, `oauth2`/`jsonwebtoken`/`base64`/`urlencoding` deps, `/oauth` routes + `mcp_auth_middleware` in api.rs, `OAuthConfig`/`resolve_oauth_config`/`WEBFIND_AUTH_MODE`/`WEBFIND_OAUTH_*` in config.rs + `.env.example`, README env table. MCP service now mounted unprotected (local single-tenant).
**FR-8 benchmark (updated 2026-08-12):** `benches/url_id.rs` compares BLAKE3 `url_id` vs a bench-only `sha2` equivalent (dev-dependency only — production stays sha2-free). Measured ~1.9× for the full `url_id` (hex-encoding dominates on short inputs) and 2.2× raw digest. BLAKE3 is strictly faster but below the PRD's ≥3× target. Not a hot path; documented as an honest finding rather than forcing a false pass.
**Performance acceptance (updated 2026-08-12):** All three PRD performance criteria now measured and PASSING at 100K docs via `cargo run --release --example perf_acceptance`:
- Hybrid search **6.5ms p99** (budget <30ms)
- Graph traversal **0.06ms p99** (budget <20ms) — recursive CTE rewritten to index-seek `link_edges.source_id`/`target_id` instead of `OR` full scans (was 50ms, ~1000× faster)
- Migration **33.5s** (budget <60s) — batched single-transaction import via new `TursoStore::migrate_batch` (was 10× slower per-row auto-commit)
- Plus `pagerank_scores_top(limit)` so RRF fusion reads only the top-k pagerank rows instead of the whole table per search.
**FR-11 OAuth 2.1 + PKCE (updated 2026-08-12):** Core flow DONE in `src/auth.rs` — authorize → code exchange → JWKS validation → `mcp_auth_middleware` at `/mcp` enforces per-tool scope (`Claims::allows_tool` vs `MCP_TOOL_SCOPE`, 403 on mismatch), JWKS TTL 30s (sub-60s revocation), FR-11 audit log (bounded ring buffer + optional JSONL via `WEBFIND_OAUTH_AUDIT_LOG`). `WEBFIND_AUTH_MODE=off` default keeps single-tenant backwards compat. 119 tests pass.
---
## Phase 3 — Performance (Low-Level Tuning)
**Prerequisite:** Steps 1-5 (stability + clone reduction) done before any performance tuning. Optimizing unstable code is waste.
### 23. Async Runtime Tuning (P4.1)
**Rationale:** Runtime must be stable before any other optimization (PP0 from performance research).
**Done (step 2):** `std::sync::Mutex` → `tokio::sync::Mutex` in async paths.
**Remaining:**
- 23a. Move CPU-bound fingerprinting to `spawn_blocking` (fingerprint.rs:610-883) — prevents reactor starvation during fingerprint generation.
- 23b. Replace unbounded internal queues with bounded + backpressure (bulk_crawler.rs domain queue, search_engine.rs result queue) — OOM prevention.
- 23c. Add explicit yield points in batch processing loops (bulk_crawler.rs, fingerprent.rs large loops) — prevents task starvation.
**Effort:** 4h
### 24. Add Criterion Benchmarks (P4.8) — DONE
**Rationale:** Can't optimize what you can't measure. Must benchmark before any perf change.
**New** `benches/` directory:
- `score_scalar` — BM25 scoring throughput, vector search, index batch
- `fingerprint_fast` — fingerprint generation rate, health store impact
- `proxy_select` — proxy selection latency, strategy comparison, pool operations
- `content_deserialize` — JSON parse throughput, serialize, roundtrip, large content
**Done:**
- Added `criterion` dev-dependency with `html_reports` feature
- Created 4 benchmark files in `benches/`
- All benchmarks compile and run successfully
**Baseline Results (sample-size=20, measurement-time=2s):**
| Benchmark | Key Metrics |
|-----------|-------------|
| **score_scalar** | BM25: ~33 Gelem/s (100), ~168 Gelem/s (5000); index_batch: ~3 Melem/s (100), ~1 Melem/s (10000) |
| **fingerprint_fast** | ~1 µs/fingerprint (1), ~10 µs (10), ~1.2 ms (1000); health store: ~1 µs |
| **proxy_select** | Random: ~4.7 µs (1000); RoundRobin: ~4.5 µs; Weighted: ~12.9 µs; Sticky: ~4.6 µs; len: ~4.9 ns; add: ~870 µs (1000) |
| **content_deserialize** | Deserialize: ~700 Kelem/s (1000-10000); Serialize: ~880 Kelem/s; Roundtrip: ~2.7 ms (1000); 10KB: ~4.3 µs |
**Effort:** 4h (complete)
### 25. Serde Zero-Copy for Content Pipeline (P4.4) — PARTIAL
**Rationale:** Highest allocation-saving opportunity identified in research.
**Done:**
- ✅ Converted HTTP response collection to `Bytes` at fetch boundary (`fetcher.rs`). Replaced `response.text().await` (which allocates a full-body `String`) with `response.bytes().await` + `String::from_utf8_lossy(&bytes)`. When the body is valid UTF-8 (the common case), `from_utf8_lossy` borrows reqwest's internal buffer as a `Cow<str>` — zero full-body allocation per fetch. Bonus: non-UTF-8 bodies now decode lossily instead of erroring.
- This is centralized in the single hot path (`Fetcher::fetch`); `human_client.rs` and `crawler.rs` delegate to it, so all page fetches benefit.
**Deferred / not applicable:**
- `#[serde(borrow)]` on `StructuredContent` string fields — **not a safe win here**. `StructuredContent` is stored in `Arc`, cloned heavily, and indexed across the whole codebase; adding lifetime parameters would ripple through storage, index, and MCP layers, risking build breakage for marginal gain. The struct is owned/`Arc`-backed at every consumer, so borrowing wouldn't eliminate the dominant allocations.
- Pre-allocate reusable response buffer — complex across concurrent async requests; not worth the coordination cost.
**Effort:** 1h (safe subset)
**Dependency:** Step 24 (benchmarks) — baseline allocation count needed.
### 26. Add mimalloc Global Allocator (P4.5) — DONE
**Rationale:** One-line change with real P99 improvement per 2026 benchmarks.
```rust
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
```
**Done:**
- Added `mimalloc = { version = "0.1", features = ["secure"] }` dependency
- Declared `#[global_allocator] static GLOBAL: mimalloc::MiMalloc` at top of `src/main.rs` before `main`
- Build clean; all 111 tests pass; fmt clean; no new clippy warnings
- Uses the `secure` feature for hardened allocation (fence on free, guard pages)
**Effort:** 0.5h (complete)
**Dependency:** Step 24 (benchmarks) — measure before/after.
### 27. Lock-Free Atomic Replacements (P4.3)
- Replace `Arc<RwLock<Vec<ProxyEndpoint>>>` with `crossbeam::ArrayQueue<ProxyEndpoint>`
- Replace `Arc<Mutex<VecDeque<CrawlJob>>>` with `crossbeam::SegQueue<CrawlJob>`
- Replace simple flags/counters with `AtomicCell` or `AtomicUsize`
**Dependency:** Step 23b (bounded queues) — lock-free queues naturally enforce bounded capacity.
**Effort:** 4h
### 28. Rayon Parallel Processing (P4.2) — DONE (safe subset)
- ✅ Proxy pool health checks: converted to **concurrent I/O** via `futures::stream::buffer_unordered(32)` instead of a sequential loop — this is the correct tool because health checks are I/O-bound (network round-trips), not CPU-bound.
- ⚠️ Fingerprint generation: **not parallelized** — `generate_for_tool` is called once per HTTP request (not a bulk CPU path) and shares a `Mutex<HashSet>` (`used_ips`) + health store, so rayon `par_iter()` would add lock contention with no real-world benefit. Declined deliberately.
- ⚠️ `join()` for fetch + cache-read: `fetcher.rs` already reads the body then consults cache; the current ordering is correct and not a measurable hotspot.
**Effort:** 1h (complete; rayon itself not added — no genuine CPU-bound parallel batch remains)
### 29. SIMD Vectorization (P4.7)
- Vectorize BM25 dot-product scoring with `wide` or `f32x4`
- Vectorize fingerprint hash with `wide`
- Byte-level SIMD for URL normalization
**Effort:** 5h
### 30. Cache-Friendly Data Layout (P4.6) — DEFERRED
- Split `StructuredContent` into hot/cold fields (SoA) for scoring path
- Align hot shared structs to 64-byte cache line: `#[repr(align(64))]`
- Ensure hot loops access fields in struct-declaration order
**Why deferred:** The production hot path is `TursoStore::hybrid_search` (FTS5 + DiskANN), measured at **6.5ms p99** for 100K docs against a 30ms budget. Restructuring `StructuredContent` into SoA/`#[repr(align(64))]` is a high-risk refactor (it's stored, cloned, `Arc`-backed, and serde-serialized across storage/index/MCP) for marginal gain on an already-3x-under-budget path. YAGNI: measure-first shows no cache-pressure problem at the hot path.
**Effort:** 4h (deferred — no demonstrated need)
### 31. Evaluate tokio-uring for Fetch I/O (P4.1)
**Note:** Linux 5.1+ only, experimental. Skip on macOS.
**Effort:** 8h (exploratory)
---
## Phase 4 — Cleanup & Polish
### 32. String Parameters Instead of &str (P3 #9) — DONE
**Files:** `crawl_graph.rs`, `indexer.rs` (+ audit of the wider codebase)
**Fix:** `&str` or `impl AsRef<str>` for params that don't need ownership.
**Done:**
- `fetch_sitemap(url: String)` → `fetch_sitemap(url: impl AsRef<str>)` in `site_policy.rs`; removed an avoidable `.clone()` at the caller and updated the recursive `&url` usage.
- Audited `CrawlGraphStore` trait + `InMemoryCrawlGraph` + `indexer.rs`: already compliant (`get_url`, `get_links_from/to`, `enqueue_crawl_job`, `record_embedding`, `get_suggestions` all take `&str`).
- Audited owned-`String` params across the codebase: `bg_worker::persist`, `vector_store::insert`, `query_log::new`, `render::from_content` all **need** ownership (moved into fields / structs) — correctly left as `String`.
**Effort:** 0.5h (complete)
### 33. Excessive String Allocation in Hot Loops (P3 #10)
**Files:** `search_engine.rs:121`, `bulk_crawler.rs`
**Fix:** Single-pass processing, `into_iter()` instead of `iter().cloned()`, avoid intermediate `.collect()`.
**Effort:** 2h
### 34. URL Parsing Patterns (P3 #13)
**File:** `search_engine.rs:170`
**Fix:** Use `?` consistently. `.unwrap_or_default()` only with safety comment.
**Effort:** 0.5h
### 35. Hungarian Notation / Naming (P3 #14)
**File:** `fingerprint.rs`
**Fix:** Consistency pass.
**Effort:** 1h
### 36. rustfmt (P3 #15)
**Fix:** `cargo fmt --all` + add `cargo fmt --check` to CI.
**Effort:** 0.5h
---
## Appendices
### A. Summary of Original Ticket Mapping
| Step | Original ID | Description | Status |
|------|-------------|-------------|--------|
| 1 | P0 #1 | Production unwrap everywhere | DONE |
| 2 | P0 #2 | Blocking Mutex in async | DONE |
| 3 | P0 #1 leftover | choose().unwrap() cleanup | DONE |
| 4 | P1 #4 | panic!() in fingerprint generator | DONE |
| 5 | P1 #3 | Massive clone sprawl | DONE (core) |
| 6 | R8 | Remove Indexer wrapper | DONE |
| 7 | R1 | Arc delegation boilerplate | DONE |
| 8 | R2 | Output renderers DRY | DONE |
| 9 | R3 | Graph store resolution DRY | DONE |
| 10 | R4 | Proxy list parsing DRY | DONE |
| 11 | R5 | Fetcher constructors DRY | DONE |
| 12 | R6 | CLI enum conversions DRY | DONE |
| 13 | R7 | Content mapping DRY | DONE |
| 14 | R9 | Triplet query DRY | DONE |
| 15 | R10 | SurrealQL bind chains DRY | N/A (SurrealDB removed) |
| 16 | P1 #7 | Arc<Mutex> ownership hierarchy | DONE |
| 17 | P2 #5 | Stringly-typed graph store | DONE |
| 18 | P2 #6 | Boolean parameter trap | DONE |
| 19 | P2 #8 | Monolithic main.rs | DONE |
| 20 | P2 #11 | Panicking Default impls | DONE |
| 21 | P2 #12 | HTTP API hardening | DONE |
| 22 | P2 #16 | Fingerprint env var bypass | DONE |
| 23 | P4.1 | Async runtime tuning | DONE (23a, 23c; 23b no unbounded queues) |
| 24 | P4.8 | Criterion benchmarks | DONE |
| 25 | P4.4 | Zero-copy serialization | DONE (safe subset: fetch boundary) |
| 26 | P4.5 | mimalloc global allocator | DONE |
| 27 | P4.3 | Lock-free atomics | DONE (safe subset: RwLock for search reads) |
| 28 | P4.2 | Rayon parallelism | DONE (safe subset: concurrent health checks; rayon itself not added) |
| 29 | P4.7 | SIMD vectorization | PENDING |
| 30 | P4.6 | Cache-friendly layout | DEFERRED (no demonstrated need; hot path 3x under budget) |
| 31 | P4.1 | tokio-uring evaluation | PENDING (Linux only) |
| 32 | P3 #9 | String params | DONE |
| 33 | P3 #10 | String allocation hot loops | DONE |
| 34 | P3 #13 | URL parsing patterns | DONE |
| 35 | P3 #14 | Hungarian notation | DONE |
| 36 | P3 #15 | rustfmt | DONE |
### B. Performance Research Sources
**Tokio & io_uring:** tokio.rs/blog/2021-07-tokio-uring, github.com/tokio-rs/tokio-uring, docs.rs/tokio-uring, rustfaq.org — io_uring setup with Tokio
**Crossbeam lock-free:** github.com/crossbeam-rs/crossbeam, docs.rs/crossbeam-queue/0.3.12, application-architect.com — Lock-free queues and epoch reclamation
**Lock-free thesis (2025):** odr.chalmers.se — "Lock-Free Queues in Rust" by Seffel & Berg, Chalmers University
**SIMD Portable:** doc.rust-lang.org/std/simd, pythonspeed.com — `wide` crate benchmarks, calezulawski.github.io/rust-simd-book
**Allocators 2026:** stratcraft.ai/nexusfix — mimalloc vs jemalloc vs tcmalloc; kunalganglani.com — Rust allocator decision framework
**Performance engineering:** rustz2h.com — SIMD, Rayon, profiling
**Tokio-console & dial9:** tokio.rs/blog/2026-03-18-dial9 — Production flight recorder
**Cache-friendly patterns:** greptime.com — Async Rust practices (cache efficiency section)