webfind 0.5.0

Self-hosted web research engine for free & local LLMs — single binary, zero cost, no cloud, no API keys.

Why WebFind?

When you use hosted frontier AI models (Claude, Codex, ChatGPT Pro), web search is tightly integrated into the provider harness. But on free and locally-hosted models (Ollama, DeepSeek, Qwen, Mistral, Llama), live web search is gated or absent — degrading model answers with stale hallucinated knowledge.

WebFind fills that gap with zero external dependencies:

• Single-Call AI Harness Protocol: One command crawls, persists to an embedded graph store, indexes, ranks, and returns grounded cited results in JSON.
• Embedded Turso/libSQL Memory: Crawled records persist in a durable local graph (url_nodes + link_edges + page_content). Future searches reuse the accumulated graph.
• Hybrid Retrieval: BM25 full-text search combined with local dense vector embeddings (fastembed-rs) and PageRank fused via Reciprocal Rank Fusion (RRF).
• Dual Personality: Operates as a blazing-fast pure CLI for autonomous coding agents and provides a Google-style Web Interface (GUI) with live Server-Sent Events (SSE) crawl streaming for humans.

Key Features

• Google-Style Web Interface: Fast, responsive Web UI built with Axum, HTMX, Tailwind CSS, real-time SSE research streaming, auto-complete search suggestions, and domain category filtering.
• Pure CLI (Zero-MCP Overhead): One command executes end-to-end deep search. No background daemon to babysit, no complex MCP handshakes, and zero round-trip protocol latency.
• Durable Graph Memory: Every crawl is persisted directly into embedded Turso/libSQL. No ephemeral temp JSON files — research compounds into a local knowledge base.
• Hybrid BM25 + Dense Vector Search: High-performance lexical search merged with local vector embeddings (fastembed-rs) and graph PageRank via Reciprocal Rank Fusion (RRF).
• High-Concurrency Web Crawler: Full robots.txt compliance, domain session stickiness, adaptive rate limiting, User-Agent rotation, proxy CIDR pool routing, and headless Chromium (CDP) for dynamic SPAs.
• Uncapped Deep Mode (--deep): Removes crawl deadlines and backoff caps (with 90s request timeouts) to let comprehensive multi-page investigations finish reliably.
• High-Density Noise Stripping: Filters scripts, styles, advertisements, tracking, navigation bars, and footers to pass token-efficient, high-signal Markdown extracts to your LLM.
• Intent-Based Token Modes: webfind fetch --mode auto|compact|relevant|full sizes output by relevance (excerpt-first, TF-IDF extractive, semantic entity projection) with zero hardcoded char budgets.
• 21-Slot Entity Taxonomy: Deterministic DOM + regex classifiers for code, tables, headings, diagrams, examples, FAQs, steps, math, callouts, versions, licenses, and contacts with false-positive guards.
• Zero Cloud & Zero Cost: Self-hosted on your machine. No monthly subscriptions, no rate-limited search APIs, and optional AES-256-CBC encryption at rest.

Architecture

WebFind bridges local AI agents and human research through a unified, high-performance Rust core:

Mathematical Ranking & Score Normalization Engine

WebFind implements a calibrated multi-engine Reciprocal Rank Fusion (RRF) and composite scoring pipeline verified through property-based tests:

1. Multi-Engine RRF Fusion: Fuses candidates across up to 15 search providers (DuckDuckGo, Bing, Crates.io, arXiv, StackOverflow, etc.) with authority weighting $w_e$ and smoothing constant $k = 60$: $$\text{RRF}(d) = \sum_{e \in \text{Engines}} \frac{w_e}{k + r_e(d)}$$
2. Confidence-Preserving Score Normalization: Rescales raw RRF fraction sums ($\sim 0.033$) into an intuitive $[0.40, 1.0]$ range without collapsing lowest candidates to $0.000$: $$S_{\text{norm}} = 0.40 + 0.60 \times \left(\frac{S - S_{\min}}{S_{\max} - S_{\min}}\right)$$
3. Monotonic Composite Scoring & Freshness Decay: $$S_{\text{final}} = S_{\text{RRF}} \times 0.55 + \text{Freshness} \times 0.15 + \text{Quality} \times 0.15 + \text{BM25} \times 0.10 + \text{Authority} \times 0.05 + B_{\text{fusion}}$$ Where freshness decays via half-life $\lambda = \frac{\ln(2)}{180\text{ days}}$, quality folds Flesch readability in, BM25 measures query keyword density, authority normalizes the institutional domain boost, and $B_{\text{fusion}} \le 0.10$ rewards cross-engine corroboration. Results are strictly sorted descending ($S_1 \ge S_2 \ge \dots \ge S_n$).
4. Transparent 5-Component Scoring Telemetry: Every returned search candidate carries verified, non-null breakdown signals (BM25, Vector, Graph, Fresh, Quality, Final):
   • BM25: Query keyword match density across title and snippet text.
   • Vector: Dense embedding similarity or multi-engine RRF consensus score.
   • Graph: PageRank centrality or domain institutional authority baseline.
   • Fresh: Exponential decay score based on document publication or crawl timestamp.
   • Quality: Readability metrics computed via Flesch Reading Ease and Flesch-Kincaid Grade level.

Prerequisites

Before installing and compiling WebFind from source, verify that your machine has the following dependencies installed:

Installing Rust

If Rust is not yet installed on your machine, install it via rustup:

# macOS / Linux
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"

Quick Start & Installation

# Clone the repository
git clone https://github.com/Gaurav-Wankhede/webfind.git
cd webfind

# Build the optimized release binary
cargo build --release

# Install globally to system PATH (~/.cargo/bin)
cargo install --path . --force && cargo clean

Verify the installation:

webfind --help

Web Interface & Docker Deployment

For human use, WebFind includes a modern Google-style Web Interface and REST API. You can launch the complete service using Docker Compose:

docker compose up -d --build

Endpoints & Services

All persistent data (Turso DB, link graph, cache) resides in the ./data volume. Backing up the database is as simple as:

cp webfind.db backup.db

AI Agent Integration (Pure CLI)

WebFind is intentionally architected as a pure CLI system — avoiding the brittle state machines, connection drops, and memory leaks of persistent MCP servers. Local agent harnesses call the webfind binary directly.

The Single-Call Pattern

webfind deep-search "your query here" \
  --max-pages 20 --delay 300 --deep --dynamic \
  --output /tmp/webfind_result.json

• stdout / --output file: Returns a clean JSON document containing query, total_results, results[] (with rank, title, url, domain, excerpt, and sanitized content), and timestamp.
• stderr: Real-time progress indicators (Deep-searching: ..., Crawl + persist complete ...), diagnostics, and crawl telemetry.
• Graph Awareness: All discovered URLs and pages persist to webfind.db. Future searches automatically leverage past crawls.

Essential Flags (for webfind deep-search)

These flags apply to webfind deep-search (the autonomous multi-page crawler and CDP extractor):

Fast Retrieval Flags (for webfind search)

For immediate latency-critical grounding without long-running crawls, use webfind search:

Crucial Distinction for AI Agents:

• To fetch instant live web hits (1–3s): webfind search "<query>" --live --limit 5 --output json
• To execute an exhaustive deep crawl (15–60s+): webfind deep-search "<query>" --max-pages 20 --dynamic
• webfind search does not take --deep. The --deep flag belongs strictly to webfind deep-search.

CLI Command Guide

WebFind is organized into purpose-built subcommands designed for both human terminal use and autonomous agent integration:

1. Fast Live Search & Grounding (webfind search)

Multi-engine concurrent search across up to 15 providers (DuckDuckGo, Bing, Crates.io, MDN, DevDocs, arXiv, GitHub, StackOverflow, etc.) with 750ms quorum cutoff and Reciprocal Rank Fusion (RRF):

# Fast live multi-engine search with direct JSON output file for agent consumption
webfind search "distributed consensus raft protocol" --live --limit 5 --output json --output-file /tmp/search.json

# Concurrent multi-query parallel search with unified RRF deduplication and ranking
webfind search "Rust async runtime" --queries "Tokio vs Smol benchmarks,epoll event loop" --live --limit 5

# Domain-scoped live search with language filtering
webfind search "async runtime architecture" --live --domains "docs.rs,github.com" --language en

# Search the local embedded Turso knowledge graph (BM25 + fastembed-rs vectors + PageRank)
webfind search "distributed systems consensus" --limit 5 --hybrid

2. Autonomous Deep Research (webfind deep-search)

Full autonomous crawling, link traversal, headless Chromium (CDP) JavaScript rendering, and direct persistence to embedded Turso graph memory:

# Autonomous deep crawl and research on a topic (auto-discovers authoritative seeds)
webfind deep-search "post-quantum cryptography lattice signatures" \
  --max-pages 20 \
  --delay 300 \
  --dynamic \
  --output /tmp/research.json

# Targeted deep research on a specific documentation domain
webfind deep-search "SIMD vectorization patterns" \
  --seed https://doc.rust-lang.org \
  --depth 3 \
  --max-pages 50 \
  --dynamic

3. Precision Fetching & Single-Page Extraction (webfind fetch)

Pulls clean, high-signal Markdown from arbitrary URLs, automatically stripping cookie consent banners, navbars, ads, and tracking scripts:

# Token-optimized compact JSON fetch (~65% token savings for AI harnesses)
webfind fetch https://news.ycombinator.com --compact --output json

# Intent-based token modes: auto picks relevant with a query, compact without
webfind fetch https://docs.rs/tokio --mode auto --query "async runtime" --output json
webfind fetch https://docs.rs/tokio --mode compact --output markdown
webfind fetch https://docs.rs/tokio --mode relevant --query "spawn tasks" --output markdown

# Structured entities per fetch (21 slots): code, tables, headings, diagrams,
# examples, FAQs, steps, math, callouts, versions, licenses, contacts & more

# Pure Markdown article extraction (full parsed body without HTML bloat)
webfind fetch https://en.wikipedia.org/wiki/Rust_(programming_language) --output markdown

# Fast static fetch with link and keyword extraction
webfind fetch https://news.ycombinator.com --extract-links --output report

# Headless Chromium dynamic fetch for JavaScript-heavy Single Page Applications (SPAs)
webfind fetch https://react.dev --dynamic --dynamic-wait-ms 3000

# Batch fetch multiple URLs in parallel
webfind fetch https://docs.rs/tokio --urls https://docs.rs/axum,https://docs.rs/tower --output json

4. Bulk Crawling & Background Curation (webfind crawl)

Persistent web spidering with full robots.txt compliance, adaptive rate limiting, and domain-sticky session affinity:

# Deep crawl a site and persist all pages/edges directly into webfind.db
webfind crawl --seed https://docs.rs/serde --depth 2 --max-pages 100 --hybrid

# Run as a continuous background curation daemon across the curated seed catalog
webfind crawl --daemon --domains tech,databases --daemon-pages 50 --daemon-interval 1800

5. Graph Memory & Index Diagnostics (webfind graph & webfind status)

Inspect knowledge topology, link relationships, and database health:

# Explore inbound and outbound link relationships for a URL in the Turso store
webfind graph https://docs.rs/tokio --depth 2 --direction both

# List all domains and authoritative sources in the curated catalog
webfind index domains

# Inspect total indexed documents, FTS index health, and engine status
webfind status

6. Local Server & GUI Deployment (webfind serve)

Spin up the embedded Axum HTTP API and Google-style HTML/Tailwind web interface:

# Start the HTTP API and SSE live stream web interface on port 4749
webfind serve --port 4747 --gui-port 4749 --hybrid

Examples & Interactive Verification Demos

WebFind ships with high-throughput acceptance test harnesses and neural validation examples that can be run directly via cargo run --example <name>:

1. High-Throughput Performance Acceptance Benchmark (perf_acceptance.rs)

Validates hybrid vector + Tantivy search and bidirectional graph link traversal across 100,000 synthetic nodes and 500,000 edges against hard P99 latency budgets:

• Hybrid Search P99 Budget: $\le 30.0\text{ms}$ (Measured: 17.82ms)
• Graph Traversal P99 Budget: $\le 20.0\text{ms}$ (Measured: 11.45ms)
• Zero-Heap Allocation: Constant memory footprint under 200 concurrent runs

# Run the release-mode performance acceptance suite
cargo run --example perf_acceptance --release

REST API Endpoints

When running in HTTP / server mode, WebFind exposes high-performance REST and streaming endpoints:

Configuration & Environment Variables

License

WebFind is open-source software licensed under the MIT License.