daedra 0.5.6

Daedra is a self-contained web search MCP server in Rust. It gives search and page-fetch tools to AI agents. It works from any IP address: datacenter, VPS, or residential. Basic search needs no API keys.

Why Daedra

Major search engines block datacenter and VPS IP addresses with CAPTCHAs. Daedra solves this with a multi-backend fan-out. All available backends run the query at once, and the results merge by relevance:

Serper (API) → Tavily (API) → Brave → Marginalia → Bing RSS → Bing → Google News → Hacker News → Google → Wikipedia → StackOverflow → GitHub → Wiby → DDG Instant → DuckDuckGo HTML

Three backend groups exist. API backends need a key. Scraper backends read HTML and sometimes meet a CAPTCHA. Machine-format backends read the RSS and JSON feeds that the engines publish for integrations. The machine-format backends work from any IP with no key.

Unkeyed general coverage depends on where daedra runs. Marginalia answers from any IP. Brave HTML may rate-limit. The knowledge and machine-format backends (wiki, HN, StackOverflow, GitHub, Bing RSS, Google News) work from any IP. Set SERPER_API_KEY or TAVILY_API_KEY for general search quality on any network.

Per-backend circuit breakers and per-backend rate limits keep the chain stable under load. The chain retries only transient errors. Bot protection and rate-limit errors fail fast, so the next backend starts at once.

Features

• 15 search backends with automatic fallback (see the table below)
• Circuit breaker (BackendHealth): opens after repeated failures, with a 30 second cooldown
• Per-backend rate limits via governor, with separate quotas for API, knowledge, and scraper backends
• Classified retry: the chain retries only transient errors
• Readability extraction: dom_smoothie extracts the article body from HTML pages
• PDF support: infer detects the MIME type, pdf-inspector extracts Markdown (OCR-needing pages are reported)
• Content classification: FetchedContent (Html / Pdf / Binary) on every fetch
• URL classification: src/url_classification.rs maps search result URLs to content types
• MCP tools: web_search, visit_page, crawl_site, and the search_duckduckgo alias

Install

cargo install daedra

Search backends

The merge ranks every result by how much of the query it shares. Results that share no query token land after every matched result. When no result matches at all, the search reports this. It does not return unrelated feed noise. Backends that ignore the query cannot outvote backends that found nothing relevant.

Without API keys the backends are the knowledge and feed sources (Wikipedia, Hacker News, StackOverflow, GitHub, RSS). This is a research slice of the web, not a general web search. Set SERPER_API_KEY or TAVILY_API_KEY for general search quality.

--backend and --exclude select or skip backends by name. --time-range d|w|m|y filters by recency on the backends that support it (Serper, Tavily, Hacker News, Google News).

Usage

MCP server (for Claude, Cursor, pawan, and similar agents)

{
  "mcpServers": {
    "daedra": {
      "command": "daedra",
      "args": ["serve", "--transport", "stdio", "--quiet"]
    }
  }
}

CLI

# Search: pick backends, filter by recency
daedra search "rust async runtime" --num-results 5
daedra search "crate release" --backend hn --time-range w
daedra search "tokio" --exclude bing-rss --exclude gnews

# Fetch a webpage as Markdown (HTML via Readability, PDF via pdf-inspector)
daedra fetch https://rust-lang.org
daedra fetch https://example.com/report.pdf --timeout 60
daedra fetch https://example.com/report.docx   # docx, odt, epub, pptx, xlsx, csv too

# Crawl with the optional crawlberg engine (rebuild with --features crawlberg)
daedra crawl https://rust-lang.org --engine crawlberg -m 10 -d 3

# Crawl: root page always included; robots.txt honored; depth follows links
daedra crawl https://rust-lang.org -m 5 --depth 2 --delay-ms 250

# Check backend health (reports missing API keys; JSON with -f json)
daedra check

# Server info (four MCP tools; JSON with -f json)
daedra info

As a Rust library

use daedra::tools::SearchProvider;
use daedra::types::SearchArgs;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let provider = SearchProvider::auto();
    let args = SearchArgs {
        query: "rust programming".to_string(),
        options: None,
    };
    let results = provider.search(&args).await?;
    for r in &results.data {
        println!("{} — {}", r.title, r.url);
    }
    Ok(())
}

MCP tools

web_search

Search the web with automatic backend fallback.

{
  "query": "search terms",
  "options": {
    "region": "wt-wt",
    "safe_search": "MODERATE",
    "num_results": 10,
    "time_range": "w"
  }
}

search_duckduckgo is an alias for web_search. It exists for backward compatibility.

visit_page

Fetch a page and extract the content as Markdown. HTML pages use dom_smoothie Readability extraction. The infer crate detects PDFs, and pdf-inspector extracts Markdown. Office documents (docx, doc, odt, rtf, epub, pptx, xlsx, csv) convert via anydoc.

{
  "url": "https://example.com",
  "selector": "article.main",
  "include_images": false
}

crawl_site

Crawl a site from a root URL and return Markdown for each page. The crawler always fetches the root page, honors robots.txt (longest-match rule; ignore_robots opts out), expands one level of sitemap indexes, and can follow same-origin link layers with depth (max 5). delay_ms staggers fetches. The command exits non-zero when it fetched zero pages.

Architecture

Daedra
├── SearchProvider (fallback chain, circuit breakers, keyed rate limits)
│   ├── SerperBackend / TavilyBackend (API, optional keys)
│   ├── BingRssBackend / GoogleNewsBackend / HnAlgoliaBackend (machine formats)
│   ├── BingBackend / GoogleBackend (HTML scraping)
│   ├── WikipediaBackend / StackExchangeBackend / GitHubBackend
│   ├── WibyBackend / DdgInstantBackend
│   └── SearchClient (DuckDuckGo HTML, last resort)
├── FetchClient (FetchedContent: Html / Pdf / Binary → Markdown)
│   ├── dom_smoothie (Readability), infer (MIME), pdf-inspector (PDF), anydoc (Office)
├── soft_block (classifies zero-result scraper pages: genuine empty or bot block)
├── url_classification (search result URL → ContentType)
├── SearchCache (moka async cache)
├── MCP Server (DaedraHandler: handle_web_search, handle_visit_page, handle_crawl_site)
│   ├── STDIO transport (JSON-RPC)
│   └── SSE transport (Axum HTTP)
└── CLI (Commands::run, CheckReporter)

Key dependencies

Configuration

# Optional API keys (improve result quality)
export SERPER_API_KEY=...     # Google results via Serper
export TAVILY_API_KEY=...     # AI-optimized search
export GITHUB_TOKEN=...       # Higher GitHub API rate limit

# Logging
export RUST_LOG=daedra=info

Ecosystem

Built by DIRMACS.

License

MIT