Web Search (Rust)

Rust implementation of the web-search library, CLI, and HTTP service. It mirrors the JavaScript package in ../js with the same 22-provider catalog, provider categories, merge strategies, and discovery surface.

Install

cargo install web-search

As a library:

[dependencies]
web-search = "0.2"

From a local checkout before a crates.io release is visible:

[dependencies]
web-search = { path = "../web-search/rust" }

Library

use web_search::{MergeOptions, MergeStrategy, SearchOptions, WebSearchEngine};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = WebSearchEngine::new();

    let results = engine
        .search_with_options(
            "graph neural networks",
            SearchOptions {
                limit: Some(10),
                ..Default::default()
            },
            Some(vec!["arxiv".to_string(), "crossref".to_string()]),
            Some(MergeOptions {
                strategy: MergeStrategy::Rrf,
                ..Default::default()
            }),
        )
        .await?;

    for result in results {
        println!("{} {}", result.title, result.url);
    }

    Ok(())
}

CLI

web-search "rust async search" --limit 10
web-search "transformer architecture" --providers arxiv,crossref --format json
web-search --list-providers

HTTP Service

web-search serve --port 3000

curl "http://localhost:3000/search?q=rust+programming&limit=10"
curl "http://localhost:3000/providers?category=papers"
curl "http://localhost:3000/categories"

Providers

The live registry has 22 providers in four categories:

google and bing use official APIs when credentials are configured and fall back to HTML parsing otherwise. GITHUB_TOKEN is optional and raises the GitHub search rate limit.

web-capture

wc:wikipedia, wc:duckduckgo, wc:google, wc:bing, and wc:brave delegate to the published web-capture crate. Runtime errors from that component are logged and converted to an empty result set so other selected providers can still return results.

use web_search::providers::{SearchOptions, SearchProvider, WebCaptureProvider};

let provider = WebCaptureProvider::new("wikipedia");
let results = provider.search("OpenAI", &SearchOptions::default()).await?;

Release

The Rust workflow publishes web-search to crates.io from main after lint, tests, doc tests, and package checks pass. GitHub releases are tagged as rust-v<version> so they stay distinct from JavaScript js-v<version> releases.

The current web-capture 0.3.30 dependency requires two transitive pins: html-to-markdown-rs = 3.5.7 and time = 0.3.36. They match the dependency graph the component was released against and avoid known compile breaks in newer compatible-range releases.

Development

cargo test --all-features
cargo test --doc
cargo fmt --all -- --check
cargo clippy --all-targets --all-features
cargo package --list --allow-dirty

Cross-language parity is checked from the repository root:

node js/scripts/check-js-rust-parity.mjs

License

Unlicense - Public Domain