Crate a3s_search 

a3s-search

An embeddable meta search engine library inspired by SearXNG.

This library provides a framework for aggregating search results from multiple search engines, with support for:

• Async parallel search execution
• Result deduplication and merging
• Configurable ranking algorithms
• Extensible engine interface
• Dynamic proxy IP pool for anti-crawler protection

Example

use a3s_search::{Search, SearchQuery, engines::DuckDuckGo};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut search = Search::new();
    search.add_engine(DuckDuckGo::new());

    let query = SearchQuery::new("rust programming");
    let results = search.search(query).await?;

    for result in results.items() {
        println!("{}: {}", result.title, result.url);
    }
    Ok(())
}

Using Proxy

use a3s_search::{Search, SearchQuery, engines::DuckDuckGo, HttpFetcher, PageFetcher};
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let fetcher: Arc<dyn PageFetcher> = Arc::new(
        HttpFetcher::with_proxy("http://proxy1.example.com:8080")?
    );

    let mut search = Search::new();
    search.add_engine(DuckDuckGo::with_fetcher(
        a3s_search::engines::DuckDuckGoParser, fetcher,
    ));

    let query = SearchQuery::new("rust programming");
    let results = search.search(query).await?;

    Ok(())
}

Using Dynamic Proxy Pool

use std::sync::Arc;
use a3s_search::{Search, SearchQuery, PooledHttpFetcher, PageFetcher};
use a3s_search::engines::{DuckDuckGo, DuckDuckGoParser};
use a3s_search::proxy::{ProxyPool, ProxyProvider, ProxyConfig, spawn_auto_refresh};

// Implement your own provider to fetch proxies from any source
struct MyProxyProvider { /* ... */ }

#[async_trait::async_trait]
impl ProxyProvider for MyProxyProvider {
    async fn fetch_proxies(&self) -> a3s_search::Result<Vec<ProxyConfig>> {
        // Fetch from your proxy API, database, etc.
        Ok(vec![ProxyConfig::new("10.0.0.1", 8080)])
    }
    fn refresh_interval(&self) -> std::time::Duration {
        std::time::Duration::from_secs(60) // refresh every minute
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let pool = Arc::new(ProxyPool::with_provider(MyProxyProvider { /* ... */ }));
    let _refresh_handle = spawn_auto_refresh(Arc::clone(&pool));

    let fetcher: Arc<dyn PageFetcher> = Arc::new(PooledHttpFetcher::new(Arc::clone(&pool)));

    let mut search = Search::new();
    search.add_engine(DuckDuckGo::with_fetcher(DuckDuckGoParser, fetcher));

    let query = SearchQuery::new("rust programming");
    let results = search.search(query).await?;
    Ok(())
}

Modules

engines

Search engine implementations.

proxy

Dynamic proxy IP pool for anti-crawler protection.

Structs

Aggregator

Aggregates and ranks search results from multiple engines.

EngineConfig

Configuration for a search engine.

EngineEntry

Per-engine configuration entry.

HealthConfig

Configuration for the health monitor.

HealthEntry

Health monitor configuration entry.

HealthMonitor

Monitors engine health and auto-suspends failing engines.

HtmlEngine

Generic base for all HTML-scraping search engines.

HttpFetcher

A page fetcher that uses plain HTTP requests via reqwest.

PooledHttpFetcher

A page fetcher that rotates proxies from a ProxyPool on each request.

Search

Meta search engine that orchestrates searches across multiple engines.

SearchConfig

Top-level search configuration.

SearchQuery

A search query with all parameters.

SearchResult

A single search result.

SearchResults

Container for aggregated search results.

Enums

EngineCategory

Categories for search engines.

ResultType

Type of search result.

SafeSearch

Safe search level.

SearchError

Errors that can occur during search operations.

TimeRange

Time range filter for search results.

WaitStrategy

Strategy for waiting until a page is considered fully loaded.

Traits

Engine

Trait for implementing search engines.

HtmlParser

Engine-specific logic for HTML-scraping search engines.

PageFetcher

Trait for fetching the full HTML content of a URL.

Functions

selector

Parse a CSS selector string, returning a SearchError::Parse on failure.

Type Aliases

Result

Result type alias for search operations.