Crate millipede_core

Expand description

§millipede-core

Core primitives for the Millipede web crawler: request model, storage traits, events, errors, configuration.

This crate provides the generic crawler engine and the request, handler, routing, session, proxy, storage, event, and statistics abstractions shared by the Millipede ecosystem.

§Installation

[dependencies]
millipede-core = "0.1"

Most users should depend on the umbrella millipede crate instead.

§Example

use std::sync::Arc;

use millipede_core::prelude::{BasicContext, BasicKind, Crawler, Request};
use millipede_storage_memory::MemoryStorageClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let crawler = Crawler::builder(BasicKind)
        .storage_client(Arc::new(MemoryStorageClient::new()))
        .request_handler(|ctx: BasicContext| async move {
            println!("handling {}", ctx.request.url);
            Ok(())
        })
        .build()
        .await?;

    let seed = Request::get("https://example.com/").build()?;
    let stats = crawler.run([seed]).await?;
    assert_eq!(stats.requests_finished, 1);
    Ok(())
}

§Part of Millipede

See the Millipede guide for crawler concepts, storage, retries, sessions, and migration guidance.

§License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.

Modules§

antibot: Content-based anti-bot and web application firewall detection. Content-based anti-bot and web application firewall detection.
autoscale: Autoscaling: dynamic concurrency, load signals, and rate limiting. Dynamic concurrency, load-signal evaluation, and politeness limits.
config: Crawler configuration and environment resolution. Crawler configuration and environment-variable resolution.
cookies: Session cookie storage and persistence. Synchronous session cookie storage and JSON persistence.
crawler: Crawler lifecycle kinds, handles, and shared state. The crawler engine: lifecycle kinds, handles, and shared state.
enqueue: Link enqueueing from handler contexts. Link extraction, filtering, transformation, and enqueueing from handler contexts.
errors: Crawl error taxonomy and retry classification. Crawl errors and their retry semantics.
events: Crawler lifecycle events and broadcast support. Crawler lifecycle events and broadcast delivery.
handler: Request handler and middleware contracts. Request handler and middleware contracts.
http_client: Backend-independent HTTP request, response, and client abstractions. Backend-independent HTTP request and response abstractions.
link_extraction: Link-extraction strategies, URL patterns, and crawl policy. Scraper-independent link extraction types, URL matching, and crawl policy.
prelude: Commonly used items from this crate.
proxy: Proxy configuration and rotation strategies. Proxy configuration and rotation.
request: Request data types and construction helpers. Request values and their builder API.
retry_strategy: Attempt-level retry strategy hooks. Attempt-level retry strategy hooks.
router: Label- and method-based request routing. Label- and method-based request routing.
session: Sessions and reusable session pools. Sessions and the session pool.
sitemap: Streaming sitemap ingestion. Streaming, gzip-aware XML sitemap ingestion.
snapshot: Failure-artifact capture and reload support. Failure-artifact capture and reload support.
statistics: Crawl statistics, rates, and persistence. Crawl statistics: live counters, sliding-window rates, and persistence.
storage: Object-safe storage abstractions and typed convenience wrappers. Object-safe storage backend contracts.