millipede-html

HtmlCrawler for the Millipede web crawler: HTML parsing via scraper.

This crate adds synchronized HTML parsing, owned CSS-selector helpers, and link extraction to Millipede's HTTP crawler. Its public selectors! macro caches validated selectors for reuse.

Installation

[dependencies]
millipede-html = "0.1"

Most users should depend on the umbrella millipede crate instead.

Example

use std::sync::Arc;

use millipede_core::prelude::Crawler;
use millipede_html::{HtmlContext, HtmlCrawler, HtmlKind};
use millipede_storage_memory::MemoryStorageClient;

millipede_html::selectors! {
    title_selector = "title";
}

# async fn crawl() -> Result<(), Box<dyn std::error::Error>> {
let crawler: HtmlCrawler = Crawler::builder(HtmlKind::new()?)
    .storage_client(Arc::new(MemoryStorageClient::new()))
    .request_handler(|ctx: HtmlContext| async move {
        if let Some(title) = ctx
            .html
            .select_first(title_selector(), |element| element.text().collect::<String>())
        {
            println!("{}: {title}", ctx.request.url);
        }
        let _ = ctx.enqueue.options().selector("a[href]").send().await?;
        Ok(())
    })
    .build()
    .await?;

crawler.run(["https://example.com/"]).await?;
# Ok(())
# }

Part of Millipede

See the Millipede guide for crawler concepts, link discovery, routing, and scraping patterns.

License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.