Crate millipede_html 

millipede-html

HtmlCrawler for the Millipede web crawler: HTML parsing via scraper.

This crate adds synchronized HTML parsing, owned CSS-selector helpers, and link extraction to Millipede’s HTTP crawler. Its public selectors! macro caches validated selectors for reuse.

Installation

[dependencies]
millipede-html = "0.1"

Most users should depend on the umbrella millipede crate instead.

Example

use std::sync::Arc;

use millipede_core::prelude::Crawler;
use millipede_html::{HtmlContext, HtmlCrawler, HtmlKind};
use millipede_storage_memory::MemoryStorageClient;

millipede_html::selectors! {
    title_selector = "title";
}

let crawler: HtmlCrawler = Crawler::builder(HtmlKind::new()?)
    .storage_client(Arc::new(MemoryStorageClient::new()))
    .request_handler(|ctx: HtmlContext| async move {
        if let Some(title) = ctx
            .html
            .select_first(title_selector(), |element| element.text().collect::<String>())
        {
            println!("{}: {title}", ctx.request.url);
        }
        let _ = ctx.enqueue.options().selector("a[href]").send().await?;
        Ok(())
    })
    .build()
    .await?;

crawler.run(["https://example.com/"]).await?;

Part of Millipede

See the Millipede guide for crawler concepts, link discovery, routing, and scraping patterns.

License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.

Re-exports

pub use scraper;

Modules

prelude

Commonly used items from this crate.

Macros

selectors

Declares lazily parsed, process-wide CSS selector accessors.

Structs

HtmlContext

Per-request context produced by HtmlKind.

HtmlKind

HTML fetching behavior that delegates transport concerns to HttpKind.

HtmlKindBuilder

Configures HtmlKind by delegating HTTP settings to HttpKindBuilder.

HtmlLinkExtractor

Extracts raw link targets from an already-parsed HTML document.

SynchronizedHtml

A parsed HTML document with the synchronization required for shared handler access.

Enums

HtmlError

Errors specific to HTML response processing.

Type Aliases

HtmlCrawler

A crawler using HtmlKind to fetch and parse HTML documents.