millipede-browser-chromiumoxide

Chromium CDP browser provider for the Millipede web crawler, via chromiumoxide.

This crate connects Millipede's provider-neutral browser crawler to Chromium or Google Chrome over the Chrome DevTools Protocol. Browser discovery checks MILLIPEDE_CHROME, then CHROME, then conventional platform installation paths.

Installation

[dependencies]
millipede-browser-chromiumoxide = "0.1"

Most users should depend on the umbrella millipede crate with its browser Chromiumoxide feature instead.

Example

This example needs a real Chrome or Chromium installation. Set MILLIPEDE_CHROME when automatic discovery cannot find the executable.

use std::sync::Arc;

use millipede_browser::{BrowserContext, BrowserCrawler, BrowserKind};
use millipede_browser_chromiumoxide::{
    ChromiumLaunchOptions, ChromiumoxideProvider, find_browser,
};
use millipede_core::prelude::Crawler;
use millipede_storage_memory::MemoryStorageClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let executable = find_browser().expect("Chrome or Chromium is required");
    let kind = BrowserKind::builder(ChromiumoxideProvider)
        .launch_options(ChromiumLaunchOptions::default().with_executable(executable))
        .build()?;
    let crawler: BrowserCrawler<ChromiumoxideProvider> = Crawler::builder(kind)
        .storage_client(Arc::new(MemoryStorageClient::new()))
        .request_handler(|ctx: BrowserContext| async move {
            let title = ctx.page.evaluate_js("document.title").await.ok();
            println!("{}: {title:?}", ctx.request.url);
            Ok(())
        })
        .build()
        .await?;

    crawler.run(["https://example.com/"]).await?;
    Ok(())
}

Part of Millipede

See the Millipede guide for browser pools, hooks, smart crawling, and fingerprinting guidance.

License

Licensed under either MIT OR Apache-2.0 at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.