Expand description
§millipede-browser
BrowserCrawler core, BrowserProvider trait, and BrowserPool for the Millipede web crawler.
This crate defines Millipede’s provider-neutral browser pages, lifecycle hooks, crawler kind, and browser pool. The concrete Chromium provider comes from millipede-browser-chromiumoxide.
§Installation
[dependencies]
millipede-browser = "0.1"Most users should depend on the umbrella millipede crate instead.
§Example
Pool policy and hooks can be prepared without choosing a provider:
use millipede_browser::{BrowserHooks, BrowserPoolOptions};
let hooks = BrowserHooks::defaults()
.with_launch_args(vec!["--disable-background-networking".to_owned()]);
let options = BrowserPoolOptions::<()>::default()
.with_max_open_pages_per_browser(4)
.with_retire_browser_after_page_count(100)
.with_hooks(hooks);
assert_eq!(options.max_open_pages_per_browser, 4);
assert_eq!(options.retire_browser_after_page_count, 100);§Part of Millipede
See the Millipede guide for browser pools, hooks, smart crawling, and fingerprinting guidance.
§License
Licensed under either MIT OR Apache-2.0 at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate is dual-licensed as above, without any additional terms or conditions.
Re-exports§
pub use detect::BrowserPromotionDetector;pub use detect::DefaultPromotionDetector;pub use detect::HttpAttemptSnapshot;pub use detect::PromotionReason;pub use error::BrowserError;pub use hooks::BrowserHooks;pub use hooks::PageClosedHook;pub use hooks::PageHook;pub use hooks::PagePrepHook;pub use hooks::PreLaunchHook;pub use kind::BrowserContext;pub use kind::BrowserCrawler;pub use kind::BrowserKind;pub use kind::BrowserKindBuilder;pub use nav::BrowserPostHookCtx;pub use nav::BrowserPreHookCtx;pub use page::BrowserPage;pub use page::BrowserResponse;pub use page::GotoOptions;pub use page::PageId;pub use page::PageOptions;pub use page::ScreenshotOptions;pub use page::WaitUntil;pub use pool::BrowserPool;pub use pool::BrowserPoolOptions;pub use pool::PageHandle;pub use provider::BrowserProvider;pub use provider::LaunchContext;pub use smart::SmartContext;pub use smart::SmartCrawler;pub use smart::SmartKind;pub use smart::SmartKindBuilder;
Modules§
- detect
- HTTP-attempt detector interfaces for smart browser promotion. Conservative HTTP-response detection for smart browser promotion.
- error
- Browser-specific errors and crawl error classification. Browser errors and their crawl retry semantics.
- hooks
- Browser lifecycle hooks. Provider-erased browser lifecycle hooks.
- kind
- Browser-backed crawler kind and handler context. Browser-backed crawler kind and handler context.
- nav
- Browser pre- and post-navigation hook contexts. Browser pre- and post-navigation hook contexts.
- page
- Provider-erased browser page operations and page options. Provider-erased browser page operations and configuration.
- pool
- Browser pool and provider-erased RAII page handles. Concurrent browser and page lifecycle management.
- prelude
- Commonly used items from this crate.
- provider
- Concrete browser provider integration points. Integration surface for concrete browser providers.
- smart
- HTTP-first crawler kind with selective browser promotion. HTTP-first crawler kind with selective browser promotion.