Crate millipede

Expand description

§millipede

An idiomatic Rust web-crawling library inspired by Crawlee.

§Quick start

[dependencies]
millipede = "0.1.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
serde_json = "1"

use std::sync::Arc;

use millipede::{CrawlPolicy, Crawler, DatasetExt, HtmlContext, HtmlCrawler, HtmlKind};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let crawler: HtmlCrawler = Crawler::builder(HtmlKind::new()?)
        .storage_client(Arc::new(millipede::MemoryStorageClient::new()))
        .crawl_policy(CrawlPolicy::new().max_requests_per_crawl(100))
        .request_handler(|ctx: HtmlContext| async move {
            ctx.storage
                .dataset()
                .push(&serde_json::json!({
                    "url": ctx.request.url.as_str(),
                    "status": ctx.response.status.as_u16(),
                }))
                .await?;
            let _ = ctx.enqueue.same_hostname().await?;
            Ok(())
        })
        .build()
        .await?;

    let stats = crawler.run("https://example.com/").await?;
    println!("finished: {}", stats.requests_finished);
    Ok(())
}

§Feature flags

http, html, and storage-memory are enabled by default. The core API is always available.

Feature	Enables	Default
`http`	Reqwest-based HTTP fetching and `HttpCrawler`	Yes
`html`	HTML parsing and `HtmlCrawler`	Yes
`storage-memory`	In-memory datasets, key-value stores, and request queues	Yes
`storage-fs`	File-system-backed storage	No
`browser`	Browser crawler abstractions and smart crawling	No
`browser-chromiumoxide`	The chromiumoxide CDP provider; also enables `browser`	No
`fingerprint`	Browser-like header generation and fingerprint hooks	No

§Crawler kinds

HttpCrawler fetches URLs over HTTP and exposes response data, sessions, proxies, and URL enqueueing to handlers.

HtmlCrawler adds synchronized HTML parsing and DOM-based link extraction to HTTP crawling.

BrowserCrawler drives pages through a browser provider for JavaScript-rendered sites.

SmartCrawler starts on the faster HTTP path and promotes requests to a browser when its promotion detector identifies a JavaScript shell or another browser-only response.

§Links

§License

Licensed under either the MIT License or the Apache License, Version 2.0, at your option. Contributions intentionally submitted for inclusion are licensed under the same terms.

Re-exports§

pub use millipede_core as core;
pub use millipede_browser as browser;
pub use millipede_browser_chromiumoxide as browser_chromiumoxide;
pub use millipede_fingerprint as fingerprint;
pub use millipede_html as html;
pub use millipede_http as http;
pub use millipede_storage_fs as storage_fs;
pub use millipede_storage_memory as storage_memory;

Modules§

prelude: Commonly used items across all enabled Millipede crates.

Structs§

AddOptions: Options controlling insertion of requests.
AddRequestsBatchedResult: Completion result for a batched request addition.
AimdController: An atomic additive-increase, multiplicative-decrease concurrency controller.
AntiBotSignals: Curated public API for ergonomic millipede::Type imports. Response signals inspected for evidence of an anti-bot challenge.
AttemptObservation: Metadata observed after a kind successfully constructs its handler context.
AttemptOutcome: Borrowed metadata describing a failed request attempt.
AttemptOverrides: Overrides carried to the next attempt of the same request.
AutoSaved: A typed value that can be explicitly persisted to a key-value store.
AutoscaledPool: Concurrency decision and politeness facade consulted by a crawler dispatch loop.
AutoscaledPoolOptions: Configuration for concurrency scaling and request politeness limits.
AutoscalerSnapshot: A point-in-time view of a crawler’s concurrency scaler.
BasicContext: The handler context for BasicKind.
BasicKind: The no-fetch crawler kind whose execution hands requests directly to handlers.
BatchAddHandle: Handle for observing completion of a batched request addition.
BrowserContext: Per-request context produced by BrowserKind.
BrowserFingerprintGenerator: A v0.1 post_page_create consistency stub that produces a consistent user agent and Accept-*/Sec-Ch-Ua-* header set only.
BrowserHooks: Provider-erased browser lifecycle hooks.
BrowserKind: Browser fetching behavior used by BrowserCrawler.
BrowserKindBuilder: Configures a BrowserKind.
BrowserPool: A lazily launched pool of provider browsers and their pages.
BrowserPoolOptions: Configuration for a BrowserPool.
BrowserPostHookCtx: Context supplied to browser post-navigation hooks.
BrowserPreHookCtx: Context supplied to browser pre-navigation hooks.
BrowserResponse: Navigation response metadata when the provider can expose it.
ChromiumBrowser: A launched Chromium process and its CDP event driver.
ChromiumLaunchOptions: Options used to launch one Chromium process.
ChromiumPage: A cloneable adapter over a chromiumoxide page.
ChromiumoxideProvider: Chromium CDP provider backed by chromiumoxide.
ClientLoadSignal: A manually-fed signal representing downstream client throttling.
ClientLoadSignalHandle: Cloneable manual observation handle for a ClientLoadSignal.
CoalescingClient: An HTTP client decorator that joins safe, identical requests already in flight.
Configuration: Fully resolved crawler configuration.
ConfigurationBuilder: Builds a resolved Configuration.
Cookie: A transport-neutral HTTP cookie shared by HTTP and browser crawler contexts.
CookieJar: A synchronized cookie store shared by crawler sessions.
CpuLoadSignal: Periodically samples aggregate system CPU usage.
CpuLoadSignalOptions: Options for periodic system CPU load sampling.
CrawlPolicy: Long-lived limits and URL admission policy applied during a crawl.
Crawler: A configured crawler using lifecycle behavior supplied by K.
CrawlerBuilder: Builds a Crawler around a crawler kind.
CrawlerEnv: Shared process-level state supplied to crawler lifecycle hooks.
CrawlerHandle: A cheaply cloned weak back-reference to a running crawler.
DatasetInfo: Dataset identity and timestamps.
DefaultAntiBotDetector: Curated public API for ergonomic millipede::Type imports. A conservative detector using bounded, vendor-specific static markers.
DefaultPromotionDetector: Conservative built-in promotion heuristics.
DomainRoundRobin: A pure frontier that rotates fairly between URL hosts.
EnqueueLinker: Enqueues child URLs through a running crawler.
EnqueueLinksOptions: Fluent options for one URLs-only enqueue operation.
EnqueueResult: Result of enqueueing a URL collection.
ErrorSnapshot: A failure-time artifact reloaded from storage.
ErrorSnapshotter: Captures and reloads failure-time artifacts in a crawler key-value store.
EventBus: A broadcast channel for crawler control-plane events.
ExtractedLink: A raw extracted link and the optional document base used to resolve it.
FailedRequestContext: Owned payload handed to the failure handler when a request permanently fails.
FinalStatistics: Crawl statistics returned when a run finishes.
FsDataset: A file-system-backed, append-only JSON dataset.
FsKeyValueStore: A file-system-backed byte-oriented key-value store.
FsRequestQueue: A durable FIFO request queue backed by atomic file replacements.
FsStorageClient: A file-system storage client using Crawlee-compatible directory layouts.
GlobPattern: An include glob or regular expression with optional per-pattern overrides.
GotoOptions: Options controlling a page navigation.
HandledRequest: The terminal result of processing one request.
HeaderGenerator: Deterministically selects browser-like header profiles from a curated dataset.
HeaderMap: A specialized multimap for header names and values.
HeaderProfile: A browser user agent and its ordered companion HTTP headers.
HtmlContext: Per-request context produced by HtmlKind.
HtmlKind: HTML fetching behavior that delegates transport concerns to HttpKind.
HtmlKindBuilder: Configures HtmlKind by delegating HTTP settings to HttpKindBuilder.
HtmlLinkExtractor: Extracts raw link targets from an already-parsed HTML document.
HttpAttemptSnapshot: A borrowed snapshot of one successful HTTP/HTML attempt.
HttpContext: Per-request context produced by HttpKind.
HttpKind: HTTP fetching behavior used by HttpCrawler.
HttpKindBuilder: Configures HttpKind.
HttpPostHookCtx: State exposed after an HTTP response is received.
HttpPreHookCtx: State exposed immediately before an HTTP request is sent.
HttpRequest: A backend-independent HTTP request.
HttpResponse: A fully buffered HTTP response.
HttpStatusError: A typed HTTP status carried inside a crate::errors::CrawlError.
KeyInfo: Metadata for one stored key.
KeyList: A page of keys and continuation metadata.
KvEntry: A stored byte value and its metadata.
LaunchContext: Process-level context applied when launching a browser.
Lease: Temporary, linear ownership of a queued request.
LeaseId: Identifier for one active request lease.
ListKeysOptions: Options controlling key pagination.
ListOptions: Options controlling dataset listing order and pagination.
LoadSnapshot: A point-in-time overload observation from a load signal.
MemoryDataset: An in-process, append-only JSON dataset.
MemoryKeyValueStore: An in-process byte-oriented key-value store.
MemoryLoadSignal: Periodically samples used system memory against a configurable budget.
MemoryLoadSignalOptions: Options for periodic system memory load sampling.
MemoryRequestQueue: A thread-safe, lease-based request queue stored entirely in memory.
MemoryStorageClient: An in-process storage client that shares named stores across open calls.
Method: The Request Method (VERB)
Page: A page of dataset items and pagination metadata.
PageHandle: Provider-erased RAII handle for a page checked out from a BrowserPool.
PageId: Stable process-local identifier for a pooled browser page.
PageOptions: Per-page creation context consumed by browser hooks.
ProcessedRequest: Result metadata for adding one request.
ProxyBuckets: Named proxy configurations with deterministic fallbacks.
ProxyConfiguration: Static, custom, or per-domain tiered proxy selection.
ProxyInfo: Parsed connection details for a selected proxy.
ProxyResolveContext: Borrowed inputs available to proxy resolution.
ProxyRouteContext: Borrowed inputs available to a ProxyStrategy.
RateLimitReportingClient: A storage client wrapper that reports healthy and rate-limited operations to autoscaling.
ReclaimOptions: Options controlling return of a leased request to the queue.
Request: A crawl request and its processing state.
RequestBuilder: Builds a Request while deferring parsing and serialization errors.
RequestEnv: Per-attempt inputs supplied to CrawlerKind::execute.
RequestId: A deterministic request identifier.
RequestPrep: Engine-owned scratch space passed to CrawlerKind::before_request.
RequestQueueWithSitemap: A request queue that lazily feeds sitemap entries into another queue.
ReqwestClient: A reqwest-backed HttpClient with manual redirect and cookie handling.
ReqwestClientOptions: Configuration for ReqwestClient.
RetryDirective: Owned instructions returned by a RetryStrategy.
Router: Routes request contexts by label and HTTP method.
ScreenshotOptions: Options controlling screenshot capture.
Session: Cookie, score, and user-data state associated with a crawling identity.
SessionConfig: Limits and scoring behavior for one session.
SessionId: Stable identifier for a crawler session.
SessionPool: A bounded collection of reusable crawler sessions.
SessionPoolOptions: Session pool capacity, creation, and persistence settings.
SessionToken: Stable token used to keep fingerprint generation consistent within a session.
SitemapEntry: One URL entry parsed from a sitemap document.
SitemapRequestList: A lazy, streaming source of requests parsed from XML sitemaps.
SitemapRequestListBuilder: Configures a streaming SitemapRequestList.
SkippedUrl: A skipped URL candidate.
SmartKind: HTTP-first execution with conservative browser promotion.
SmartKindBuilder: Configures SmartKind.
Snapshotter: Coordinates lifecycle and access to configured load signals.
SnapshotterOptions: Configuration for a collection of load signals and their sampling window.
StatisticsHandle: A cheaply cloned handle for recording and reading crawl statistics.
StatisticsSnapshot: A point-in-time view of crawl statistics.
StorageHandle: Open storage resources shared by crawler contexts.
StreamingResponse: An HTTP response whose body arrives as a byte stream.
SynchronizedHtml: A parsed HTML document with the synchronization required for shared handler access.
SystemStatus: Evaluates load-signal histories into scaling decisions.
SystemStatusOptions: Options controlling load-history evaluation.
TokioRuntimeLoadSignal: Detects Tokio executor load by measuring stable-API timer scheduling lag.
TokioRuntimeLoadSignalOptions: Options for detecting Tokio executor scheduling lag.
UrlMatch: A URL pattern with request fields to apply when it matches.
UserData: User-defined JSON metadata attached to a request.

Enums§

AntiBotTech: A recognized anti-bot or web application firewall technology.
AutoscaleMode: Strategy used to adjust desired concurrency.
BrowserError: An error produced while launching or operating a browser.
ConfigError: Errors produced while resolving crawler configuration.
CookieJarError: An error serializing or deserializing a cookie jar.
CrawlError: An error produced while processing a crawl request.
CrawlerBuildError: An error produced while building a crawler.
CrawlerEvent: A control-plane event emitted during a crawler run.
EnqueueStrategy: Controls how closely a discovered URL must relate to its parent URL.
HtmlError: Errors specific to HTML response processing.
HttpClientError: An error produced while preparing or executing an HTTP request.
LinkPatternError: An error produced while compiling link patterns.
LogLevel: Logging verbosity for crawler diagnostics.
MemoryQueuePolicy: Ordering policy used by an in-memory request queue.
MethodFilter: Restricts a route to selected HTTP methods.
PromotionReason: Why an HTTP attempt should be repeated through a browser.
ProxyKind: Logical proxy configuration bucket.
RequestBody: A supported request-body representation.
RequestBuildError: Errors encountered while building a request.
RequestFinalState: A request’s terminal processing state.
RequestOutcome: The outcome supplied to per-attempt cleanup.
RequestSource: A source from which requests can be added.
RequestState: The processing lifecycle state of a request.
RotationStrategy: Selection policy for a static proxy list.
SameSite: A cookie’s cross-site request policy.
ScaleDecision: The concurrency adjustment recommended by current system load.
SessionRetryAction: Session disposition for a strategy-authorized retry.
SkipReason: Why an enqueue candidate was skipped.
SmartContext: A context produced by either the HTTP/HTML or browser execution path.
StorageError: An error produced by a storage backend.
TransformResult: The outcome of transforming a candidate request before enqueueing.
UrlPattern: A URL include or exclude pattern.
WaitUntil: Browser lifecycle event awaited after navigation.

Constants§

SESSION_POOL_PERSIST_KEY: Default key used to persist the session pool.
SITEMAP_STATE_KEY: Conventional key used to persist sitemap request-list progress.
STATISTICS_PERSIST_KEY: KVS key used for statistics persistence (matches Crawlee).

Traits§

AntiBotDetector: Curated public API for ergonomic millipede::Type imports. Detects anti-bot or web application firewall responses from response signals.
BrowserPage: Object-safe browser page surface implemented by concrete providers.
BrowserPromotionDetector: Decides whether a successful HTTP/HTML attempt needs browser execution.
BrowserProvider: Concrete browser backend used by a browser pool.
CrawlerKind: Defines the complete lifecycle for one crawler flavor.
Dataset: Object-safe storage for append-only JSON records.
DatasetExt: Typed convenience operations available on every Dataset.
FailedRequestHandler: Handles a request after it has permanently failed.
HasRequest: Provides request metadata used to select a route.
HttpClient: An object-safe asynchronous HTTP client backend.
IntoStartRequest: A single value convertible into one start Request.
IntoStartRequests: A collection of start requests accepted by super::Crawler::run.
IntoUrl: Conversion into a parsed URL for request builders.
KeyValueStore: Object-safe byte-oriented key-value storage.
KeyValueStoreExt: Typed JSON convenience operations available on every KeyValueStore.
LinkExtractor: Extracts links from a static document or live browser page.
LoadSignal: A source of recent system or client load observations.
Middleware: Transforms a request context before its matched handler runs.
ProxyResolver: Asynchronous custom proxy URL resolver.
ProxyStrategy: Synchronous policy selecting a proxy bucket for a request.
RequestHandler: Processes an owned request context.
RequestQueue: Object-safe queue with temporary lease ownership.
RetryStrategy: Controls retries and next-attempt overrides for non-critical failures.
SkippedHandler: Receives notifications for URL candidates skipped during enqueueing.
StorageClient: Opens named or default storage objects supplied by a backend.

Functions§

find_browser: Finds a supported Chromium or Google Chrome executable.

Type Aliases§

BasicCrawler: The no-HTTP crawler: drives the queue and hands requests straight to the handler.
BrowserCrawler: A crawler using BrowserKind to render requests in browser pages.
BrowserPostNavigationHook: Asynchronous browser hook run after navigation and status classification.
BrowserPreNavigationHook: Asynchronous browser hook run after page creation and before navigation.
EventStream: A receiver for crawler events.
HtmlCrawler: A crawler using HtmlKind to fetch and parse HTML documents.
HttpCrawler: A crawler using HttpKind to fetch raw HTTP responses.
HttpPostNavigationHook: Asynchronous hook run after an HTTP response is received and anti-bot detection completes.
HttpPreNavigationHook: Asynchronous hook run immediately before an HTTP request is sent.
PageClosedHook: Synchronous notification run after a page has closed.
PageHook: Asynchronous hook operating on a provider-erased page and its creation context.
PagePrepHook: Synchronous hook that prepares per-page creation context.
PreLaunchHook: Synchronous hook run before a browser launches.
QueueOpInfo: Alternate interface spelling for ProcessedRequest; both names describe the same payload.
ResultStream: A receiver for terminal request snapshots (the data-plane feed).
SmartCrawler: A crawler using SmartKind for HTTP-first browser promotion.
StorageResult: Result type returned by every storage operation.

Crate millipede

Crate millipede Copy item path

§millipede

§Quick start

§Feature flags

§Crawler kinds

§Links

§License

Re-exports§

Modules§

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Crate millipede