Product OS : Crawler

Product OS : Crawler is a browser based crawler that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.

What is Product OS?

Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.

Feature Flags

default features: scoring, following, throttling, actor-support

Installation

[dependencies]
product-os-crawler = "0.0.1"

Pin the version to match the crate Cargo.toml when using path or git dependencies.

Documentation

Full API documentation is available at docs.rs/product-os-crawler.

Usage

Overview

Product OS : Crawler is a browser based crawler that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.

Installation

Use the Rust crate package manager cargo to install Product OS : Crawler.

cargo add product-os-crawler

or add Product OS : Crawler to your cargo.toml [dependencies] section.

[dependencies]
product-os-crawler = { version = "0.0.16", features = [], default-features = true }

Features

Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:

• Browser-Based Crawling: Leverages headless browsers for JavaScript-rendered content
• Multiple Crawl Strategies: Focus, Deep, and Broad-Deep crawling modes
• Intelligent Throttling: Adaptive request rate limiting with automatic back-off
• Link Following: Automatic discovery and queueing of related pages
• Content Scoring: Extensible scoring system for content prioritization
• Actor System: Pluggable actor-based content processing pipeline

Usage

Basic Example

use product_os_crawler::{Crawler, CrawlMethod};

#[tokio::main]
async fn main() {
    // Create configuration instances
    let crawler_config = product_os_configuration::Crawler::default();
    let browser_config = product_os_browser::Browser::default();
    let proxy = product_os_proxy::Proxy::default();
    let kv_store_config = product_os_configuration::KeyValueStore::default();
    let queue_store_config = product_os_configuration::QueueStore::default();

    // Create the crawler
    let mut crawler = Crawler::new(
        crawler_config,
        browser_config,
        proxy,
        kv_store_config,
        queue_store_config,
        vec![], // actor scripts
        vec![], // browser helpers
    );

    // Initialize the crawler (connects stores, starts proxy, initializes browsers)
    crawler.init_service_mut().await.expect("Failed to initialize crawler");

    // Seed a URL to start crawling
    crawler.seed_url(
        "https://example.com".to_string(),
        CrawlMethod::FocusCrawl
    );

    // The crawler will now process URLs in the background
}

Crawl Methods

The crawler supports different strategies for following links:

Focus Crawl

Stays strictly within the seed domain.

crawler.seed_url(
    "https://example.com".to_string(),
    CrawlMethod::FocusCrawl
);

Deep Crawl

Follows links deeply within the same domain with configurable depth limits.

crawler.seed_url(
    "https://example.com".to_string(),
    CrawlMethod::DeepCrawl
);

Broad-Deep Crawl

Can traverse to related domains based on breadth settings.

crawler.seed_url(
    "https://example.com".to_string(),
    CrawlMethod::BroadDeepCrawl
);

Ignore

Processes single URLs without following any links.

crawler.seed_url(
    "https://example.com/specific-page".to_string(),
    CrawlMethod::Ignore
);

Architecture

The crawler consists of several key components:

• Crawler: Main service coordinating all operations
• Seeder: Manages browser instances and request queuing
• CrawlerActor: Implements content processing pipeline
• Follower: Discovers and enqueues links from pages
• Scorer: Assigns priority scores to content

Configuration

The crawler behavior can be configured through:

• Crawl depth and breadth limits
• Revisit delays
• Throttling parameters (frequency, rate, wait times)
• Feature toggles (scrolling, clicks, scoring, following, acting)
• Browser and proxy settings

See the product-os-browser crate for browser configuration options.

Agent-browser stack (LLM automation)

product-os-crawler embeds product-os-browser directly for scheduled/batch crawling (Focus/Deep/Broad-Deep strategies, throttling, scoring).

For interactive LLM agents (observe/act loop, MCP, collect forms, session capture), use product-os-agents-browser on top of the same browser + proxy stack. It exposes HTTP/MCP APIs and vault-backed secret handling rather than crawl queues.

See src/product-os-agents-browser/docs/browser-stack-architecture.md.

Examples

See the examples/ directory for more detailed usage examples:

• basic_crawler.rs: Simple crawler setup

Run examples with:

cargo run --example basic_crawler

Contributing

Contributions are not currently available but will be available on a public repository soon.

License

This project is licensed under the GNU GPLv3.