Skip to main content

Crate celestia_spider

Crate celestia_spider 

Source
Expand description

§celestia-spider

A high-performance Rust web browsing library and AI Intermediate Representation (IR) generator. Built around a streaming builder architecture (Browser::builder()) with support for static HTTP fetching, dynamic JavaScript rendering, anti-bot stealth mode via Headless Chrome, and one-off single page crawling utilities (crawl_single_page).

§Interleaved Queue-Based Streaming Architecture

Pushes starting landing page onto navigation queue, pops tasks one by one to fetch HTML, extract PageIR markdown, execute analysis callbacks, export payloads to disk/storage sinks, scan HTML for same-domain child links, and push discovered tasks back onto the queue until draining completes.

§Quick Stealth Dynamic Rendering Example

use std::time::Duration;
use celestia_spider::{Browser, RenderMode, WaitUntil, TimeoutStrategy};
 
#[tokio::main]
async fn main() -> Result<(), String> {
    let browser = Browser::builder()
        .start_url("https://example.com")
        .render_mode(RenderMode::Dynamic)
        .stealth(true)
        .wait_until(WaitUntil::NetworkIdle)
        .render_timeout(Duration::from_secs(10))
        .timeout_strategy(TimeoutStrategy::ExtractPartial)
        .on_page(|page_ir| async move {
            println!("Analyzing URL: {}", page_ir.url);
            Ok(())
        })
        .build()?;
 
    let summary = browser.run().await?;
    println!("Pages Processed: {}", summary.pages_processed);
    Ok(())
}

§Single Page Crawl Utility Example

use celestia_spider::{crawl_single_page, RenderOptions, RenderMode};

#[tokio::main]
async fn main() -> Result<(), String> {
    let options = RenderOptions {
        render_mode: RenderMode::Static,
        ..Default::default()
    };
    let page_ir = crawl_single_page("https://example.com", &options).await?;
    println!("Title: {}", page_ir.title);
    println!("Markdown IR:\n{}", page_ir.markdown_ir);
    Ok(())
}

§UI / GUI Integration (Tauri, Electron, egui, …)

For UI hosts, CrawlSession is the integration surface: a serde-serializable CrawlConfig, a broadcast CrawlerEvent stream, and a crate::CrawlSessionHandle with cancel / pause / resume / join.

use celestia_spider::{CrawlConfig, CrawlSession, CrawlerEvent};

#[tokio::main]
async fn main() -> Result<(), String> {
    let session = CrawlSession::spawn(CrawlConfig {
        urls: vec!["https://example.com".into()],
        max_depth: 2,
        ..Default::default()
    })?;
    let mut events = session.take_events().unwrap();

    // In a real UI: forward each event to the window (Tauri emit, IPC, ...).
    tokio::spawn(async move {
        while let Ok(event) = events.recv().await {
            if let CrawlerEvent::Finished { summary } = event {
                println!("done: {} pages", summary.results);
                break;
            }
        }
    });

    // From a UI button: session.pause() / session.resume() / session.cancel()
    let summary = session.join().await?;
    println!("visited {} urls", summary.visited_urls.len());
    Ok(())
}

Re-exports§

pub use builder::Browser;
pub use builder::BrowserBuilder;
pub use control::CrawlControl;
pub use engine::common::Crawler;
pub use engine::common::PageFetch;
pub use exporter::FileStorageExporter;
pub use exporter::FileStorageExporterBuilder;
pub use mapper::SiteMapper;
pub use mapper::SiteMapperBuilder;
pub use output::configure_output;
pub use output::StandardWriter;
pub use pipeline::BrowserPipeline;
pub use pipeline::BrowserPipelineBuilder;
pub use pipeline::CrawlerPipeline;
pub use renderer::crawl_single_page;
pub use renderer::PageFetcher;
pub use runner::Runner;
pub use runner::RunnerSummary;
pub use session::CrawlConfig;
pub use session::CrawlSession;
pub use session::CrawlSessionHandle;
pub use session::SessionPhase;
pub use session::SessionSnapshot;
pub use stealth::stealth_chrome_args;
pub use stealth::STEALTH_JS;
pub use transformer::transform_html_to_ir;
pub use types::result::Result as CrawlResult;
pub use types::events::CrawlerEvent;
pub use types::events::SessionSummary;
pub use types::AnalysisCallback;
pub use types::CrawlSummary;
pub use types::Options;
pub use types::PageIR;
pub use types::RenderMode;
pub use types::RenderOptions;
pub use types::Request;
pub use types::Response;
pub use types::SitemapNode;
pub use types::Strategy;
pub use types::StorageExporter;
pub use types::TimeoutStrategy;
pub use types::WaitUntil;

Modules§

builder
control
Cooperative crawl control for UI hosts: external cancellation and pause/resume.
engine
Port of the reference crawler pkg/engine — engine selection and dispatch (standard / headless / hybrid).
exporter
mapper
output
Port of the reference crawler pkg/output — the standard writer handling screen output (verbose decorations + colors), JSONL output (with field exclusion), custom output templates, field selection, per-host field storage, raw response storage, and the error log.
pipeline
renderer
runner
Port of the reference crawler internal/runner — option validation, input parsing (args + stdin), engine execution, ctrl-c handling, health check, and the crawl summary.
session
UI-integration facade: spawn a crawl as a managed background task with a serializable config, a broadcast event stream, and cancel/pause/resume control.
stealth
transformer
types
utils
Utility modules ported from reference crawler pkg/utils and pkg/similarity.