Expand description
§celestia-spider
A high-performance Rust web browsing library and AI Intermediate Representation (IR) generator.
Built around a streaming builder architecture (Browser::builder()) with support for static HTTP fetching,
dynamic JavaScript rendering, anti-bot stealth mode via Headless Chrome, and one-off single page crawling utilities (crawl_single_page).
§Interleaved Queue-Based Streaming Architecture
Pushes starting landing page onto navigation queue, pops tasks one by one to fetch HTML, extract PageIR markdown,
execute analysis callbacks, export payloads to disk/storage sinks, scan HTML for same-domain child links, and push
discovered tasks back onto the queue until draining completes.
§Quick Stealth Dynamic Rendering Example
use std::time::Duration;
use celestia_spider::{Browser, RenderMode, WaitUntil, TimeoutStrategy};
#[tokio::main]
async fn main() -> Result<(), String> {
let browser = Browser::builder()
.start_url("https://example.com")
.render_mode(RenderMode::Dynamic)
.stealth(true)
.wait_until(WaitUntil::NetworkIdle)
.render_timeout(Duration::from_secs(10))
.timeout_strategy(TimeoutStrategy::ExtractPartial)
.on_page(|page_ir| async move {
println!("Analyzing URL: {}", page_ir.url);
Ok(())
})
.build()?;
let summary = browser.run().await?;
println!("Pages Processed: {}", summary.pages_processed);
Ok(())
}§Single Page Crawl Utility Example
use celestia_spider::{crawl_single_page, RenderOptions, RenderMode};
#[tokio::main]
async fn main() -> Result<(), String> {
let options = RenderOptions {
render_mode: RenderMode::Static,
..Default::default()
};
let page_ir = crawl_single_page("https://example.com", &options).await?;
println!("Title: {}", page_ir.title);
println!("Markdown IR:\n{}", page_ir.markdown_ir);
Ok(())
}§UI / GUI Integration (Tauri, Electron, egui, …)
For UI hosts, CrawlSession is the integration surface: a serde-serializable
CrawlConfig, a broadcast CrawlerEvent stream, and a
crate::CrawlSessionHandle with cancel / pause / resume / join.
use celestia_spider::{CrawlConfig, CrawlSession, CrawlerEvent};
#[tokio::main]
async fn main() -> Result<(), String> {
let session = CrawlSession::spawn(CrawlConfig {
urls: vec!["https://example.com".into()],
max_depth: 2,
..Default::default()
})?;
let mut events = session.take_events().unwrap();
// In a real UI: forward each event to the window (Tauri emit, IPC, ...).
tokio::spawn(async move {
while let Ok(event) = events.recv().await {
if let CrawlerEvent::Finished { summary } = event {
println!("done: {} pages", summary.results);
break;
}
}
});
// From a UI button: session.pause() / session.resume() / session.cancel()
let summary = session.join().await?;
println!("visited {} urls", summary.visited_urls.len());
Ok(())
}Re-exports§
pub use builder::Browser;pub use builder::BrowserBuilder;pub use control::CrawlControl;pub use engine::common::Crawler;pub use engine::common::PageFetch;pub use exporter::FileStorageExporter;pub use exporter::FileStorageExporterBuilder;pub use mapper::SiteMapper;pub use mapper::SiteMapperBuilder;pub use output::configure_output;pub use output::StandardWriter;pub use pipeline::BrowserPipeline;pub use pipeline::BrowserPipelineBuilder;pub use pipeline::CrawlerPipeline;pub use renderer::crawl_single_page;pub use renderer::PageFetcher;pub use runner::Runner;pub use runner::RunnerSummary;pub use session::CrawlConfig;pub use session::CrawlSession;pub use session::CrawlSessionHandle;pub use session::SessionPhase;pub use session::SessionSnapshot;pub use stealth::stealth_chrome_args;pub use stealth::STEALTH_JS;pub use transformer::extract_links;pub use transformer::transform_html_to_ir;pub use types::result::Result as CrawlResult;pub use types::events::CrawlerEvent;pub use types::events::SessionSummary;pub use types::AnalysisCallback;pub use types::CrawlSummary;pub use types::Options;pub use types::PageIR;pub use types::RenderMode;pub use types::RenderOptions;pub use types::Request;pub use types::Response;pub use types::SitemapNode;pub use types::Strategy;pub use types::StorageExporter;pub use types::TimeoutStrategy;pub use types::WaitUntil;
Modules§
- builder
- control
- Cooperative crawl control for UI hosts: external cancellation and pause/resume.
- engine
- Port of the reference crawler
pkg/engine— engine selection and dispatch (standard / headless / hybrid). - exporter
- mapper
- output
- Port of the reference crawler
pkg/output— the standard writer handling screen output (verbose decorations + colors), JSONL output (with field exclusion), custom output templates, field selection, per-host field storage, raw response storage, and the error log. - pipeline
- renderer
- runner
- Port of the reference crawler
internal/runner— option validation, input parsing (args + stdin), engine execution, ctrl-c handling, health check, and the crawl summary. - session
- UI-integration facade: spawn a crawl as a managed background task with a serializable config, a broadcast event stream, and cancel/pause/resume control.
- stealth
- transformer
- types
- utils
- Utility modules ported from reference crawler
pkg/utilsandpkg/similarity.