kreuzcrawl 0.2.0

High-performance web crawling engine
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

//! Firecrawl v1-compatible REST API for kreuzcrawl.
//!
//! This module provides an Axum-based HTTP server exposing crawl, scrape, map,
//! and batch operations through a JSON API that follows the Firecrawl v1 wire
//! format.
//!
//! # Endpoints
//!
//! - `POST /v1/scrape` - Scrape a single URL
//! - `POST /v1/crawl` - Start an asynchronous crawl job
//! - `GET  /v1/crawl/{id}` - Poll crawl job status
//! - `DELETE /v1/crawl/{id}` - Cancel a running crawl job
//! - `POST /v1/map` - Discover URLs on a site
//! - `POST /v1/batch/scrape` - Start an asynchronous batch scrape job
//! - `GET  /v1/batch/scrape/{id}` - Poll batch scrape job status
//! - `POST /v1/download` - Download a document from a URL
//! - `GET  /health` - Health check
//! - `GET  /version` - Version information

mod error;
mod handlers;
mod jobs;
pub mod openapi;
mod router;
mod startup;
mod state;
mod types;

pub use startup::serve_with_config;