Skip to main content

crawl_stream_handler

essence::api::crawl

Function crawl_stream_handler 

Source 
pub async fn crawl_stream_handler(
    __arg0: Json<CrawlRequest>,
) -> Result<Sse<impl Stream<Item = Result<Event, Infallible>>>, ScrapeError>
Expand description

Handler for POST /api/v1/crawl/stream - SSE streaming endpoint

Streams crawl events as Server-Sent Events (SSE):

  • status: Crawl progress updates
  • document: Completed documents
  • error: Error events for individual URLs
  • complete: Final summary

Example usage with curl:

curl -N -X POST http://localhost:8080/api/v1/crawl/stream \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 50, "max_depth": 2}'