codoseo-crawler 0.0.1

The CodoSEO crawler: fetching, robots.txt, sitemaps and streaming field extraction.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

//! The CodoSEO crawler. No database: the CLI, local MCP and the worker all use it directly.

pub use codoseo_core::output::*;

pub mod crawl;
pub mod extract;
pub mod fetch;
pub mod frontier;
pub mod guard;
pub mod politeness;
pub mod preflight;
pub(crate) mod record;
pub mod robots;
pub mod scope;
pub mod sitemap;