cc_downloader/lib.rs
1//! A polite and user-friendly client for downloading [Common Crawl](https://commoncrawl.org) data.
2//!
3//! The library implements a two-step workflow that mirrors the CLI:
4//!
5//! 1. **Fetch the index** — download a `.paths.gz` file that lists every file
6//! in a crawl snapshot for a given data type.
7//! 2. **Download the data** — read that index and download every file it lists,
8//! concurrently, with exponential-backoff retries.
9//!
10//! All configuration for both steps is expressed through [`download::DownloadOptions`].
11//!
12//! # Quick start
13//!
14//! ```no_run
15//! use cc_downloader::download::{DownloadOptions, download_paths, download};
16//!
17//! #[tokio::main]
18//! async fn main() -> Result<(), cc_downloader::errors::DownloadError> {
19//! // Step 1: fetch the paths index for WET files from a crawl snapshot.
20//! let paths_options = DownloadOptions {
21//! snapshot: "CC-MAIN-2024-46".to_string(),
22//! data_type: "wet",
23//! dst: std::path::Path::new("./output"),
24//! ..Default::default()
25//! };
26//! download_paths(paths_options).await?;
27//!
28//! // Step 2: download every file listed in the index.
29//! let download_options = DownloadOptions {
30//! paths: std::path::Path::new("./output/wet.paths.gz"),
31//! dst: std::path::Path::new("./output"),
32//! threads: 10,
33//! progress: true,
34//! ..Default::default()
35//! };
36//! download(download_options).await?;
37//!
38//! Ok(())
39//! }
40//! ```
41//!
42//! # Contributor datasets
43//!
44//! For datasets hosted under `https://data.commoncrawl.org/contrib/` that don't
45//! follow the standard snapshot/data-type schema, use
46//! [`download::download_contrib_paths`] instead of [`download::download_paths`]:
47//!
48//! ```no_run
49//! use cc_downloader::download::download_contrib_paths;
50//!
51//! # #[tokio::main]
52//! # async fn main() -> Result<(), cc_downloader::errors::DownloadError> {
53//! download_contrib_paths(
54//! "https://data.commoncrawl.org/contrib/my-dataset/paths.gz",
55//! std::path::Path::new("./output"),
56//! 1000,
57//! ).await?;
58//! # Ok(())
59//! # }
60//! ```
61//!
62//! # Retries and politeness
63//!
64//! Every HTTP request is wrapped in an exponential-backoff retry policy (bounds:
65//! 1 s – 1 h, bounded jitter, base 2). The default of 1 000 retries is intentionally
66//! high: Common Crawl is a shared public resource and transient failures are common
67//! under load. Reduce [`download::DownloadOptions::max_retries`] only if you have a
68//! specific reason to fail fast.
69//!
70//! # Error handling
71//!
72//! All public functions return [`errors::DownloadError`], which wraps the
73//! underlying error from whichever subsystem failed. See that type for the full
74//! list of variants and a usage example.
75
76/// Core download functionality: path resolution, file downloading, and subset filtering.
77pub mod download;
78
79/// Error types returned by all fallible operations in this library.
80pub mod errors;