capp 0.7.2

Common things i use to build Rust CLI tools for web crawlers.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

//! # CAPP - "Comprehensive Asynchronous Parallel Processing" or just "Crawler APP"
//!
//! `capp` is a Rust library designed to provide powerful and flexible tools for building efficient web crawlers and other asynchronous, parallel processing applications. It offers a robust framework for managing concurrent tasks, handling network requests, and processing large amounts of data in a scalable manner.
//!
//! ## Features
//!
//! - **Asynchronous Task Management**: Utilize tokio-based asynchronous processing for efficient, non-blocking execution of tasks.
//! - **Flexible Task Queue**: Use in-memory queues for tests and lightweight runs, and Fjall for persistent task storage.
//! - **Round-Robin Task Distribution**: Ensure fair distribution of tasks across different domains or categories.
//! - **Configurable Workers**: Set up and manage multiple worker instances to process tasks concurrently.
//! - **Error Handling and Retry Mechanisms**: Robust error handling with configurable retry policies for failed tasks.
//! - **Dead Letter Queue (DLQ)**: Automatically move problematic tasks to a separate queue for later analysis or reprocessing.
//! - **Health Checks**: Built-in health check functionality to ensure the stability of your crawling or processing system.
//! - **Extensible Architecture**: Easily extend the library with custom task types, processing logic, and storage backends.
//!
//! ## Use Cases
//!
//! While `capp` is primarily designed for building web crawlers, its architecture makes it suitable for a variety of parallel processing tasks, including:
//!
//! - Web scraping and data extraction
//! - Distributed task processing
//! - Batch job management
//! - Asynchronous API clients
//! - Large-scale data processing pipelines
//!
//! ## Getting Started
//!
//! To use `capp` in your project, add it to your `Cargo.toml`:
//!
//! ```toml
//! [dependencies]
//! capp = "0.5"
//! ```
//!
//! Check examples!
//!
//! ## Modules
//!
//! - `config`: Configuration management for your application.
//! - `healthcheck`: Functions for performing health checks on your system.
//! - `http`: Utilities for making HTTP requests and handling responses.
//! - `manager`: Task and worker management structures.
//! - `queue`: Task queue implementations and traits.
//! - `task`: Definitions and utilities for working with tasks.
pub mod manager;
pub mod prelude;
#[cfg(feature = "cache")]
pub use capp_cache as cache;
pub use capp_config as config;
#[cfg(feature = "http")]
pub use capp_config::backoff;
pub use capp_queue as queue;
#[cfg(feature = "router")]
pub use capp_router as router;
#[cfg(feature = "http")]
pub use config::http;
#[cfg(feature = "observability")]
pub mod observability;
#[cfg(feature = "stats-http")]
pub mod stats_http;
#[cfg(feature = "http")]
pub use reqwest;
pub use tower;
// re-export
pub use async_trait;
pub use rand;
pub use serde;
pub use serde_json;
pub use thiserror;
pub use toml;
pub use tracing;
pub use tracing_subscriber;
pub use uuid;