spider-lib 0.2.0-beta.1

A Rust-based web scraping framework inspired by Scrapy (Python).
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# spider-lib 🕷️🕸️

[![Rust](https://github.com/mzyui/spider-lib/actions/workflows/rust.yml/badge.svg)](https://github.com/mzyui/spider-lib/actions/workflows/rust.yml)

`spider-lib` is an asynchronous web scraping library for Rust, inspired by Scrapy. It features a powerful, modular architecture designed for high-performance data fetching. Currently under active development.

## Architecture

Leverages Rust's async capabilities for efficient I/O and parallel scraping. Employs a modular, actor-based design with core components: `Downloader`, `Scheduler`, and `ItemPipeline` for flexible workflow customization.

## Quick Start

To begin, clone the repository and execute an example:

```bash
cargo run --example books
```

This command initiates a spider to collect book data from a sample website and exports it to a CSV file.

## Contribution

For ideas or bug reports, please open an *issue* or submit a *pull request*.