spider-core 2.0.0

Core functionality for the spider-lib web scraping framework.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

# spider-core

Core crawling engine for `spider-lib`: spider trait, crawler runtime, scheduler, builder, state, and stats.

Most users should start with `spider-lib`. Use `spider-core` directly when you want lower-level control over runtime composition.

## Installation

```toml
[dependencies]
spider-core = "2.0.0"
```

## Main Components

- `Spider`: trait for crawl logic.
- `Crawler`: runtime engine that drives requests and parsing.
- `CrawlerBuilder`: runtime configuration and composition.
- `Scheduler`: request queueing and dedup behavior.
- `CrawlerState`: shared runtime state.
- `StatCollector`: runtime statistics.

## Minimal Usage

```rust,ignore
use spider_core::{async_trait, CrawlerBuilder, Spider};
use spider_util::{error::SpiderError, item::ParseOutput, response::Response};

#[spider_macro::scraped_item]
struct Item {
    title: String,
}

#[derive(Clone, Default)]
struct State;

struct MySpider;

#[async_trait]
impl Spider for MySpider {
    type Item = Item;
    type State = State;

    fn start_urls(&self) -> Vec<&'static str> {
        vec!["https://example.com"]
    }

    async fn parse(
        &self,
        _response: Response,
        _state: &Self::State,
    ) -> Result<ParseOutput<Self::Item>, SpiderError> {
        Ok(ParseOutput::new())
    }
}

```

## Feature Flags

- `core` (default)
- `live-stats`: enables in-place terminal stat updates.
- `checkpoint`: enables checkpoint/resume support.
- `cookie-store`: enables `cookie_store` integration.

```toml
[dependencies]
spider-core = { version = "2.0.0", features = ["checkpoint"] }
```

## Custom Extension Guides

For extension points built around crawler composition, see:

- Custom downloader guide: [`spider-downloader`](../spider-downloader/README.md)
- Custom middleware guide: [`spider-middleware`](../spider-middleware/README.md)
- Custom pipeline guide: [`spider-pipeline`](../spider-pipeline/README.md)

## Related Crates

- [`spider-lib`](../README.md)
- [`spider-downloader`](../spider-downloader/README.md)
- [`spider-middleware`](../spider-middleware/README.md)
- [`spider-pipeline`](../spider-pipeline/README.md)
- [`spider-util`](../spider-util/README.md)

## License

MIT. See [LICENSE](../LICENSE).