spider_worker-1.26.0 is not a library.
Spider Worker
A spider worker to decentralize the crawl lifting.
Dependencies
This project depends on the spider crate.
Usage
The worker starts on port 3030 by default.
cargo run
Feature Flags
scrape
- When the html is needed run the instance with the flag. Requires spider feature flag matching on the client to start. This also starts the instance on port 3031 instead.all
- Start the basic worker to gather links and scraper together.
Ports
By default the instance runs on port 3030
.
The scraper runs on port 3031
when enabled.
Todo
- Allow port configuration.