crawler
A gRPC web indexer turbo charged for performance.
This project is capable of handling millions of pages per second efficiently.
Getting Started
Make sure to have Rust installed or Docker.
This project requires that you start up another gRPC server on port 50051
following the proto spec.
The user agent is spoofed on each crawl to a random agent and the indexer extends spider as the base.
cargo run
ordocker compose up
Installation
You can install easily with the following:
Cargo
The crate is available to setup a gRPC server within rust projects.
Docker
You can use also use the docker image at a11ywatch/crawler.
Set the CRAWLER_IMAGE
env var to darwin-arm64
to get the native m1 mac image.
crawler:
container_name: crawler
image: "a11ywatch/crawler:${CRAWLER_IMAGE:-latest}"
ports:
- 50055
Node / Bun
We also release the package to npm @a11ywatch/crawler.
After import at the top of your project to start the gRPC server or run node directly against the module.
import "@a11ywatch/crawler";
Example
This is a basic example crawling a web page, add spider to your Cargo.toml
:
[]
= "0.8.3"
And then the code:
extern crate spider;
use Website;
use tokio;
async
You can use Configuration
object to configure your crawler:
// ..
let mut website: Website = new;
website.configuration.blacklist_url.push;
website.configuration.respect_robots_txt = true;
website.configuration.subdomains = true;
website.configuration.tld = false;
website.configuration.sitemap = false;
website.configuration.proxy = "http://username:password@localhost:1234";
website.configuration.delay = 0; // Defaults to 250 ms
website.configuration.user_agent = "myapp/version".to_string; // Defaults to spider/x.y.z, where x.y.z is the library version
website.crawl.await;
A basic example can also be done with:
One terminal run the server
Another terminal run the client/server
https://user-images.githubusercontent.com/8095978/221221122-cfed83aa-6ca1-4122-a1db-0d9948e9f9d9.mov
Dependencies
In order to build crawler
locally >= 0.5.0, you need the protoc
Protocol Buffers compiler, along with Protocol Buffers resource files.
Ubuntu
proto compiler needs to be at v3 in order to compile. Ubuntu 18+ auto installs.
&&
Alpine Linux
macOS
Assuming Homebrew is already installed. (If not, see instructions for installing Homebrew on the Homebrew website.)
Features
jemalloc
- use jemalloc memory allocator (default disabled).
About
This crawler is optimized for reduced latency and uses isolated based concurrency as it can handle over 10,000 pages within seconds.
In order to receive the links found for the crawler you need to add the website.proto
to your server.
This is required since every request spawns a thread. Isolating the context drastically improves performance (preventing shared resources / communication ).
Help
If you need help implementing the gRPC server to receive the pages or links when found check out the gRPC node example for a starting point .
LICENSE
Check the license file in the root of the project.