crawler

Crawls websites to gather all possible pages really fast and uses gRPC.

Getting Started

Make sure to have Rust installed or use Docker. This project requires that you start up another gRPC server on port 50051 following proto spec. We are working on adding streams instead of a post hook approach. The server sends all request into another thread and follows up on extracting the links via gRPC callbacks.

1. cargo run or docker compose up

Docker Image

You can use the program as a docker image.

a11ywatch/crawler.

Crate

You can use the crate to setup a gRPC server to run on the machine.

gRPC

In order to use the crawler atm you need to add the grpc client based in the proto location called website.proto. Streams support is in the making to remove the extra need for the client.

LICENSE

Check the license file in the root of the project.