Spider

crate version

Multithreaded web crawler/indexer written in Rust main repo.

Dependencies

On Linux

OpenSSL 1.0.1, 1.0.2, 1.1.0, or 1.1.1

Example

This is a basic blocking example crawling a web page, add spider to your Cargo.toml:

[dependencies]
spider = "1.13.7"

And then the code:

extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://choosealicense.com");
    website.crawl().await;

    for page in website.get_pages() {
        println!("- {}", page.get_url());
    }
}

You can use Configuration object to configure your crawler:

// ..
let mut website: Website = Website::new("https://choosealicense.com");
website.configuration.blacklist_url.push("https://choosealicense.com/licenses/".to_string());
website.configuration.respect_robots_txt = true;
website.configuration.subdomains = true;
website.configuration.delay = 2000; // Defaults to 250 ms
website.configuration.user_agent = "myapp/version".to_string(); // Defaults to spider/x.y.z, where x.y.z is the library version
website.on_link_find_callback = |s| { println!("link target: {}", s); s }; // Callback to run on each link find

website.crawl().await;

Regex Blacklisting

There is an optional "regex" crate that can be enabled:

[dependencies]
spider = { version = "1.13.7", features = ["regex"] }

extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://choosealicense.com");
    website.configuration.blacklist_url.push("/licenses/".to_string());
    website.crawl().await;

    for page in website.get_pages() {
        println!("- {}", page.get_url());
    }
}

Features

Currently we have three optional feature flags. Regex blacklisting and randomizing User-Agents.

[dependencies]
spider = { version = "1.13.7", features = ["regex", "ua_generator"] }

Jemalloc performs better for concurrency and allows memory to release easier.

This changes the global allocator of the program so test accordingly to measure impact.

[dependencies]
spider = { version = "1.13.7", features = ["jemalloc"] }

Blocking