Crate ahref 

Crawl a website into a portable directed hyperlink graph.

use ahref::{crawl, CrawlOptions, export::{self, Format}};

let graph = crawl("https://example.com", CrawlOptions {
    max_pages: 100,
    concurrency: 3,
    ..Default::default()
}).await?;
export::write(&graph, Format::Json, std::io::stdout())?;

Offline analysis is available through Graph::analyze. HTTP failures become node outcomes; invalid options, invalid graph input and output failures return Error.

Re-exports

pub use crawler::crawl;

pub use crawler::CrawlOptions;

pub use error::Error;

pub use error::Result;

pub use graph::Graph;

pub use graph::LinkEdge;

pub use graph::LinkType;

pub use graph::PageNode;

pub use parser::Parser;

Modules

crawler

error

export

graph

Portable domain types and reproducible graph analysis.

parser

urls

Functions

get_a_tags

Compatibility helper. HTML is serialized by the HTML5 parser.

get_url_from_tags

Compatibility helper returning raw href attributes in document order.