Module website 

A website to crawl.

Structs

ChannelGuard

Guard a channel from closing until all concurrent operations are done.

DEFAULT_PERMITS

The default Semaphore limits.

Website

Represents a website to crawl and gather all links or page content.

Enums

CrawlStatus

the active status of the crawl.

CronType

The type of cron job to run

OnShouldCrawlCallback

Callback closure or function pointer that determines if a link should be crawled or not.

ProcessLinkStatus

The link activity for the crawl.

WebsiteMetaInfo

Generic website meta info for handling retries.

Traits

OnShouldCrawlClosure

Callback closure that determines if a link should be crawled or not.

Functions

calc_limits

calculate the base limits

channel_send_page

Channel broadcast send the Page to receivers.

is_safe_javascript_challenge

check if the page is a javascript challenge

set_interface

Bind connections only on the specified network interface.

Type Aliases

OnLinkFindCallback

On link find callback rewrite a url if it meets a condition.