Struct CrawlRequest 

pub struct CrawlRequest { /* private fields */ }

A request a spider wants the crawler to schedule.

Use Output::follow for the common case, or CrawlRequest when you need priority, headers, metadata, or dedup control for a followed URL.

Implementations

impl CrawlRequest

pub fn get(url: impl Into<String>) -> Self

Schedule a GET request for url.

pub fn post(url: impl Into<String>, body: impl Into<Vec<u8>>) -> Self

Schedule a POST request for url.

pub fn url(&self) -> &str

pub fn method_ref(&self) -> &Method

pub fn headers(&self) -> &HeaderMap

pub fn body_bytes(&self) -> Option<&[u8]>

pub fn priority_value(&self) -> i32

pub fn meta_value(&self, key: &str) -> Option<&Value>

pub fn dont_filter_enabled(&self) -> bool

pub fn method(self, method: Method) -> Self

pub fn header(self, name: HeaderName, value: HeaderValue) -> Self

pub fn body(self, body: impl Into<Vec<u8>>) -> Self

pub fn priority(self, priority: i32) -> Self

Higher priority requests are crawled before lower priority requests by the default in-memory frontier.

pub fn meta(self, key: impl Into<String>, value: impl Into<Value>) -> Self

pub fn dont_filter(self, value: bool) -> Self

Bypass URL deduplication for this request.

Trait Implementations

impl Clone for CrawlRequest

fn clone(&self) -> CrawlRequest

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlRequest

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&str> for CrawlRequest

fn from(url: &str) -> Self

Converts to this type from the input type.

impl From<String> for CrawlRequest

fn from(url: String) -> Self

Converts to this type from the input type.