Trait voyager::Scraper

pub trait Scraper: Sized {
    type Output;
    type State: Debug;
    fn scrape(
        &mut self, 
        response: Response<Self::State>, 
        crawler: &mut Crawler<Self>
    ) -> Result<Option<Self::Output>>;
}

A trait that is takes in successfully fetched responses, scrapes the valuable content from the responses html document and provides the with additional requests to visit and drive the scraper’s model completion.

Associated Types

type Output

The type this scraper eventually produces

type State: Debug

The type used to track the progress of a scraping task that needs several consecutive request.

Required methods

fn scrape(
    &mut self,
    response: Response<Self::State>,
    crawler: &mut Crawler<Self>
) -> Result<Option<Self::Output>>

Is called after the Crawler successfully receives a response for an issued request.

This function can return a finished Output object. The crawler accepts additional requests. To advance the scraper’s State, pass the state of the response along with the request to the Crawler.

Implementors