Trait scrupy::spider::Spider

pub trait Spider: Send + Sync {
    type ItemType;
    fn name(&self) -> &str;
    fn allowed_domains(&self) -> &[String];
    fn start_urls(&self) -> &[String];
    fn parse(&self, response: Response) -> (Vec<Request>, Vec<Self::ItemType>);

    fn start_requests(&self) -> Vec<Request> { ... }
    fn log(&self, _str: &str) { ... }
}

Associated Types

type ItemType

Required Methods

fn name(&self) -> &str

Returns the name of the spider.

fn allowed_domains(&self) -> &[String]

Returns the allowed domains of the spider.

fn start_urls(&self) -> &[String]

Returns a set of urls for the spider to start with.

fn parse(&self, response: Response) -> (Vec<Request>, Vec<Self::ItemType>)

Parse the Response and get a set of new Requests and items.

Provided Methods

fn start_requests(&self) -> Vec<Request>

Returns a set of start requests for the spider to start with. By default It will call start_urls() to get start urls and issue http get requests to those urls.

fn log(&self, _str: &str)

Logging functions of the spider. By default it will print error to stdout.

Implementors