Crate ingester 

Web scraping and data ingestion.

Implement Endpoint for a type that describes your data source. The endpoint makes the next request and parses each response into items. The endpoint sees the previous response, thus cursor pagination is possible. Give the endpoint to an Ingester with an HttpClient and a RateLimit. Then read the items from the stream that the ingester makes.

use ingester::{Endpoint, Error, Request, Response, ingest};

struct StoryIds {
    page: u32,
    max_pages: Option<u64>,
}

impl Endpoint for StoryIds {
    type Item = u64;

    fn next_request(&mut self, last: Option<&Response>) -> Option<Request> {
        if let Some(resp) = last {
            self.max_pages = resp.json::<serde_json::Value>().ok()?["nbPages"].as_u64();
        }
        if self.max_pages.is_some_and(|max| u64::from(self.page) >= max) {
            return None;
        }
        let url = format!(
            "https://hn.algolia.com/api/v1/search?tags=front_page&page={}",
            self.page
        );
        self.page += 1;
        Some(Request::get(url.parse().unwrap()))
    }

    fn parse(&self, response: &Response) -> Result<Vec<u64>, Error> {
        let body: serde_json::Value = response.json()?;
        Ok(body["hits"]
            .as_array()
            .into_iter()
            .flatten()
            .filter_map(|hit| hit["objectID"].as_str()?.parse().ok())
            .collect())
    }
}

let stories = ingest(StoryIds { page: 0, max_pages: None })
    .collect()
    .await?;

Re-exports

pub use client::HttpClient;

pub use rate_limit::NoRateLimit;

pub use rate_limit::RateLimit;

Modules

client

rate_limit

Structs

Ingester

Operates an Endpoint: sends its requests through an HttpClient, obeys a RateLimit, and parses the responses into a stream of items.

Request

One HTTP request that an Endpoint makes.

Reqwest

The reqwest client, re-exported for use as an HttpClient backend: HttpClient<Reqwest>.

Response

The response to one Request.

Enums

Error

The errors that can occur during the ingestion of an Endpoint.

Traits

Endpoint

Your description of the procedure that ingests one endpoint.

Functions

ingest

Start ingestion of an endpoint with the default configuration: an HttpClient that uses Reqwest, and no rate limit. Use Ingester::with_client and Ingester::with_rate_limit to change the configuration.

Type Aliases

Governor

The default direct rate limiter of the governor crate, re-exported for use as a RateLimit backend: RateLimit<Governor>.