Crate progscrape_scrapers

Modules

feed

hacker_news

lobsters

reddit

slashdot

Structs

ExtractedScrapeCollection

Collection of scrape data that has been extracted from a ScrapeCollection.

ScrapeCollection

Collection of scrapes, which can also extract the best title, etc.

ScrapeConfig

Configuration for all scrapers.

ScrapeCore

ScrapeExtractor

ScrapeId

Identify a scrape by source an ID.

ScraperPossibilities

Accumulates the URLs required to scrape for all the services.

Scrapers

StoryDate

Story-specific date that wraps all of the operations we’re interested in. This is a thin wrapper on top of DateTime<Utc> and other chrono utilities for now.

StoryDuration

StoryUrl

Story-specific URL that caches the normalization information and other important parts of the URL.

StoryUrlNorm

TypedScrapeMap

A strongly-typed scrape map that can be used to collect values by scrape source without allocations.

Enums

LegacyError

ScrapeError

ScrapeSource

ScraperHttpResponseInput

ScraperHttpResult

TypedScrape

Functions

import_backup

Import a backup-formatted JSON file, which is a JSON file of TypedScrape object, separated by newlines.