Skip to main content

poll_feed

Function poll_feed 

Source
pub async fn poll_feed(
    pool: &SqlitePool,
    client: &Client,
    feed: &Feed,
    max_entries_per_feed: i64,
) -> Result<PollOutcome>
Expand description

Fetch, parse, sanitize, normalize, and store a single feed.

Performs a conditional GET using the feed’s stored ETag / Last-Modified. On 304 it returns PollOutcome::NotModified without touching entries. On 200 it parses with feed-rs, sanitizes every entry’s HTML with ammonia, upserts the feed row (carrying the fresh validators) and inserts new entries (deduped by GUID). Any fetch/parse error is logged and returned as PollOutcome::Failed — it never panics and never propagates as Err for a merely-broken feed, so one bad publisher can’t stall the scheduler.

Err is reserved for store failures (a broken local DB is a real error the caller should see), not for feed misbehaviour.

max_entries_per_feed caps how many entries this feed retains after insert (newest N by published date); <= 0 disables the per-feed trim.

Bodies are sanitized off the async runtime with a per-entry timeout (SANITIZE_TIMEOUT; see normalize_entries). A poll in which a sanitize timed out does not save the response’s ETag / Last-Modified — the next poll must get a full 200 to store the bodies this one could not, and a saved validator would answer it with 304s until the feed next changed. Such a poll is reported as a FailureKind::Body failure, after storing what it has, so the feed backs off and shows why in /stats: the body was the feed’s. A poll that could not get a sanitize permit at all — every one held by OTHER feeds’ abandoned sanitizes — stores nothing and returns PollOutcome::Deferred, which counts against no one.