Module feed 

Feed fetch → parse → sanitize → store pipeline.

This is the module that turns a feed URL into rows in the store. It is deliberately conservative on three axes, because a feed reader ingests hostile, arbitrary web input:

1. Politeness — fetches use a conditional GET (If-None-Match / If-Modified-Since from the stored ETag / Last-Modified), an identifiable crate::USER_AGENT, a request timeout, and a simple exponential backoff hint on error. A 304 Not Modified is a no-op: the feed is untouched apart from bumping its next-poll time.
2. Safety — every entry’s HTML is run through ammonia before it is ever stored. Scripts, event handlers, javascript: URLs, tracking pixels’ dangerous attributes, and other XSS vectors are stripped. Feeds carrying <script> is not hypothetical; treat all feed HTML as untrusted. The reader does not rely on this alone: it re-cleans the stored body with the same sanitize_html at render, through crate::sanitized_html::SanitizedHtml (#151).
3. Robustness — a malformed feed is logged and skipped, never a panic. One bad publisher must not take down the poller. All non-test paths use Result/anyhow; there are no unwrap/expects.

The normalized shape written to the store is the store’s own store::NewFeed / store::NewEntry; dedup is by feed-native GUID via store::insert_entries’s ON CONFLICT (feed_id, guid) upsert.

Structs

Starvation

Whether ingest is being starved of sanitize permits, for /stats and the logs (review of #274).

Enums

FailureKind

Why a poll failed, as a closed set.

FeedKind

What the poller does with a feed row.

FeedPrivacy

The privacy classification of a feed URL — the output of classify_feed_privacy.

PollOutcome

The outcome of polling a single feed. Lets the scheduler decide how to reschedule (and lets tests assert what happened) without inspecting the DB.

Constants

MAX_FAILURE_DETAIL_CHARS

Cap on a failure detail, applied where the string is BUILT.

STARVED_ALERT_SECS

After this long without a free permit, a deferral logs at error level.

Statics

SANITIZE_STARVATION

The production Starvation, which /stats reads.

Functions

backoff_for

Compute the backoff for the nth consecutive failure (1-based), clamped to BACKOFF_MAX. Exponential in the error count so transient blips retry soon while a durably-broken feed backs off toward daily.

build_client

Build a reqwest::Client configured for polite and safe feed fetching.

classify_feed_privacy

Classify whether a feed URL carries a secret credential in the URL itself.

discover_feed

Discover a feed URL from a site’s HTML via <link rel="alternate" type="application/rss+xml|atom+xml" href="…">.

failure_detail

Render an error chain into a bounded PollOutcome::Failed detail.

is_storable_feed_url

Whether a URL may be stored or published as a feed URL at all.

poll_feed

Fetch, parse, sanitize, normalize, and store a single feed.

poll_feed_by_kind

Poll one feed by what it is: an RSS/Atom/JSON document over HTTP, or a standard.site publication read from its author’s PDS.

poll_publication_group

Read several publications from one repo with one walk of its documents (standard_site::fetch_repo), and store each. One outcome per feed, in order. The caller groups by repo; a feed from another repo, or one whose URL is not a publication URI, gets its own failure and does not affect the rest.

settle_poll

Apply a PollOutcome to the feed’s row: settle the error columns AND reschedule it. Both halves, always, from one place.