Expand description
Fleet mode: several servers sharing one Postgres.
The address store is the easy half (see crate::store). This is the
other half: health. Two tables, because health holds two kinds of fact.
instance_health: one row per instance, its wholeHealthStoresnapshot plusupdated_at(stamped by the database clock, so instance clock skew cannot make a dead peer look alive). Each row has exactly one writer, so no locking is needed. Rows older thanstale_afterare ignored: a crashed or scaled-down instance stops voting by itself.provider_quarantine: global. Schema drift means the adapter is wrong everywhere it runs, so the first instance to see it inserts a row (later inserts are no-ops), every instance adopts it, and deleting the row (DELETE /v1/providers/{name}/quarantineon any instance) clears it fleet-wide.
§The routing rule
cowbird wrote per-instance rows but never read peers’ back, so every
instance paid its own failed requests (and timeouts) to learn that a
provider had died. Here peers’ fresh rows feed routing through a
TierOverlay, under a rule chosen so a peer can save us a failed call
but can never cost us much:
- Peers only demote. If this instance excludes a provider, it stays excluded; no peer can route us to something we saw fail.
- Each informed instance votes, this one included. An entry votes
down while it is DOWN inside its cooldown (
retry_atin the future), and up when it is OK or SLOW. Everything else abstains: no entry, probation (cooldown over, outcome unknown), quarantine (the global table handles that), and evidence older thanEVIDENCE_HORIZON. - Egress-specific failures abstain. A DOWN caused by a challenge
(
needs_residential_ip) is about that instance’s IP; fan-out exists to have several, so it must not take the provider away from the others. - Strict majority demotes; ties route. The provider leaves routing
while down votes outnumber up votes. A peer that just saw a failure
keeps an uninformed instance from repeating it; a single peer cannot
overrule an instance that saw a success. Being wrong toward “up” costs
one failed call (the local breaker then trips); being wrong toward
“down” is bounded by the voters’ own cooldowns (at most 15 minutes).
Down votes are re-checked against their
retry_atat routing time, so a peer’s verdict lapses on schedule even between syncs.
Peers’ votes never enter this instance’s snapshot, so a verdict cannot echo around the fleet and outlive the measurement behind it.
Structs§
- Health
Sync - Publishes this instance’s health, reads its peers’, and keeps quarantine in step with the global table.
- Peer
View - Peers’ votes as of the last sync, consulted by
HealthStore::tier.
Enums§
Constants§
- EVIDENCE_
HORIZON - Evidence older than this abstains. The longest a DOWN can last, so an old success cannot outvote a failure that is still inside its cooldown for longer than that failure could have lasted anyway.
Functions§
- decide
- The rule in the module docs: the tier to route at, given the local entry, the local verdict and the peers’ votes.
- peer_
votes - Every fresh peer’s votes, by provider.
- vote
- What one instance’s entry says about a provider;
Noneabstains.