Skip to main content

Module fleet

Module fleet 

Source
Expand description

Fleet mode: several servers sharing one Postgres.

The address store is the easy half (see crate::store). This is the other half: health. Two tables, because health holds two kinds of fact.

  • instance_health: one row per instance, its whole HealthStore snapshot plus updated_at (stamped by the database clock, so instance clock skew cannot make a dead peer look alive). Each row has exactly one writer, so no locking is needed. Rows older than stale_after are ignored: a crashed or scaled-down instance stops voting by itself.
  • provider_quarantine: global. Schema drift means the adapter is wrong everywhere it runs, so the first instance to see it inserts a row (later inserts are no-ops), every instance adopts it, and deleting the row (DELETE /v1/providers/{name}/quarantine on any instance) clears it fleet-wide.

§The routing rule

cowbird wrote per-instance rows but never read peers’ back, so every instance paid its own failed requests (and timeouts) to learn that a provider had died. Here peers’ fresh rows feed routing through a TierOverlay, under a rule chosen so a peer can save us a failed call but can never cost us much:

  1. Peers only demote. If this instance excludes a provider, it stays excluded; no peer can route us to something we saw fail.
  2. Each informed instance votes, this one included. An entry votes down while it is DOWN inside its cooldown (retry_at in the future), and up when it is OK or SLOW. Everything else abstains: no entry, probation (cooldown over, outcome unknown), quarantine (the global table handles that), and evidence older than EVIDENCE_HORIZON.
  3. Egress-specific failures abstain. A DOWN caused by a challenge (needs_residential_ip) is about that instance’s IP; fan-out exists to have several, so it must not take the provider away from the others.
  4. Strict majority demotes; ties route. The provider leaves routing while down votes outnumber up votes. A peer that just saw a failure keeps an uninformed instance from repeating it; a single peer cannot overrule an instance that saw a success. Being wrong toward “up” costs one failed call (the local breaker then trips); being wrong toward “down” is bounded by the voters’ own cooldowns (at most 15 minutes). Down votes are re-checked against their retry_at at routing time, so a peer’s verdict lapses on schedule even between syncs.

Peers’ votes never enter this instance’s snapshot, so a verdict cannot echo around the fleet and outlive the measurement behind it.

Structs§

HealthSync
Publishes this instance’s health, reads its peers’, and keeps quarantine in step with the global table.
PeerView
Peers’ votes as of the last sync, consulted by HealthStore::tier.

Enums§

Vote

Constants§

EVIDENCE_HORIZON
Evidence older than this abstains. The longest a DOWN can last, so an old success cannot outvote a failure that is still inside its cooldown for longer than that failure could have lasted anyway.

Functions§

decide
The rule in the module docs: the tier to route at, given the local entry, the local verdict and the peers’ votes.
peer_votes
Every fresh peer’s votes, by provider.
vote
What one instance’s entry says about a provider; None abstains.