Expand description
Multiplexed eth_getLogs polling: one poll tick, one get_logs call per
backfill window, demuxed by (address, topic0) into per-route sinks.
resumable_watcher gives every watcher its own
cursor loop and its own eth_getLogs call each tick. That is right when a
watcher’s filter genuinely differs from its siblings’, but five watchers on
this node scan disjoint (address, topic0) slices of the same few
contracts — so five independent loops cost five eth_getLogs calls per tick
where one merged call, demuxed after the fact, would do. MultiplexedPoller
is that merge: one filter, one eth_getLogs per window, fanned out by
(address, topic0) into each watcher’s own [ErasedSink].
Three properties carry over from resumable_watcher unchanged:
- Resumability and idempotency (see that module’s doc) — the cursor
machinery ([
CursorStart::seed]/[CursorStart::initial_from]/ [CursorStart::persist]/[CursorStart::flush]) is reused verbatim, one instance per route. - Per-route cursors, not one shared cursor. Merging the RPC call does
not merge the cursors: each route floor derives, advances, and persists
independently, scanned as
[min(route floors), head]in one call. This is what preserves settlement’s deepFromCheckpointbackfill without dragging every sibling route through the same history, and what makes per-route error isolation possible below. - Per-route error isolation. A route whose sink errors is marked
erroredfor the rest of the tick: it holds its cursor (re-scanning the same range next tick) while sibling routes keep advancing and persisting. The tick as a whole still returnsErrwhen any route errored, so the loop backs off — a route that is failing should not poll as fast as one that is not — but that backoff never discards a healthy sibling’s progress.
§Hooks fire per route, not per loop
A single-sink loop would fire on_backoff/on_established once per tick,
because there is exactly one sink. Here the tick fires each route’s hooks
independently (routes fail independently), and the loop’s Err arm only
sleeps the backoff — it never fires a hook itself. The exception is a
failure in the shared window scan: the head read, a route’s checkpoint-load
floor derivation, or a merged get_logs call that is not retried, all
before any route-specific step runs; or a stall past
[GET_LOGS_STALL_BUDGET], after the tick’s completed windows are applied
and persisted but before the reconcile. Those fail the whole tick, so
[fail_whole_tick] fires on_backoff for every route directly at the
failure site — every route is equally down, as five independent watchers
that each read the shared head through
super::shared_head::SharedHead would all convoy into backoff together.
The on_established/on_backoff edges are suppressed once shutdown is
cancelled (a tick that only “succeeded” because a sink observed the cancel
token must not flip a readiness gate open), while on_tick_success keeps
stamping unconditionally.
§Provider range caps
A get_logs failure that [is_range_rejection] recognises does not fail the
tick. [run_tick] sets the window span to half the rejected window’s length
and retries the same start block at once; no route hook fires. The span
doubles back after a run of accepted windows ([WindowSpan]), so a
transient rejection does not cost throughput until restart. A rejected one-block
window, and any other get_logs failure that is not retried below, goes
through [fail_whole_tick].
§Transient window failures
A tick that is behind head scans several windows, and it reaches head only
if every window succeeds. A provider that fails a fraction of its calls
would therefore stop most long ticks short of head. So [run_tick] retries a
failed window in place:
a get_logs error that [is_transient_window_error] accepts is retried on
the same block range up to [GET_LOGS_WINDOW_RETRIES] times, after
[GET_LOGS_WINDOW_RETRY_DELAY] each. A retry that succeeds is a plain
success: no route’s on_backoff fires and no route is marked down. Each
retry logs its cause at info and fires the on_window_retry hook (the
decdn_chain_get_logs_retries_total counter), so an unreliable provider
stays visible although the retries keep it out of watcher downtime.
Three kinds of failure are not retried in place. A range rejection takes the
shrink path above. An error that [is_permanent_rpc_error] calls
deterministic fails the same way on every attempt. A rate limit
([is_rate_limit]) goes to the loop’s exponential backoff, because a
fixed-delay retry spends more of a quota that is already used up.
The retries lengthen a failing window. A provider that hangs costs each
window GET_LOGS_WINDOW_RETRIES + 1 per-call timeouts plus the retry
sleeps (34 s at the 10 s default timeout) before the window is deferred.
§Deferred windows
A provider’s transient failures cluster in time, so a window that fails
every retry often fails again a few seconds later. Failing the tick then
would mark every route down and count a watcher restart for an outage that
loses no event. So [run_tick] defers such a window instead: it ends the
tick there, keeps the progress of the windows before it, and the next tick
resumes at the deferred window’s start. Each deferral logs its cause at
info and fires the on_window_deferred hook (the
decdn_chain_get_logs_deferred_total counter).
A deferred tick that completed a window is a success that ended early: it
reconciles and fires every route’s hooks. A deferred tick that completed no
window fires no hook and skips the reconcile, so no tick gauge, freshness
stamp or on_tick_complete backstop moves. A stall
clock starts at the first deferred tick and clears only when a tick reaches
head, so progress that keeps falling behind head cannot hide behind the
early successes. A deferred tick fails through [fail_whole_tick] once no
tick has reached head for [GET_LOGS_STALL_BUDGET].
The runtime registers each eth_getLogs watcher’s Route on one
poller in build_chain_and_handlers and spawns it once — one merged loop in
place of five independent per-watcher loops (four when the fee-shares
route is absent: it registers only when the startup feeRouter() read
succeeds).
This module is pub only so Route can appear in the pub watcher
bootstrap signatures and the external settlement e2e can drive spawn; its
docs still reference the crate-internal collaborators (ErasedSink,
CursorStart, the resumable_watcher cursor vocabulary), so intra-doc links
to those private items are allowed here rather than downgraded to prose.
Structs§
- Multiplexed
Poller - Poller configuration, its resolved routes, and the demux index built once
at
MultiplexedPollerBuilder::build. - Multiplexed
Poller Builder - Accumulates
Routes and builds aMultiplexedPoller. - Route
- One registered watcher on the multiplexed poller.
addresses/topic0sare its demux keys; every(address, topic0)pair in the cartesian product is a route key, and every route key must be globally unique across the poller’s routes (MultiplexedPollerBuilder::buildchecks this). Each existing watcher is single-address, soaddressesis a 1-element vec today, but the field is a set so a future multi-address watcher needs no reshape.
Functions§
- spawn
- Mint the shutdown token, spawn [
run], and return the owningWatcherHandle, the only pairing of a live task with its shutdown token (#1236).