Skip to main content

Module multiplexed_poller

Module multiplexed_poller 

Source
Expand description

Multiplexed eth_getLogs polling: one poll tick, one get_logs call per backfill window, demuxed by (address, topic0) into per-route sinks.

resumable_watcher gives every watcher its own cursor loop and its own eth_getLogs call each tick. That is right when a watcher’s filter genuinely differs from its siblings’, but five watchers on this node scan disjoint (address, topic0) slices of the same few contracts — so five independent loops cost five eth_getLogs calls per tick where one merged call, demuxed after the fact, would do. MultiplexedPoller is that merge: one filter, one eth_getLogs per window, fanned out by (address, topic0) into each watcher’s own [ErasedSink].

Three properties carry over from resumable_watcher unchanged:

  • Resumability and idempotency (see that module’s doc) — the cursor machinery ([CursorStart::seed]/[CursorStart::initial_from]/ [CursorStart::persist]/[CursorStart::flush]) is reused verbatim, one instance per route.
  • Per-route cursors, not one shared cursor. Merging the RPC call does not merge the cursors: each route floor derives, advances, and persists independently, scanned as [min(route floors), head] in one call. This is what preserves settlement’s deep FromCheckpoint backfill without dragging every sibling route through the same history, and what makes per-route error isolation possible below.
  • Per-route error isolation. A route whose sink errors is marked errored for the rest of the tick: it holds its cursor (re-scanning the same range next tick) while sibling routes keep advancing and persisting. The tick as a whole still returns Err when any route errored, so the loop backs off — a route that is failing should not poll as fast as one that is not — but that backoff never discards a healthy sibling’s progress.

§Hooks fire per route, not per loop

A single-sink loop would fire on_backoff/on_established once per tick, because there is exactly one sink. Here the tick fires each route’s hooks independently (routes fail independently), and the loop’s Err arm only sleeps the backoff — it never fires a hook itself. The exception is a failure in the shared window scan: the head read, a route’s checkpoint-load floor derivation, or a merged get_logs call that is not retried, all before any route-specific step runs; or a stall past [GET_LOGS_STALL_BUDGET], after the tick’s completed windows are applied and persisted but before the reconcile. Those fail the whole tick, so [fail_whole_tick] fires on_backoff for every route directly at the failure site — every route is equally down, as five independent watchers that each read the shared head through super::shared_head::SharedHead would all convoy into backoff together.

The on_established/on_backoff edges are suppressed once shutdown is cancelled (a tick that only “succeeded” because a sink observed the cancel token must not flip a readiness gate open), while on_tick_success keeps stamping unconditionally.

§Provider range caps

A get_logs failure that [is_range_rejection] recognises does not fail the tick. [run_tick] sets the window span to half the rejected window’s length and retries the same start block at once; no route hook fires. The span doubles back after a run of accepted windows ([WindowSpan]), so a transient rejection does not cost throughput until restart. A rejected one-block window, and any other get_logs failure that is not retried below, goes through [fail_whole_tick].

§Transient window failures

A tick that is behind head scans several windows, and it reaches head only if every window succeeds. A provider that fails a fraction of its calls would therefore stop most long ticks short of head. So [run_tick] retries a failed window in place: a get_logs error that [is_transient_window_error] accepts is retried on the same block range up to [GET_LOGS_WINDOW_RETRIES] times, after [GET_LOGS_WINDOW_RETRY_DELAY] each. A retry that succeeds is a plain success: no route’s on_backoff fires and no route is marked down. Each retry logs its cause at info and fires the on_window_retry hook (the decdn_chain_get_logs_retries_total counter), so an unreliable provider stays visible although the retries keep it out of watcher downtime.

Three kinds of failure are not retried in place. A range rejection takes the shrink path above. An error that [is_permanent_rpc_error] calls deterministic fails the same way on every attempt. A rate limit ([is_rate_limit]) goes to the loop’s exponential backoff, because a fixed-delay retry spends more of a quota that is already used up.

The retries lengthen a failing window. A provider that hangs costs each window GET_LOGS_WINDOW_RETRIES + 1 per-call timeouts plus the retry sleeps (34 s at the 10 s default timeout) before the window is deferred.

§Deferred windows

A provider’s transient failures cluster in time, so a window that fails every retry often fails again a few seconds later. Failing the tick then would mark every route down and count a watcher restart for an outage that loses no event. So [run_tick] defers such a window instead: it ends the tick there, keeps the progress of the windows before it, and the next tick resumes at the deferred window’s start. Each deferral logs its cause at info and fires the on_window_deferred hook (the decdn_chain_get_logs_deferred_total counter).

A deferred tick that completed a window is a success that ended early: it reconciles and fires every route’s hooks. A deferred tick that completed no window fires no hook and skips the reconcile, so no tick gauge, freshness stamp or on_tick_complete backstop moves. A stall clock starts at the first deferred tick and clears only when a tick reaches head, so progress that keeps falling behind head cannot hide behind the early successes. A deferred tick fails through [fail_whole_tick] once no tick has reached head for [GET_LOGS_STALL_BUDGET].

The runtime registers each eth_getLogs watcher’s Route on one poller in build_chain_and_handlers and spawns it once — one merged loop in place of five independent per-watcher loops (four when the fee-shares route is absent: it registers only when the startup feeRouter() read succeeds).

This module is pub only so Route can appear in the pub watcher bootstrap signatures and the external settlement e2e can drive spawn; its docs still reference the crate-internal collaborators (ErasedSink, CursorStart, the resumable_watcher cursor vocabulary), so intra-doc links to those private items are allowed here rather than downgraded to prose.

Structs§

MultiplexedPoller
Poller configuration, its resolved routes, and the demux index built once at MultiplexedPollerBuilder::build.
MultiplexedPollerBuilder
Accumulates Routes and builds a MultiplexedPoller.
Route
One registered watcher on the multiplexed poller. addresses/topic0s are its demux keys; every (address, topic0) pair in the cartesian product is a route key, and every route key must be globally unique across the poller’s routes (MultiplexedPollerBuilder::build checks this). Each existing watcher is single-address, so addresses is a 1-element vec today, but the field is a set so a future multi-address watcher needs no reshape.

Functions§

spawn
Mint the shutdown token, spawn [run], and return the owning WatcherHandle, the only pairing of a live task with its shutdown token (#1236).