1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
//! One shared, TTL-cached `eth_blockNumber` read for every chain watcher.
//!
//! Without a shared read, every tick would cost one `eth_blockNumber` *per
//! watcher* — ~7 head reads per `event_poll_interval` against a single endpoint,
//! on top of each watcher's `eth_getLogs`. [`SharedHead`] collapses them: the
//! first caller within a TTL window issues the RPC, everyone else reads the
//! cache.
//!
//! Two properties matter:
//!
//! - **Monotonicity — why a stale head is safe.** [`SharedHead`] only ever
//! returns a head at-or-before the true head at the call instant; it can never
//! report a head *ahead* of the chain. Every consumer of the head in
//! `resumable_watcher` clamps *against* it (`resolve_persisted_start`,
//! `resolve_head_window_start`, and the `from > to` idle branch), so a head
//! that lags only ever *widens* an already-idempotent rescan or defers work to
//! the next tick — it can never open a gap. Note this is a property of the
//! cache direction, not a coincidence: a cache that could run *ahead* would
//! skip blocks, which is exactly the #751/#762 hazard (anchoring a durable
//! checkpoint floor too high). The cost is bounded observation latency (≤ TTL),
//! not correctness.
//! - **Single-flight, including failures.** The mutex is held across the RPC, so
//! concurrent callers queue and then read the fresh entry rather than each
//! issuing their own read. Failures are cached for the same TTL, which is what
//! keeps a wedged provider from serializing timeouts: without it, N watchers
//! queued behind a stalled endpoint would each wait the full
//! `chain_events::DEFAULT_RPC_CALL_TIMEOUT` in turn, so the last one's tick — and
//! therefore its `*_backoff_started` gauge — would be delayed by N × 10s. With
//! it, one caller pays the timeout and the rest fail instantly, so every
//! watcher enters backoff at the same moment.
use Arc;
use Duration;
use BlockId;
use Address;
use Provider;
use Result;
use async_trait;
use Instant;
use info;
use timed;
/// A source of the current chain head block number.
///
/// Implementations MAY serve from a short TTL cache. The returned value is always
/// at-or-before the true head at the call instant and never ahead of it — callers
/// may rely on that direction (see the module doc), but MUST NOT assume the value
/// is exactly current. A caller needing the head for a *deadline* decision ("has
/// block X been reached yet?") rather than a scan upper bound would be delayed by
/// up to the TTL; no watcher does that today.
///
/// `Err` keeps the provider's typed cause in its chain. A watcher tick treats
/// every `Err` as retryable and fails into backoff; a boot read classifies it
/// (`boot_retry::is_permanent_boot_error`), so a deterministic failure such as a
/// rejected API key fails boot at once.
/// How far below the reported head an enumeration snapshot pins its reads.
///
/// A load-balanced RPC reports the head of its freshest upstream, then routes
/// each call to any upstream. An `eth_call` pinned to that exact block fails on
/// every upstream that has not reached it yet, while an older block resolves on
/// all of them. So the reported head is not a block every upstream can serve.
/// 256 blocks is about 64s on Arbitrum, above the 40–200 block upstream lag seen
/// behind a hosted balancer (#2164).
///
/// The margin is a block count, sized for Arbitrum's ~250ms blocks and a
/// provider that serves state that far back. A full node that prunes state
/// within 256 blocks (geth keeps 128) answers every pinned read with a
/// transient error, so a boot against one spends its retry budget.
///
/// The margin costs no correctness. A boot enumeration seeds its tail cursor at
/// the snapshot block and every sink is idempotent, so the first tail tick
/// replays `[snapshot, head]` as a harmless overlap. A periodic re-enumeration
/// keeps every entry the tail changed after the snapshot block (see each
/// watcher's fold).
pub const SNAPSHOT_LAG_MARGIN_BLOCKS: u64 = 256;
/// The block an enumeration of `contract` pins its reads to:
/// [`SNAPSHOT_LAG_MARGIN_BLOCKS`] below the head.
///
/// A contract with no code at that block was deployed inside the margin — a
/// fresh local chain, or a boot right after a deploy. Reads pinned there would
/// decode an empty return, so the snapshot takes the head instead. That head pin
/// is exposed to upstream lag again; a failed read retries, and the lagged pin
/// returns once the chain is `SNAPSHOT_LAG_MARGIN_BLOCKS` past the deploy.
pub async
/// A cached head read plus the instant it was taken.
/// A failed head read, shared by every caller inside the TTL.
///
/// The failure is kept whole behind an `Arc` (`anyhow::Error` is not `Clone`)
/// and exposed as this error's [`source`](std::error::Error::source), so a
/// caller's `err.chain()` still reaches the typed cause — the provider's
/// `TransportError` — and can classify it.
;
/// TTL-cached, single-flight [`HeadSource`] over one provider.
/// Hand-written rather than derived: `P` is a provider and carries no `Debug`
/// bound, and the cached entry sits behind an async mutex this must not block on.