decdn_node/blacklist_watcher.rs
1//! Blacklist compliance watcher (ADR 011 § Content Takedown).
2//!
3//! Every paid-delivery node runs this task after resolving the mandatory
4//! `blockchain.content_blacklist_address`. It keeps the local blob store
5//! compliant with `ContentBlacklist`: it evicts any blob whose hash is
6//! blacklisted *in scope* for this operator (global ∪ current-region ∪
7//! ripening-prev-region). The scope decision is the contract's
8//! `isHashBlacklistedForOperator` view, so region packing and the ADR 030
9//! ripening math never leave the chain.
10//!
11//! **Enumerated deny-set, re-scoped.** The set of blacklisted entries is read
12//! straight from `ContentBlacklist`'s enumeration views at one pinned block on
13//! boot, rather than replayed from `HashBlacklisted` logs (#1497). Two views feed
14//! two sets:
15//!
16//! - **Hash entries.** `getScopeRegions(operator)` gives the one-to-three region
17//! keys in scope right now; each region's set is paged from
18//! `blacklistedHashes` / `blacklistedHashCount`. Entries are keyed by
19//! `(region, hash)` — the contract's own key (`_hashEntries[region][hash]`) —
20//! so a same-hash entry under another region survives a `HashRemoved` for the
21//! first. Every seen-but-not-yet-evicted entry is retained in `known` and
22//! re-scoped on a periodic (`on_tick_complete`) pass, because a hash can become
23//! live + in scope with **no** `HashBlacklisted` event, via an operator
24//! region/ripening change (`CapacityBond.updateRegion`), which emits nothing on
25//! this contract at all.
26//! - **Address union.** `blacklistedAddresses` / `blacklistedAddressCount`
27//! enumerate the origin ∪ operator deny set. Each address is liveness-filtered
28//! by the contract's own `isOriginBlacklisted(a) || isOperatorBlacklisted(a)`
29//! disjunction (the same one `OriginAssignment._syncAddr` evaluates), which
30//! keeps voted-out OPERATORS (`addOperator` sets only the operator mapping) and
31//! live origins while dropping emergency origins past their auto-expiry. The
32//! survivors seed [`crate::content_deny::ContentDenylist`], which the delivery
33//! path consults to refuse any `StreamRequest` funded by a blacklisted address.
34//!
35//! **No durable projection, no scan cursor.** Both sets are enumerable on chain,
36//! so a restart rebuilds them by reading the chain rather than by reloading a
37//! persisted mirror or replaying from a deploy-block floor. The live tail is
38//! seeded at the enumeration snapshot block (`CursorStart::Seeded`, no persist) —
39//! there is no historical scan on any boot, warm or cold. This is the same
40//! shape the origin directory and slash watchers already use.
41//!
42//! The pinned-block reads are load-bearing, not tidiness: the enumeration views
43//! remove by swap-and-pop, so a page and the count it is checked against MUST be
44//! read at one block height, or a concurrent removal can move an unread element
45//! into an already-read slot and skip it. A `seen != count` mismatch aborts the
46//! snapshot rather than seating a partial set.
47//!
48//! **The live tail** follows both event families on the shared `resumable_watcher`
49//! `eth_getLogs` poller: `HashBlacklisted`/`HashRemoved` for the hash set and
50//! `OriginBlacklistUpdated` + `OperatorBlacklisted`/`OperatorBlacklistCleared`
51//! for the address union (both routed to the one deny-set, mirroring
52//! `OriginAssignment`'s union). It is a low-latency signal to re-read; the
53//! periodic re-enumeration is the backstop that catches anything the tail missed
54//! (a reorg at the tip, or a region/ripening transition into a never-enumerated
55//! region).
56//!
57//! Enforcement is two writes per hash, in this order: the *deny* records that the
58//! refusal is governance-sourced, then the *eviction* reclaims the bytes.
59//!
60//! [`decdn_cache::CacheEngine::evict`] durably records the takedown (survives
61//! restart via `evicted.log`) and cascades to every serving surface through
62//! [`decdn_cache::CacheEngine::refuses`]: the DHT republisher drops the hash on
63//! its next tick, the probe handler stops signing `has_blob: true`, and the
64//! client handler never re-pull-fills it. `evict` is sticky and works on absent
65//! hashes, so a hash blacklisted while the node was offline (and not yet held) is
66//! still pre-blocked.
67//!
68//! The deny is what `evicted.log` cannot express: *why*. Every one of these
69//! refusals signs the one class `Declined`, so no client learns which list a
70//! hash is on (ADR 011 §`StreamRequest` Response); the deny keeps the cause on
71//! the operator's per-reason metric. The governance deny-set the delivery path reads
72//! (`CacheEngine::set_chain_denied_one`) is rebuilt each boot by re-checking every
73//! enumerated hash through the same `isHashBlacklistedForOperator` liveness the
74//! tail uses, so a lapsed entry is not enforced.
75//!
76//! A fail-closed readiness gate (`InitialSyncGate`) keeps every ALPN listener shut
77//! until the first enumeration + enforcement pass completes: a node refuses to
78//! serve until blacklist enforcement is live.
79//!
80//! **Resilience.** The re-scope + re-enumeration pass runs on the operator's
81//! rescan cadence (checked at the end of every poll tick), pulled forward to the
82//! poll cadence whenever a live re-check fails — an enforcement failure must not
83//! wait out the full cadence while the blob stays slashably servable; every RPC
84//! read is bounded by a per-call timeout; the re-scope is interruptible by
85//! shutdown (checked between hashes) so a large backlog cannot overrun the runtime
86//! shutdown deadline. Serving a blacklisted hash is slashable
87//! (`SlashJudge.submitBlacklistChallenge`), so prompt eviction is the node's only
88//! local protection.
89
90use std::collections::{HashMap, HashSet};
91use std::future::Future;
92use std::sync::{Arc, Mutex};
93use std::time::Duration;
94
95use alloy::eips::BlockId;
96use alloy::primitives::{Address, B256, U256};
97use alloy::providers::Provider;
98use alloy::rpc::types::Log;
99use alloy::sol_types::SolEvent;
100use anyhow::{Context as _, Result};
101use decdn_cache::{CacheEngine, Hash};
102use decdn_common::redact::sanitize_err_chain;
103use decdn_incentive::content_blacklist::ContentBlacklist;
104use decdn_incentive::content_blacklist::ContentBlacklist::{
105 HashBlacklisted, HashRemoved, OperatorBlacklistCleared, OperatorBlacklisted,
106 OriginBlacklistUpdated,
107};
108use tokio::sync::oneshot;
109use tokio::time::Instant;
110use tokio_util::sync::CancellationToken;
111use tracing::{debug, info, warn};
112
113use crate::chain_events::boot_retry::{BootFault, BootRetry};
114use crate::chain_events::multiplexed_poller::{Route, SinkSource};
115use crate::chain_events::resumable_watcher::{CursorStart, LogSink, clear_cadence_on_recovery};
116use crate::chain_events::shared_head::{HeadSource, snapshot_block};
117use crate::chain_events::timed;
118use crate::chain_freshness::ChainFreshness;
119use crate::content_deny::ContentDenylist;
120use crate::metrics::{Metrics, metric_hook};
121use crate::warming_allowance::WarmingAllowance;
122
123/// Page size for the swap-and-pop enumeration views. Bounded so one huge deny-set
124/// cannot ask for an unbounded array in a single `eth_call`; the paging loop keeps
125/// reading until it has `count` entries at the pinned block.
126const BLACKLIST_ENUM_PAGE_SIZE: u64 = 100;
127
128/// Result reported exactly once when the first enumeration + enforcement pass
129/// either establishes compliance or proves startup cannot safely continue.
130pub(crate) type InitialSyncResult = std::result::Result<(), String>;
131
132#[derive(Clone)]
133struct InitialSyncGate(Arc<Mutex<Option<oneshot::Sender<InitialSyncResult>>>>);
134
135impl InitialSyncGate {
136 fn new(sender: oneshot::Sender<InitialSyncResult>) -> Self {
137 Self(Arc::new(Mutex::new(Some(sender))))
138 }
139
140 fn signal(&self, result: InitialSyncResult) {
141 let sender = self
142 .0
143 .lock()
144 .unwrap_or_else(std::sync::PoisonError::into_inner)
145 .take();
146 if let Some(sender) = sender {
147 let _ = sender.send(result);
148 }
149 }
150}
151
152/// The chain reads the boot enumeration and the periodic re-enumeration perform,
153/// behind a trait so the pure paging + liveness helpers are unit-testable with a
154/// scripted stub and no live chain.
155///
156/// Spelled RPITIT with an explicit `+ Send` (rather than `async fn`, whose futures
157/// carry no `Send` bound) because the re-enumeration calls these from inside the
158/// spawned watcher's `on_tick_complete`, whose future `tokio::spawn` requires to be
159/// `Send`. Production monomorphizes to the alloy contract impl
160/// ([`ContractReads`]).
161trait BlacklistChainReads: Send + Sync {
162 /// The block every enumeration read is pinned to
163 /// (`shared_head::snapshot_block`): the lag margin below the head.
164 fn snapshot_block(&self) -> impl Future<Output = Result<u64>> + Send;
165 /// The one-to-three region keys in scope for `operator` right now
166 /// (`getScopeRegions`, GLOBAL first).
167 fn scope_regions(
168 &self,
169 operator: Address,
170 at: u64,
171 ) -> impl Future<Output = Result<Vec<B256>>> + Send;
172 /// Number of entries indexed under `region` (`blacklistedHashCount`).
173 fn blacklisted_hash_count(
174 &self,
175 region: B256,
176 at: u64,
177 ) -> impl Future<Output = Result<U256>> + Send;
178 /// A page of `region`'s RAW entry set (`blacklistedHashes`).
179 fn blacklisted_hashes(
180 &self,
181 region: B256,
182 offset: U256,
183 limit: U256,
184 at: u64,
185 ) -> impl Future<Output = Result<Vec<B256>>> + Send;
186 /// Number of addresses in the origin ∪ operator deny set
187 /// (`blacklistedAddressCount`).
188 fn blacklisted_address_count(&self, at: u64) -> impl Future<Output = Result<U256>> + Send;
189 /// A page of the RAW origin ∪ operator address set (`blacklistedAddresses`).
190 fn blacklisted_addresses(
191 &self,
192 offset: U256,
193 limit: U256,
194 at: u64,
195 ) -> impl Future<Output = Result<Vec<Address>>> + Send;
196 /// Origin-level liveness (`isOriginBlacklisted`, honouring emergency
197 /// auto-expiry). `false` for an operator-only entry — must be OR-ed with
198 /// [`Self::is_operator_blacklisted`].
199 fn is_origin_blacklisted(
200 &self,
201 addr: Address,
202 at: u64,
203 ) -> impl Future<Output = Result<bool>> + Send;
204 /// Operator-level liveness (`isOperatorBlacklisted`, the `addOperator` leg).
205 fn is_operator_blacklisted(
206 &self,
207 addr: Address,
208 at: u64,
209 ) -> impl Future<Output = Result<bool>> + Send;
210}
211
212/// Production [`BlacklistChainReads`] over the live `ContentBlacklist` contract.
213/// Every read is bounded by the shared [`timed`] per-call timeout.
214#[derive(Clone)]
215struct ContractReads<P: Provider + Clone> {
216 contract: ContentBlacklist::ContentBlacklistInstance<P>,
217 /// The shared, TTL-cached single-flight head source every watcher reads
218 /// through (`chain_events::shared_head`), so the enumeration snapshot's block
219 /// pin does not cost its own per-boot `eth_blockNumber` call.
220 head: Arc<dyn HeadSource>,
221}
222
223impl<P: Provider + Clone> BlacklistChainReads for ContractReads<P> {
224 async fn snapshot_block(&self) -> Result<u64> {
225 snapshot_block(
226 self.contract.provider(),
227 &*self.head,
228 *self.contract.address(),
229 )
230 .await
231 }
232
233 async fn scope_regions(&self, operator: Address, at: u64) -> Result<Vec<B256>> {
234 timed(
235 None,
236 "getScopeRegions",
237 self.contract
238 .getScopeRegions(operator)
239 .block(BlockId::Number(at.into()))
240 .call(),
241 )
242 .await
243 }
244
245 async fn blacklisted_hash_count(&self, region: B256, at: u64) -> Result<U256> {
246 timed(
247 None,
248 "blacklistedHashCount",
249 self.contract
250 .blacklistedHashCount(region)
251 .block(BlockId::Number(at.into()))
252 .call(),
253 )
254 .await
255 }
256
257 async fn blacklisted_hashes(
258 &self,
259 region: B256,
260 offset: U256,
261 limit: U256,
262 at: u64,
263 ) -> Result<Vec<B256>> {
264 timed(
265 None,
266 "blacklistedHashes",
267 self.contract
268 .blacklistedHashes(region, offset, limit)
269 .block(BlockId::Number(at.into()))
270 .call(),
271 )
272 .await
273 }
274
275 async fn blacklisted_address_count(&self, at: u64) -> Result<U256> {
276 timed(
277 None,
278 "blacklistedAddressCount",
279 self.contract
280 .blacklistedAddressCount()
281 .block(BlockId::Number(at.into()))
282 .call(),
283 )
284 .await
285 }
286
287 async fn blacklisted_addresses(
288 &self,
289 offset: U256,
290 limit: U256,
291 at: u64,
292 ) -> Result<Vec<Address>> {
293 timed(
294 None,
295 "blacklistedAddresses",
296 self.contract
297 .blacklistedAddresses(offset, limit)
298 .block(BlockId::Number(at.into()))
299 .call(),
300 )
301 .await
302 }
303
304 async fn is_origin_blacklisted(&self, addr: Address, at: u64) -> Result<bool> {
305 timed(
306 None,
307 "isOriginBlacklisted",
308 self.contract
309 .isOriginBlacklisted(addr)
310 .block(BlockId::Number(at.into()))
311 .call(),
312 )
313 .await
314 }
315
316 async fn is_operator_blacklisted(&self, addr: Address, at: u64) -> Result<bool> {
317 timed(
318 None,
319 "isOperatorBlacklisted",
320 self.contract
321 .isOperatorBlacklisted(addr)
322 .block(BlockId::Number(at.into()))
323 .call(),
324 )
325 .await
326 }
327}
328
329/// The current on-chain deny-set as enumerated at one pinned block.
330struct BootstrapSnapshot {
331 /// The block every read below was pinned to; the tail is seeded here.
332 block: u64,
333 /// The origin ∪ operator address deny set, liveness-filtered by the union
334 /// predicate `isOriginBlacklisted || isOperatorBlacklisted`.
335 origins: HashSet<Address>,
336 /// Every `(region, hash)` entry across the in-scope regions.
337 known: HashSet<(B256, Hash)>,
338}
339
340/// Enumerate the on-chain deny-set at one pinned block, the lag margin below
341/// head: page the address union and the per-region hash sets against that
342/// height.
343async fn bootstrap_snapshot<R: BlacklistChainReads>(
344 reads: &R,
345 operator: Address,
346) -> Result<BootstrapSnapshot> {
347 let block = reads
348 .snapshot_block()
349 .await
350 .context("read the ContentBlacklist enumeration snapshot block")?;
351 let origins = enumerate_address_union(reads, operator, block).await?;
352 let known = enumerate_known(reads, operator, block).await?;
353 Ok(BootstrapSnapshot {
354 block,
355 origins,
356 known,
357 })
358}
359
360/// Page the origin ∪ operator address set at `at`, count-checked, then keep only
361/// the addresses that are LIVE by the UNION predicate.
362///
363/// The liveness filter is `isOriginBlacklisted(a) || isOperatorBlacklisted(a)` —
364/// the same disjunction `ContentBlacklist._syncAddr` / `OriginAssignment` use.
365/// Filtering by `isOriginBlacklisted` ALONE would drop every voted-out operator
366/// (`addOperator` sets only the operator mapping and never touches the origin
367/// mapping) = the #1499 hole, a restarted node serving a governance-ejected
368/// operator. The RAW enumeration also lists emergency origins past their
369/// auto-expiry, which the union drops (fail-open) — operators never auto-expire.
370async fn enumerate_address_union<R: BlacklistChainReads>(
371 reads: &R,
372 _operator: Address,
373 at: u64,
374) -> Result<HashSet<Address>> {
375 let count = reads
376 .blacklisted_address_count(at)
377 .await
378 .context("blacklistedAddressCount")?;
379 let mut raw: HashSet<Address> = HashSet::new();
380 let mut offset = U256::ZERO;
381 let page = U256::from(BLACKLIST_ENUM_PAGE_SIZE);
382 while offset < count {
383 let batch = reads
384 .blacklisted_addresses(offset, page, at)
385 .await
386 .with_context(|| format!("blacklistedAddresses(offset={offset}, limit={page})"))?;
387 if batch.is_empty() {
388 break;
389 }
390 offset = offset.saturating_add(U256::from(batch.len()));
391 raw.extend(batch);
392 }
393 let seen = U256::from(raw.len());
394 anyhow::ensure!(
395 seen == count,
396 "address deny-set enumeration read {seen} of {count} entries at block {at}; the pinned \
397 set was read inconsistently (a mid-page swap-and-pop removal, or an inconsistent/reorged \
398 RPC view of this block), so the snapshot would be missing an address — aborting rather \
399 than seating a partial set"
400 );
401
402 let mut live: HashSet<Address> = HashSet::with_capacity(raw.len());
403 for addr in raw {
404 let denied = reads
405 .is_origin_blacklisted(addr, at)
406 .await
407 .with_context(|| format!("isOriginBlacklisted({addr})"))?
408 || reads
409 .is_operator_blacklisted(addr, at)
410 .await
411 .with_context(|| format!("isOperatorBlacklisted({addr})"))?;
412 if denied {
413 live.insert(addr);
414 }
415 }
416 Ok(live)
417}
418
419/// Page `region`'s RAW hash set at `at`, count-checked. Liveness (emergency
420/// auto-expiry, region scope) is applied later by `isHashBlacklistedForOperator`
421/// during enforcement, matching the tail's `scope_check`.
422async fn enumerate_region_hashes<R: BlacklistChainReads>(
423 reads: &R,
424 region: B256,
425 at: u64,
426) -> Result<HashSet<Hash>> {
427 let count = reads
428 .blacklisted_hash_count(region, at)
429 .await
430 .with_context(|| format!("blacklistedHashCount({region})"))?;
431 let mut hashes: HashSet<Hash> = HashSet::new();
432 let mut offset = U256::ZERO;
433 let page = U256::from(BLACKLIST_ENUM_PAGE_SIZE);
434 while offset < count {
435 let batch = reads
436 .blacklisted_hashes(region, offset, page, at)
437 .await
438 .with_context(|| {
439 format!("blacklistedHashes(region={region}, offset={offset}, limit={page})")
440 })?;
441 if batch.is_empty() {
442 break;
443 }
444 offset = offset.saturating_add(U256::from(batch.len()));
445 hashes.extend(batch.into_iter().map(|h| Hash::from_bytes(h.0)));
446 }
447 let seen = U256::from(hashes.len());
448 anyhow::ensure!(
449 seen == count,
450 "region {region} hash enumeration read {seen} of {count} entries at block {at}; the \
451 pinned set was read inconsistently (a mid-page swap-and-pop removal, or an \
452 inconsistent/reorged RPC view of this block), so the snapshot would be missing a hash — \
453 aborting rather than seating a partial set"
454 );
455 Ok(hashes)
456}
457
458/// Build the `(region, hash)` deny-set across the operator's in-scope regions.
459async fn enumerate_known<R: BlacklistChainReads>(
460 reads: &R,
461 operator: Address,
462 at: u64,
463) -> Result<HashSet<(B256, Hash)>> {
464 let regions = reads
465 .scope_regions(operator, at)
466 .await
467 .with_context(|| format!("getScopeRegions({operator})"))?;
468 let mut known = HashSet::new();
469 for region in regions {
470 for hash in enumerate_region_hashes(reads, region, at).await? {
471 known.insert((region, hash));
472 }
473 }
474 Ok(known)
475}
476
477/// Mutable deny-set carried across poll ticks. The scan cursor lives on the
478/// resumable watcher (seeded at the boot snapshot); this holds only the
479/// re-scopable entry set.
480struct WatcherState {
481 /// Every blacklisted `(region, hash)` entry seen and not yet locally evicted —
482 /// including out-of-scope entries — re-scoped on each `on_tick_complete` pass
483 /// so a later region/ripening transition (which emits no `HashBlacklisted`)
484 /// still leads to eviction. Keyed like the contract's `_hashEntries`.
485 known: HashSet<(B256, Hash)>,
486 /// The live origin deny-set the delivery path reads (ADR 011 §On Blacklist
487 /// Event). Seeded from the enumerated address union on boot and fed by
488 /// `OriginBlacklistUpdated` / `OperatorBlacklisted` on the tail.
489 denylist: Arc<ContentDenylist>,
490 /// The latest tail change to each origin: its block and the blacklisted
491 /// state it set. A re-enumeration applies every change above its snapshot
492 /// block on top of the snapshot's origins, then forgets the rest.
493 origin_changes: HashMap<Address, (u64, bool)>,
494 /// The block of each `(region, hash)` entry the tail removed. A
495 /// re-enumeration skips a snapshot entry the tail removed above its snapshot
496 /// block, then forgets the rest.
497 removed_entries: HashMap<(B256, Hash), u64>,
498}
499
500impl WatcherState {
501 /// Record a `(region, hash)` entry in the re-scoping worklist.
502 fn add_entry(&mut self, region: B256, hash: Hash) {
503 self.known.insert((region, hash));
504 }
505
506 /// Fold a fresh re-enumeration snapshot into the projections. The asymmetry is
507 /// deliberate and correctness-critical: the `known` worklist is UNIONED so an
508 /// out-of-scope entry the tail learned (and retained for a future ripening
509 /// transition, which emits no event) is not dropped by an in-scope-only
510 /// enumeration, while the chain origins are authoritative and REPLACED
511 /// wholesale.
512 ///
513 /// The snapshot reads the lagged block, so it misses any change the tail
514 /// already applied above that block. Both halves keep those changes: every
515 /// origin change above the block is applied on top of the snapshot's origins,
516 /// and a snapshot entry the tail removed above the block is not re-added.
517 /// Without that, the fold would drop an origin the tail blacklisted inside the
518 /// margin.
519 fn fold_reenumeration(&mut self, snapshot: BootstrapSnapshot) {
520 let at = snapshot.block;
521 self.origin_changes.retain(|_, (block, _)| *block > at);
522 self.removed_entries.retain(|_, block| *block > at);
523 let mut origins = snapshot.origins;
524 for (origin, (_, blacklisted)) in &self.origin_changes {
525 if *blacklisted {
526 origins.insert(*origin);
527 } else {
528 origins.remove(origin);
529 }
530 }
531 let removed = &self.removed_entries;
532 self.known.extend(
533 snapshot
534 .known
535 .into_iter()
536 .filter(|entry| !removed.contains_key(entry)),
537 );
538 self.denylist.set_chain_origins(origins);
539 }
540
541 /// Drop exactly the `(region, hash)` entry — same-hash entries under other
542 /// regions stay retained for re-scoping — and note the block of the removal
543 /// for [`Self::fold_reenumeration`].
544 fn remove_entry(&mut self, region: B256, hash: Hash, block: u64) {
545 self.known.remove(&(region, hash));
546 self.removed_entries.insert((region, hash), block);
547 }
548
549 /// Drop every entry for `hash` (once locally evicted, the sticky eviction
550 /// covers all regions).
551 fn drop_hash(&mut self, hash: Hash) {
552 self.known.retain(|(_, known_hash)| *known_hash != hash);
553 }
554
555 /// Distinct hashes across all regions — the scope view
556 /// (`isHashBlacklistedForOperator`) is per `(operator, hash)`, so each hash
557 /// needs exactly one `eth_call` per pass regardless of how many regional
558 /// entries reference it.
559 fn distinct_hashes(&self) -> Vec<Hash> {
560 let unique: HashSet<Hash> = self.known.iter().map(|(_, hash)| *hash).collect();
561 unique.into_iter().collect()
562 }
563
564 /// Apply an origin/operator blacklist state change to the live deny-set the
565 /// delivery path and the origin-routing filter both read. Both on-chain lists
566 /// (`OriginBlacklistUpdated` and `OperatorBlacklisted`) feed this one set as a
567 /// union `isOriginBlacklisted(op) || isOperatorBlacklisted(op)` — the node does
568 /// not need to know which list an address came from, only that governance put
569 /// it on one. Notes the change and its block for
570 /// [`Self::fold_reenumeration`].
571 fn set_origin(&mut self, origin: Address, blacklisted: bool, block: u64) {
572 self.denylist.apply_chain_origin(origin, blacklisted);
573 self.origin_changes.insert(origin, (block, blacklisted));
574 }
575}
576
577/// Applies the five live event families to the deny-set, enforces compliance, and
578/// re-enumerates + re-scopes on the operator's cadence. `apply` records each entry
579/// and, for a `HashBlacklisted`, immediately re-checks scope + evicts (prompt live
580/// enforcement); an undecodable ORIGIN log aborts the tick (there is no per-tick
581/// backstop to reveal a skipped one until the next re-enumeration), while an
582/// undecodable HASH log is logged and skipped (re-scoped
583/// every pass, sticky eviction). [`Self::on_tick_complete`] runs the batched
584/// re-enumeration + re-scope.
585struct BlacklistSink<P: Provider + Clone> {
586 contract: ContentBlacklist::ContentBlacklistInstance<P>,
587 reads: ContractReads<P>,
588 operator: Address,
589 cache: CacheEngine,
590 warming: Arc<WarmingAllowance>,
591 state: WatcherState,
592 shutdown: CancellationToken,
593 /// How often the batched re-enumeration + re-scope runs (the operator's
594 /// `content_blacklist_poll_interval_sec`); the getLogs poll cadence itself is
595 /// faster so live entries enforce promptly.
596 rescan_interval: Duration,
597 /// When the last batched pass ran. Seeded to `Some(now)` at spawn (boot just
598 /// enumerated + enforced), so the first backstop is one interval out; cleared
599 /// by a failed live re-check to pull the next pass forward.
600 last_rescan: Option<Instant>,
601 /// Node metrics — feeds `blacklist_enforcement_failures` from a re-scope that
602 /// could not enforce every entry (#1319).
603 metrics: Arc<Metrics>,
604}
605
606impl<P: Provider + Clone> LogSink for BlacklistSink<P> {
607 async fn apply(&mut self, log: Log) -> Result<()> {
608 let failed = handle_log(
609 &self.contract,
610 self.operator,
611 &self.cache,
612 &self.warming,
613 &mut self.state,
614 log,
615 )
616 .await?;
617 // A live `HashBlacklisted` whose scope-check or eviction failed must retry
618 // at the poll cadence (seconds), not the operator's re-scope cadence
619 // (default 10 min) — serving the blob meanwhile is slashable. Clearing
620 // `last_rescan` forces the batched pass on this tick's `on_tick_complete`.
621 if failed {
622 self.last_rescan = None;
623 }
624 Ok(())
625 }
626
627 async fn on_tick_complete(&mut self) -> Result<()> {
628 // Re-enumerate + re-scope the whole deny-set on the operator's cadence (not
629 // every poll tick): catches a region/ripening transition that emits no
630 // event, and a tail event lost to a reorg or RPC backoff.
631 let due = self
632 .last_rescan
633 .is_none_or(|at| at.elapsed() >= self.rescan_interval);
634
635 // The re-enumeration backstop. Rebuild the address union + in-scope hash set
636 // at the lagged snapshot block and fold it into the projections: origins are
637 // authoritative (replace wholesale, with the tail's newer changes on top),
638 // the `known` worklist is UNIONed so an out-of-scope entry the tail learned
639 // (retained for future ripening) is not dropped by an in-scope-only
640 // enumeration. Build-then-fold; Ok-on-failure keeps the current set and
641 // counts `decdn_blacklist_reenumeration_failures_total`. The `rescan` below
642 // then re-scopes + evicts.
643 if due {
644 match bootstrap_snapshot(&self.reads, self.operator).await {
645 Ok(snapshot) => self.state.fold_reenumeration(snapshot),
646 Err(err) => {
647 self.metrics.blacklist_reenumeration_failure();
648 warn!(
649 error = %sanitize_err_chain(&err),
650 "blacklist watcher: periodic re-enumeration failed; keeping the current \
651 deny-set (the live tail is still the primary path)"
652 );
653 }
654 }
655 }
656
657 if due {
658 let RescanOutcome { clean, failed, .. } = rescan(
659 &self.contract,
660 self.operator,
661 &self.cache,
662 &self.warming,
663 &mut self.state,
664 &self.shutdown,
665 OnScopeFailure::Continue,
666 )
667 .await;
668 // Surface the slashable "deny-set not fully enforced" condition (#1319):
669 // a failed re-scope otherwise returns `Ok(())` while every downtime
670 // metric reads healthy. Separate from the down-family (chain-read
671 // outages only).
672 if failed > 0 {
673 self.metrics.blacklist_enforcement_failure(failed);
674 }
675 // A pass with any failed re-check retries at the poll cadence until it
676 // comes back clean; only a clean pass waits out the full cadence again.
677 self.last_rescan = clean.then(Instant::now);
678 }
679 Ok(())
680 }
681
682 /// Force the batched re-scope on the tick the watcher recovers.
683 ///
684 /// The same clock `apply` clears after a failed per-event re-scope, cleared
685 /// for the same reason: a stale deny set means serving content a takedown
686 /// covers, which is slashable. An outage is when the set is most likely to
687 /// have drifted and when the cadence helps least, since the reconcile does
688 /// not run at all while the route is errored.
689 fn on_recovered(&mut self) {
690 clear_cadence_on_recovery(&mut self.last_rescan, self.rescan_interval);
691 }
692}
693
694/// Outcome of one batched re-scope pass.
695struct RescanOutcome {
696 /// `true` iff every entry was re-verified with no failed re-check. A
697 /// shutdown-cancelled pass is unclean (`false`); it is decidedly **not** a
698 /// drift window, but that is enforced in `multiplexed_poller::run` (which
699 /// suppresses the backoff edge under a cancelled token), not here (#1321).
700 clean: bool,
701 /// Distinct hashes this pass could not enforce (`Recheck::Failed` — a cache
702 /// error or a scope `eth_call` failure). Feeds `blacklist_enforcement_failures`
703 /// so the slashable "deny-set not fully enforced" condition has a metric of its
704 /// own. A shutdown-cancelled pass reports the failures observed **before** the
705 /// cancel (not `0`).
706 failed: u64,
707 /// The failure that decides a boot attempt's fate: an eviction failure if
708 /// any (a local disk fault a retry does not repair), else the first scope
709 /// read failure. A boot attempt reports it as the cause, so it can tell a
710 /// deterministic fault from a transient one.
711 decisive_failure: Option<RecheckFailure>,
712}
713
714/// What a re-scope pass does after a scope read fails.
715#[derive(Clone, Copy, Debug, PartialEq, Eq)]
716enum OnScopeFailure {
717 /// Re-check every remaining hash: the periodic re-scope enforces every hash
718 /// it can, and retries the rest on the next tick.
719 Continue,
720 /// Stop the pass: a boot attempt with a failed scope read is lost and
721 /// retries whole, so its remaining reads only add load to a failing
722 /// provider.
723 Stop,
724}
725
726/// Re-scope every distinct hash in `known` (one scope `eth_call` per hash, not per
727/// regional entry) and evict those now in scope. Interruptible by shutdown between
728/// hashes; `on_scope_failure` decides whether a failed scope read ends the pass.
729async fn rescan<P>(
730 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
731 operator: Address,
732 cache: &CacheEngine,
733 warming: &Arc<WarmingAllowance>,
734 state: &mut WatcherState,
735 shutdown: &CancellationToken,
736 on_scope_failure: OnScopeFailure,
737) -> RescanOutcome
738where
739 P: Provider + Clone,
740{
741 let snapshot = state.distinct_hashes();
742 let mut evicted = 0usize;
743 let mut failed = 0u64;
744 let mut decisive_failure: Option<RecheckFailure> = None;
745 let mut clean = true;
746 for hash in snapshot {
747 if shutdown.is_cancelled() {
748 return RescanOutcome {
749 clean: false,
750 failed,
751 decisive_failure,
752 };
753 }
754 match recheck(contract, operator, cache, warming, state, hash).await {
755 Recheck::Evicted => evicted = evicted.saturating_add(1),
756 Recheck::NoAction => {}
757 Recheck::Failed(cause) => {
758 clean = false;
759 failed = failed.saturating_add(1);
760 let is_scope = matches!(cause, RecheckFailure::Scope(_));
761 let replaces = match &decisive_failure {
762 None => true,
763 Some(RecheckFailure::Scope(_)) => !is_scope,
764 Some(RecheckFailure::Evict(_)) => false,
765 };
766 if replaces {
767 decisive_failure = Some(cause);
768 }
769 if is_scope && on_scope_failure == OnScopeFailure::Stop {
770 break;
771 }
772 }
773 }
774 }
775 if evicted > 0 {
776 info!(evicted, "blacklist watcher evicted blacklisted blobs");
777 }
778 if failed > 0 {
779 // The negative counterpart to the `evicted` line: the aggregate count of
780 // entries left unenforced this pass (#1319).
781 warn!(
782 unenforced = failed,
783 "blacklist re-scope could not enforce every entry"
784 );
785 }
786 RescanOutcome {
787 clean,
788 failed,
789 decisive_failure,
790 }
791}
792
793/// Handle one live log: `HashBlacklisted` records the `(region, hash)` entry and
794/// re-checks the hash; `HashRemoved` drops exactly that entry (hygiene — eviction
795/// stays sticky, and same-hash entries in other regions survive). `Ok(true)` iff a
796/// re-check failed and needs a prompt retry.
797///
798/// `Err` is reserved for an undecodable ORIGIN-class log: until the next
799/// re-enumeration there is nothing to sweep against, so
800/// skipping one is a silent deny-set gap — aborting the tick holds the scan cursor
801/// so the readiness gate keeps the router closed rather than opening on a set we
802/// know is incomplete. An undecodable HASH log is skipped (re-scoped every pass,
803/// sticky eviction).
804async fn handle_log<P>(
805 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
806 operator: Address,
807 cache: &CacheEngine,
808 warming: &Arc<WarmingAllowance>,
809 state: &mut WatcherState,
810 log: Log,
811) -> Result<bool>
812where
813 P: Provider + Clone,
814{
815 match log.topic0() {
816 Some(topic) if *topic == HashBlacklisted::SIGNATURE_HASH => Ok(on_blacklisted_log(
817 contract, operator, cache, warming, state, &log,
818 )
819 .await
820 .is_failed()),
821 Some(topic) if *topic == HashRemoved::SIGNATURE_HASH => {
822 on_removed_log(contract, operator, cache, state, &log).await;
823 Ok(false)
824 }
825 Some(topic) if *topic == OriginBlacklistUpdated::SIGNATURE_HASH => {
826 on_origin_log(state, &log)?;
827 Ok(false)
828 }
829 Some(topic) if *topic == OperatorBlacklisted::SIGNATURE_HASH => {
830 on_operator_log(state, &log, true)?;
831 Ok(false)
832 }
833 Some(topic) if *topic == OperatorBlacklistCleared::SIGNATURE_HASH => {
834 on_operator_log(state, &log, false)?;
835 Ok(false)
836 }
837 _ => Ok(false),
838 }
839}
840
841/// Decode an `OperatorBlacklisted` / `OperatorBlacklistCleared` log and apply it to
842/// the same origin deny-set `OriginBlacklistUpdated` feeds.
843fn on_operator_log(state: &mut WatcherState, log: &Log, blacklisted: bool) -> Result<()> {
844 let operator = if blacklisted {
845 match OperatorBlacklisted::decode_log_data(&log.inner.data) {
846 Ok(event) => event.operator,
847 Err(err) => return Err(undecodable_origin_log(&err, log, "OperatorBlacklisted")),
848 }
849 } else {
850 match OperatorBlacklistCleared::decode_log_data(&log.inner.data) {
851 Ok(event) => event.operator,
852 Err(err) => {
853 return Err(undecodable_origin_log(
854 &err,
855 log,
856 "OperatorBlacklistCleared",
857 ));
858 }
859 }
860 };
861 state.set_origin(operator, blacklisted, log_block(log));
862 debug!(%operator, blacklisted, "blacklist watcher: operator blacklist updated");
863 Ok(())
864}
865
866/// The block a tail log was mined in. A log without one never comes back from
867/// `eth_getLogs`; one that did would count as the newest change, which a
868/// re-enumeration keeps.
869fn log_block(log: &Log) -> u64 {
870 log.block_number.unwrap_or(u64::MAX)
871}
872
873/// An origin-class log we cannot decode is an ENFORCEMENT failure, not a parse
874/// curiosity, so it aborts the tick and holds the scan cursor.
875///
876/// The hash events can afford to skip-and-continue: they are re-scoped every pass
877/// and eviction is sticky. The origin events have no such per-tick backstop, so a
878/// skipped log is gone until the next
879/// re-enumeration and the deny-set is silently short an entry meanwhile. Holding
880/// the cursor lets the readiness gate keep the router closed rather than opening on
881/// a set we know is incomplete.
882fn undecodable_origin_log(err: &alloy::sol_types::Error, log: &Log, event: &str) -> anyhow::Error {
883 anyhow::anyhow!("{err}").context(format!(
884 "undecodable {event} log at block {:?} tx {:?}; refusing to advance the scan cursor \
885 past an unreadable takedown event",
886 log.block_number, log.transaction_hash
887 ))
888}
889
890fn on_origin_log(state: &mut WatcherState, log: &Log) -> Result<()> {
891 let event = match OriginBlacklistUpdated::decode_log_data(&log.inner.data) {
892 Ok(event) => event,
893 Err(err) => return Err(undecodable_origin_log(&err, log, "OriginBlacklistUpdated")),
894 };
895 state.set_origin(event.origin, event.blacklisted, log_block(log));
896 debug!(
897 origin = %event.origin,
898 blacklisted = event.blacklisted,
899 "blacklist watcher: origin blacklist updated"
900 );
901 Ok(())
902}
903
904/// Decode a `HashBlacklisted` log, record its `(region, hash)` entry, and re-check
905/// the hash. An undecodable log is [`Recheck::NoAction`] (skipped, per the
906/// `LogSink` contract).
907async fn on_blacklisted_log<P>(
908 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
909 operator: Address,
910 cache: &CacheEngine,
911 warming: &Arc<WarmingAllowance>,
912 state: &mut WatcherState,
913 log: &Log,
914) -> Recheck
915where
916 P: Provider + Clone,
917{
918 match HashBlacklisted::decode_log_data(&log.inner.data) {
919 Ok(event) => {
920 let hash = Hash::from_bytes(event.hash.0);
921 state.add_entry(event.region, hash);
922 recheck(contract, operator, cache, warming, state, hash).await
923 }
924 Err(err) => {
925 warn!(error = %err, "blacklist watcher: undecodable HashBlacklisted log");
926 Recheck::NoAction
927 }
928 }
929}
930
931/// Decode a `HashRemoved` log and drop exactly that `(region, hash)` entry.
932///
933/// Also retires the hash from the governance deny-set, but only on a definitive
934/// out-of-scope read: `HashRemoved` is per-region, and a same-hash entry under
935/// another region can still cover this operator. `isHashBlacklistedForOperator` is
936/// the authoritative union, so it — not the event — decides. An RPC failure keeps
937/// the hash denied, the conservative direction (the hash stays evicted regardless,
938/// so the cost is a stale refusal *code*, not a stale refusal).
939async fn on_removed_log<P: Provider + Clone>(
940 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
941 operator: Address,
942 cache: &CacheEngine,
943 state: &mut WatcherState,
944 log: &Log,
945) {
946 match HashRemoved::decode_log_data(&log.inner.data) {
947 Ok(event) => {
948 let hash = Hash::from_bytes(event.hash.0);
949 state.remove_entry(event.region, hash, log_block(log));
950 if cache.is_chain_denied(hash) {
951 lift_deny_if_out_of_scope(contract, operator, cache, hash).await;
952 }
953 debug!(
954 region = %event.region,
955 %hash,
956 "blacklist entry removed on-chain (local eviction stays sticky)"
957 );
958 }
959 Err(err) => warn!(error = %err, "blacklist watcher: undecodable HashRemoved log"),
960 }
961}
962
963/// Lift the governance deny on `hash` only on a definitive out-of-scope read. A
964/// failed read keeps the deny: over-denying is the safe direction.
965async fn lift_deny_if_out_of_scope<P: Provider + Clone>(
966 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
967 operator: Address,
968 cache: &CacheEngine,
969 hash: Hash,
970) {
971 match scope_check(contract, operator, hash).await {
972 Ok(false) => undeny_hash(cache, hash),
973 Ok(true) => {}
974 Err(err) => warn!(
975 %hash,
976 error = %sanitize_err_chain(&err),
977 "blacklist watcher: isHashBlacklistedForOperator failed after HashRemoved; \
978 keeping the hash governance-denied"
979 ),
980 }
981}
982
983/// Outcome of one scope re-check, so callers can distinguish an enforcement
984/// *failure* (retry promptly — the entry may be live and slashable) from a
985/// legitimately out-of-scope entry (the periodic re-scope keeps watching it).
986#[derive(Debug)]
987enum Recheck {
988 /// The hash was in scope and its eviction succeeded.
989 Evicted,
990 /// Nothing to do: already evicted, or currently out of scope.
991 NoAction,
992 /// The scope read or the eviction failed — the entry is retained and must be
993 /// re-checked promptly.
994 Failed(RecheckFailure),
995}
996
997impl Recheck {
998 const fn is_failed(&self) -> bool {
999 matches!(self, Self::Failed(_))
1000 }
1001}
1002
1003/// Why a re-check failed.
1004#[derive(Debug)]
1005enum RecheckFailure {
1006 /// The `isHashBlacklistedForOperator` read failed (RPC error or timeout).
1007 Scope(anyhow::Error),
1008 /// The local cache could not evict the hash (a disk error).
1009 Evict(anyhow::Error),
1010}
1011
1012/// Record that `hash` is refused because *governance* blacklisted it, and publish
1013/// that to the live deny-set the delivery path reads. This is what selects the
1014/// `chain_hash_denied` refusal reason over the local-eviction one; both sign
1015/// `Declined`.
1016fn deny_hash(cache: &CacheEngine, hash: Hash) {
1017 cache.set_chain_denied_one(hash, true);
1018}
1019
1020/// Stop treating `hash` as governance-denied. The hash stays *refused* — eviction
1021/// is sticky and one-way — so this only moves its refusal reason from
1022/// `chain_hash_denied` back to the eviction, which is what any other evicted hash
1023/// answers. The wire class stays `Declined`.
1024fn undeny_hash(cache: &CacheEngine, hash: Hash) {
1025 cache.set_chain_denied_one(hash, false);
1026}
1027
1028/// Evict `hash` if in scope. Out-of-scope (`Ok(false)`) and RPC-error (`Err`)
1029/// hashes keep their `known` entries for the next re-scope (callers insert before
1030/// calling); evicted hashes drop *all* their regional entries — eviction is sticky
1031/// and region-independent.
1032async fn recheck<P>(
1033 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1034 operator: Address,
1035 cache: &CacheEngine,
1036 warming: &Arc<WarmingAllowance>,
1037 state: &mut WatcherState,
1038 hash: Hash,
1039) -> Recheck
1040where
1041 P: Provider + Clone,
1042{
1043 if cache.is_evicted(hash) {
1044 // Back-fill the governance deny-set for a hash that is already evicted but
1045 // not yet recorded as governance-denied. `evicted.log` persists the eviction
1046 // but records no cause, and the deny-set lives only in memory, so after a
1047 // restart every takedown evicted on a prior boot would otherwise keep
1048 // answering `EvictedSinceProbe` forever.
1049 if !cache.is_chain_denied(hash) {
1050 match scope_check(contract, operator, hash).await {
1051 Ok(true) => deny_hash(cache, hash),
1052 Ok(false) => {}
1053 Err(err) => warn!(
1054 %hash,
1055 error = %sanitize_err_chain(&err),
1056 "blacklist watcher: isHashBlacklistedForOperator failed for an evicted \
1057 hash; its governance deny waits for the next re-enumeration"
1058 ),
1059 }
1060 }
1061 state.drop_hash(hash);
1062 return Recheck::NoAction;
1063 }
1064 match scope_check(contract, operator, hash).await {
1065 Ok(true) => {
1066 // Deny before evicting: eviction is what retires the hash from `known`,
1067 // and `known` is the retry backstop. Denying first also means an
1068 // eviction that fails on a disk error still stops the serving, since
1069 // `CacheEngine::refuses` honors this set too.
1070 deny_hash(cache, hash);
1071 match evict(cache, warming, hash).await {
1072 Ok(()) => {
1073 state.drop_hash(hash);
1074 Recheck::Evicted
1075 }
1076 Err(err) => Recheck::Failed(RecheckFailure::Evict(err)),
1077 }
1078 }
1079 Ok(false) => Recheck::NoAction,
1080 Err(err) => {
1081 warn!(
1082 %hash,
1083 error = %sanitize_err_chain(&err),
1084 "blacklist watcher: isHashBlacklistedForOperator failed; keeping for re-scope"
1085 );
1086 Recheck::Failed(RecheckFailure::Scope(err))
1087 }
1088 }
1089}
1090
1091/// `isHashBlacklistedForOperator` bounded by the shared
1092/// [`chain_events::DEFAULT_RPC_CALL_TIMEOUT`]. `Err` on timeout or RPC error;
1093/// each caller logs it with what the failure means on its path.
1094///
1095/// [`chain_events::DEFAULT_RPC_CALL_TIMEOUT`]: crate::chain_events::DEFAULT_RPC_CALL_TIMEOUT
1096async fn scope_check<P>(
1097 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1098 operator: Address,
1099 hash: Hash,
1100) -> Result<bool>
1101where
1102 P: Provider + Clone,
1103{
1104 let hash_key = B256::from(*hash.as_bytes());
1105 timed(
1106 None,
1107 "isHashBlacklistedForOperator",
1108 contract
1109 .isHashBlacklistedForOperator(hash_key, operator)
1110 .call(),
1111 )
1112 .await
1113 .with_context(|| format!("isHashBlacklistedForOperator({hash})"))
1114}
1115
1116/// Evict `hash` from the cache (durable + sticky). A failure is logged here and
1117/// returned.
1118async fn evict(cache: &CacheEngine, warming: &Arc<WarmingAllowance>, hash: Hash) -> Result<()> {
1119 match cache.evict(hash).await {
1120 Ok(()) => {
1121 info!(%hash, "evicted blacklisted blob (ADR 011 compliance)");
1122 // ADR 041: drop the warming tag for the evicted hash, so a later
1123 // reuse of this slot can never credit a stale source's allowance.
1124 warming.forget(hash);
1125 Ok(())
1126 }
1127 Err(err) => {
1128 // Through `anyhow` so the log carries the whole chain down to the I/O
1129 // cause, not only the cache error's top line.
1130 let err = anyhow::Error::new(err).context(format!("evict {hash}"));
1131 warn!(%hash, error = %sanitize_err_chain(&err), "blacklist watcher: evict failed; will retry");
1132 Err(err)
1133 }
1134 }
1135}
1136
1137/// One boot attempt: enumerate the current on-chain deny-set at one pinned
1138/// block, seed the live origin deny-set the delivery path reads, then enforce —
1139/// deny + evict every enumerated hash that is in scope right now, decided by the
1140/// same `isHashBlacklistedForOperator` liveness the tail uses (so a lapsed
1141/// emergency entry is not enforced).
1142///
1143/// The pass stops at the first failed scope read: the attempt is lost and
1144/// retries whole, so the remaining reads only add load to a failing provider.
1145/// An unclean pass fails the attempt with its decisive failure as the cause: a
1146/// local eviction error is a [`BootFault`], since a retry does not repair a
1147/// disk, and otherwise the scope read keeps its typed RPC error for the retry
1148/// classifier. The inline pass is not shutdown-interruptible; the periodic
1149/// re-scope on the sink is.
1150#[allow(clippy::too_many_arguments)]
1151async fn enumerate_and_enforce<P>(
1152 reads: &ContractReads<P>,
1153 contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1154 operator: Address,
1155 cache: &CacheEngine,
1156 warming: &Arc<WarmingAllowance>,
1157 denylist: &Arc<ContentDenylist>,
1158 metrics: &Arc<Metrics>,
1159 shutdown: &CancellationToken,
1160) -> Result<BootEnforced>
1161where
1162 P: Provider + Clone,
1163{
1164 let snapshot = bootstrap_snapshot(reads, operator).await?;
1165 denylist.set_chain_origins(snapshot.origins.clone());
1166 let mut state = WatcherState {
1167 known: snapshot.known,
1168 denylist: Arc::clone(denylist),
1169 origin_changes: HashMap::new(),
1170 removed_entries: HashMap::new(),
1171 };
1172 let RescanOutcome {
1173 clean,
1174 failed,
1175 decisive_failure,
1176 } = rescan(
1177 contract,
1178 operator,
1179 cache,
1180 warming,
1181 &mut state,
1182 shutdown,
1183 OnScopeFailure::Stop,
1184 )
1185 .await;
1186 if failed > 0 {
1187 metrics.blacklist_enforcement_failure(failed);
1188 }
1189 if !clean {
1190 let cause = match decisive_failure {
1191 Some(RecheckFailure::Scope(err)) => err,
1192 Some(RecheckFailure::Evict(err)) => BootFault(format!("{err:#}")).into(),
1193 // Only a cancelled pass is unclean with no failure, and nothing
1194 // cancels the boot pass's token.
1195 None => anyhow::anyhow!("enforcement pass was cancelled"),
1196 };
1197 return Err(cause.context(format!(
1198 "initial ContentBlacklist enforcement could not enforce every entry ({failed} failed)"
1199 )));
1200 }
1201 Ok(BootEnforced {
1202 block: snapshot.block,
1203 origin_count: snapshot.origins.len(),
1204 state,
1205 })
1206}
1207
1208/// A clean boot attempt: the deny-set enumerated and enforced.
1209struct BootEnforced {
1210 /// The pinned block the enumeration read, where the live tail starts.
1211 block: u64,
1212 /// Live origin and operator addresses in the seeded origin deny-set.
1213 origin_count: usize,
1214 /// The enforced deny-set the sink carries forward.
1215 state: WatcherState,
1216}
1217
1218/// Enumerate the on-chain deny-set at one pinned block, enforce it, then return
1219/// the [`Route`] that follows the live tail seeded at that block on the shared
1220/// multiplexed poller.
1221///
1222/// The pinned block sits `SNAPSHOT_LAG_MARGIN_BLOCKS` below head (or at head for
1223/// a contract deployed inside the margin), so the readiness gate opens on the
1224/// deny-set as of that block. A takedown inside the margin is enforced by the
1225/// first tail tick, which replays the margin.
1226///
1227/// One boot attempt is the enumeration plus the enforcement pass. An attempt that
1228/// cannot read the chain (block or enumeration RPC error) or cannot enforce
1229/// every entry retries on `boot`'s budget. `initial_sync_tx` fires once, when
1230/// the attempts resolve: `Ok` after a clean pass, so the runtime can gate the
1231/// ALPN router on blacklist enforcement being live, or `Err` on a deterministic
1232/// fault or an exhausted budget. That failure also returns `Err`, so the router
1233/// never opens on an un-vetted deny-set. `rescan_interval` is the batched
1234/// re-enumeration + re-scope cadence.
1235///
1236/// The returned route carries a [`SinkSource::Factory`]: the blacklist sink must
1237/// observe the poller's own shutdown token (its re-scope polls it between per-hash
1238/// `eth_call`s), which the poller mints only at spawn — so the sink is built
1239/// inside that spawn from the freshly-minted token.
1240#[allow(clippy::too_many_arguments)]
1241pub(crate) async fn bootstrap<P>(
1242 provider: P,
1243 contract_addr: Address,
1244 operator: Address,
1245 cache: CacheEngine,
1246 warming: Arc<WarmingAllowance>,
1247 head: Arc<dyn HeadSource>,
1248 rescan_interval: Duration,
1249 initial_sync_tx: oneshot::Sender<InitialSyncResult>,
1250 metrics: &Arc<Metrics>,
1251 denylist: Arc<ContentDenylist>,
1252 chain_freshness: ChainFreshness,
1253 boot: &BootRetry,
1254) -> Result<Route>
1255where
1256 P: Provider + Clone + 'static,
1257{
1258 let contract = ContentBlacklist::new(contract_addr, provider.clone());
1259 let reads = ContractReads {
1260 contract: contract.clone(),
1261 head: Arc::clone(&head),
1262 };
1263 let initial_sync = InitialSyncGate::new(initial_sync_tx);
1264
1265 // An attempt that cannot enforce every entry is retried whole, unless its
1266 // first failure is deterministic; re-seeding and re-enforcing are
1267 // idempotent.
1268 let boot_shutdown = CancellationToken::new();
1269 let attempt = boot
1270 .run("ContentBlacklist boot snapshot", || {
1271 enumerate_and_enforce(
1272 &reads,
1273 &contract,
1274 operator,
1275 &cache,
1276 &warming,
1277 &denylist,
1278 metrics,
1279 &boot_shutdown,
1280 )
1281 })
1282 .await;
1283 let BootEnforced {
1284 block: snapshot_block,
1285 origin_count,
1286 state,
1287 } = match attempt {
1288 Ok(done) => done,
1289 Err(err) => {
1290 // Fail CLOSED: signal the readiness gate so the runtime keeps every ALPN
1291 // listener shut, then abort startup — a node that cannot vet the
1292 // takedown set must not serve.
1293 initial_sync.signal(Err(format!(
1294 "initial ContentBlacklist sync failed: {}",
1295 sanitize_err_chain(&err)
1296 )));
1297 return Err(err).context("enumerate and enforce the ContentBlacklist boot snapshot");
1298 }
1299 };
1300 // The boot enumeration + enforcement read the chain successfully, so seed the
1301 // freshness clock: a node that has just vetted its deny-set must not read as
1302 // stale in the window before the first poll tick stamps it.
1303 chain_freshness.stamp();
1304 initial_sync.signal(Ok(()));
1305
1306 info!(
1307 %contract_addr,
1308 %operator,
1309 snapshot_block,
1310 known_hashes = state.known.len(),
1311 origins = origin_count,
1312 "blacklist compliance watcher enumerated its boot snapshot"
1313 );
1314
1315 let sink_metrics = Arc::clone(metrics);
1316 // Unlike the flush-only sinks, `BlacklistSink` must observe the *same* token
1317 // the poller cancels: `rescan` polls it between per-hash `eth_call`s so a
1318 // large deny-set re-scope yields promptly to shutdown. The poller mints one
1319 // token at spawn and hands it to this factory, so sink and loop share it
1320 // (#1236).
1321 let sink_factory: SinkSource = SinkSource::Factory(Box::new(move |shutdown| {
1322 Box::new(BlacklistSink {
1323 contract,
1324 reads,
1325 operator,
1326 cache,
1327 warming,
1328 state,
1329 shutdown: shutdown.clone(),
1330 rescan_interval: rescan_interval.max(Duration::from_secs(1)),
1331 // Boot enumeration + enforcement just ran; first backstop is one
1332 // interval out.
1333 last_rescan: Some(Instant::now()),
1334 metrics: sink_metrics,
1335 }) as Box<dyn crate::chain_events::multiplexed_poller::ErasedSink>
1336 }));
1337
1338 Ok(Route {
1339 addresses: vec![contract_addr],
1340 topic0s: blacklist_route_topic0s(),
1341 start: blacklist_cursor_start(snapshot_block),
1342 sink: sink_factory,
1343 label: "blacklist",
1344 on_established: Some(metric_hook(
1345 metrics,
1346 Metrics::blacklist_watcher_cycle_established,
1347 )),
1348 on_backoff: Some(metric_hook(
1349 metrics,
1350 Metrics::blacklist_watcher_backoff_started,
1351 )),
1352 // Every successful poll tick (including idle ones) both stamps the
1353 // liveness gauge and refreshes the serve-path staleness clock — the two
1354 // are the same signal, so they fire from one hook.
1355 on_tick_success: Some({
1356 let metrics = Arc::clone(metrics);
1357 let freshness = chain_freshness.clone();
1358 Box::new(move || {
1359 metrics.blacklist_watcher_tick();
1360 freshness.stamp();
1361 })
1362 }),
1363 on_task_panic: Some(metric_hook(
1364 metrics,
1365 Metrics::blacklist_watcher_task_panicked,
1366 )),
1367 })
1368}
1369
1370/// The blacklist route's demux key: hash takedowns plus both origin-blacklist
1371/// paths. Split out from [`bootstrap`] so the exact topic0 set is
1372/// unit-testable without a provider.
1373///
1374/// Origin blacklisting rides the same scan (ADR 011 § Hash Evasion). Like the
1375/// hash events, it has no version counter to detect a missed one — the
1376/// re-enumeration is the backstop. `addOperator` is the PRIMARY governance
1377/// origin-blacklist path — it writes a SEPARATE mapping and emits
1378/// `OperatorBlacklisted`/`OperatorBlacklistCleared`, never
1379/// `OriginBlacklistUpdated`. `OriginAssignment` unions the two mappings
1380/// on-chain; watching only the first would leave the delivery gate enforcing
1381/// the softer list and missing the voted one.
1382fn blacklist_route_topic0s() -> Vec<B256> {
1383 vec![
1384 HashBlacklisted::SIGNATURE_HASH,
1385 HashRemoved::SIGNATURE_HASH,
1386 OriginBlacklistUpdated::SIGNATURE_HASH,
1387 OperatorBlacklisted::SIGNATURE_HASH,
1388 OperatorBlacklistCleared::SIGNATURE_HASH,
1389 ]
1390}
1391
1392/// The blacklist route's cursor start: seed the tail at the enumeration
1393/// snapshot block. No durable cursor and no historical replay — the boot
1394/// enumeration rebuilt the whole deny-set, so the tail only follows forward
1395/// from the snapshot. Split out from [`bootstrap`] so the cursor shape is
1396/// unit-testable without a provider.
1397const fn blacklist_cursor_start(snapshot_block: u64) -> CursorStart {
1398 CursorStart::Seeded { at: snapshot_block }
1399}
1400
1401#[cfg(test)]
1402// Test-only: the assertion style below intentionally panics on the negative
1403// branch, and unwraps/expects on results the fixtures guarantee. Matches the
1404// convention in `content_deny.rs` / `config/mod.rs`.
1405#[allow(
1406 clippy::panic,
1407 clippy::unwrap_used,
1408 clippy::expect_used,
1409 clippy::indexing_slicing
1410)]
1411mod tests;