Skip to main content

decdn_node/
blacklist_watcher.rs

1//! Blacklist compliance watcher (ADR 011 § Content Takedown).
2//!
3//! Every paid-delivery node runs this task after resolving the mandatory
4//! `blockchain.content_blacklist_address`. It keeps the local blob store
5//! compliant with `ContentBlacklist`: it evicts any blob whose hash is
6//! blacklisted *in scope* for this operator (global ∪ current-region ∪
7//! ripening-prev-region). The scope decision is the contract's
8//! `isHashBlacklistedForOperator` view, so region packing and the ADR 030
9//! ripening math never leave the chain.
10//!
11//! **Enumerated deny-set, re-scoped.** The set of blacklisted entries is read
12//! straight from `ContentBlacklist`'s enumeration views at one pinned block on
13//! boot, rather than replayed from `HashBlacklisted` logs (#1497). Two views feed
14//! two sets:
15//!
16//! - **Hash entries.** `getScopeRegions(operator)` gives the one-to-three region
17//!   keys in scope right now; each region's set is paged from
18//!   `blacklistedHashes` / `blacklistedHashCount`. Entries are keyed by
19//!   `(region, hash)` — the contract's own key (`_hashEntries[region][hash]`) —
20//!   so a same-hash entry under another region survives a `HashRemoved` for the
21//!   first. Every seen-but-not-yet-evicted entry is retained in `known` and
22//!   re-scoped on a periodic (`on_tick_complete`) pass, because a hash can become
23//!   live + in scope with **no** `HashBlacklisted` event, via an operator
24//!   region/ripening change (`CapacityBond.updateRegion`), which emits nothing on
25//!   this contract at all.
26//! - **Address union.** `blacklistedAddresses` / `blacklistedAddressCount`
27//!   enumerate the origin ∪ operator deny set. Each address is liveness-filtered
28//!   by the contract's own `isOriginBlacklisted(a) || isOperatorBlacklisted(a)`
29//!   disjunction (the same one `OriginAssignment._syncAddr` evaluates), which
30//!   keeps voted-out OPERATORS (`addOperator` sets only the operator mapping) and
31//!   live origins while dropping emergency origins past their auto-expiry. The
32//!   survivors seed [`crate::content_deny::ContentDenylist`], which the delivery
33//!   path consults to refuse any `StreamRequest` funded by a blacklisted address.
34//!
35//! **No durable projection, no scan cursor.** Both sets are enumerable on chain,
36//! so a restart rebuilds them by reading the chain rather than by reloading a
37//! persisted mirror or replaying from a deploy-block floor. The live tail is
38//! seeded at the enumeration snapshot block (`CursorStart::Seeded`, no persist) —
39//! there is no historical scan on any boot, warm or cold. This is the same
40//! shape the origin directory and slash watchers already use.
41//!
42//! The pinned-block reads are load-bearing, not tidiness: the enumeration views
43//! remove by swap-and-pop, so a page and the count it is checked against MUST be
44//! read at one block height, or a concurrent removal can move an unread element
45//! into an already-read slot and skip it. A `seen != count` mismatch aborts the
46//! snapshot rather than seating a partial set.
47//!
48//! **The live tail** follows both event families on the shared `resumable_watcher`
49//! `eth_getLogs` poller: `HashBlacklisted`/`HashRemoved` for the hash set and
50//! `OriginBlacklistUpdated` + `OperatorBlacklisted`/`OperatorBlacklistCleared`
51//! for the address union (both routed to the one deny-set, mirroring
52//! `OriginAssignment`'s union). It is a low-latency signal to re-read; the
53//! periodic re-enumeration is the backstop that catches anything the tail missed
54//! (a reorg at the tip, or a region/ripening transition into a never-enumerated
55//! region).
56//!
57//! Enforcement is two writes per hash, in this order: the *deny* records that the
58//! refusal is governance-sourced, then the *eviction* reclaims the bytes.
59//!
60//! [`decdn_cache::CacheEngine::evict`] durably records the takedown (survives
61//! restart via `evicted.log`) and cascades to every serving surface through
62//! [`decdn_cache::CacheEngine::refuses`]: the DHT republisher drops the hash on
63//! its next tick, the probe handler stops signing `has_blob: true`, and the
64//! client handler never re-pull-fills it. `evict` is sticky and works on absent
65//! hashes, so a hash blacklisted while the node was offline (and not yet held) is
66//! still pre-blocked.
67//!
68//! The deny is what `evicted.log` cannot express: *why*. Every one of these
69//! refusals signs the one class `Declined`, so no client learns which list a
70//! hash is on (ADR 011 §`StreamRequest` Response); the deny keeps the cause on
71//! the operator's per-reason metric. The governance deny-set the delivery path reads
72//! (`CacheEngine::set_chain_denied_one`) is rebuilt each boot by re-checking every
73//! enumerated hash through the same `isHashBlacklistedForOperator` liveness the
74//! tail uses, so a lapsed entry is not enforced.
75//!
76//! A fail-closed readiness gate (`InitialSyncGate`) keeps every ALPN listener shut
77//! until the first enumeration + enforcement pass completes: a node refuses to
78//! serve until blacklist enforcement is live.
79//!
80//! **Resilience.** The re-scope + re-enumeration pass runs on the operator's
81//! rescan cadence (checked at the end of every poll tick), pulled forward to the
82//! poll cadence whenever a live re-check fails — an enforcement failure must not
83//! wait out the full cadence while the blob stays slashably servable; every RPC
84//! read is bounded by a per-call timeout; the re-scope is interruptible by
85//! shutdown (checked between hashes) so a large backlog cannot overrun the runtime
86//! shutdown deadline. Serving a blacklisted hash is slashable
87//! (`SlashJudge.submitBlacklistChallenge`), so prompt eviction is the node's only
88//! local protection.
89
90use std::collections::{HashMap, HashSet};
91use std::future::Future;
92use std::sync::{Arc, Mutex};
93use std::time::Duration;
94
95use alloy::eips::BlockId;
96use alloy::primitives::{Address, B256, U256};
97use alloy::providers::Provider;
98use alloy::rpc::types::Log;
99use alloy::sol_types::SolEvent;
100use anyhow::{Context as _, Result};
101use decdn_cache::{CacheEngine, Hash};
102use decdn_common::redact::sanitize_err_chain;
103use decdn_incentive::content_blacklist::ContentBlacklist;
104use decdn_incentive::content_blacklist::ContentBlacklist::{
105    HashBlacklisted, HashRemoved, OperatorBlacklistCleared, OperatorBlacklisted,
106    OriginBlacklistUpdated,
107};
108use tokio::sync::oneshot;
109use tokio::time::Instant;
110use tokio_util::sync::CancellationToken;
111use tracing::{debug, info, warn};
112
113use crate::chain_events::boot_retry::{BootFault, BootRetry};
114use crate::chain_events::multiplexed_poller::{Route, SinkSource};
115use crate::chain_events::resumable_watcher::{CursorStart, LogSink, clear_cadence_on_recovery};
116use crate::chain_events::shared_head::{HeadSource, snapshot_block};
117use crate::chain_events::timed;
118use crate::chain_freshness::ChainFreshness;
119use crate::content_deny::ContentDenylist;
120use crate::metrics::{Metrics, metric_hook};
121use crate::warming_allowance::WarmingAllowance;
122
123/// Page size for the swap-and-pop enumeration views. Bounded so one huge deny-set
124/// cannot ask for an unbounded array in a single `eth_call`; the paging loop keeps
125/// reading until it has `count` entries at the pinned block.
126const BLACKLIST_ENUM_PAGE_SIZE: u64 = 100;
127
128/// Result reported exactly once when the first enumeration + enforcement pass
129/// either establishes compliance or proves startup cannot safely continue.
130pub(crate) type InitialSyncResult = std::result::Result<(), String>;
131
132#[derive(Clone)]
133struct InitialSyncGate(Arc<Mutex<Option<oneshot::Sender<InitialSyncResult>>>>);
134
135impl InitialSyncGate {
136    fn new(sender: oneshot::Sender<InitialSyncResult>) -> Self {
137        Self(Arc::new(Mutex::new(Some(sender))))
138    }
139
140    fn signal(&self, result: InitialSyncResult) {
141        let sender = self
142            .0
143            .lock()
144            .unwrap_or_else(std::sync::PoisonError::into_inner)
145            .take();
146        if let Some(sender) = sender {
147            let _ = sender.send(result);
148        }
149    }
150}
151
152/// The chain reads the boot enumeration and the periodic re-enumeration perform,
153/// behind a trait so the pure paging + liveness helpers are unit-testable with a
154/// scripted stub and no live chain.
155///
156/// Spelled RPITIT with an explicit `+ Send` (rather than `async fn`, whose futures
157/// carry no `Send` bound) because the re-enumeration calls these from inside the
158/// spawned watcher's `on_tick_complete`, whose future `tokio::spawn` requires to be
159/// `Send`. Production monomorphizes to the alloy contract impl
160/// ([`ContractReads`]).
161trait BlacklistChainReads: Send + Sync {
162    /// The block every enumeration read is pinned to
163    /// (`shared_head::snapshot_block`): the lag margin below the head.
164    fn snapshot_block(&self) -> impl Future<Output = Result<u64>> + Send;
165    /// The one-to-three region keys in scope for `operator` right now
166    /// (`getScopeRegions`, GLOBAL first).
167    fn scope_regions(
168        &self,
169        operator: Address,
170        at: u64,
171    ) -> impl Future<Output = Result<Vec<B256>>> + Send;
172    /// Number of entries indexed under `region` (`blacklistedHashCount`).
173    fn blacklisted_hash_count(
174        &self,
175        region: B256,
176        at: u64,
177    ) -> impl Future<Output = Result<U256>> + Send;
178    /// A page of `region`'s RAW entry set (`blacklistedHashes`).
179    fn blacklisted_hashes(
180        &self,
181        region: B256,
182        offset: U256,
183        limit: U256,
184        at: u64,
185    ) -> impl Future<Output = Result<Vec<B256>>> + Send;
186    /// Number of addresses in the origin ∪ operator deny set
187    /// (`blacklistedAddressCount`).
188    fn blacklisted_address_count(&self, at: u64) -> impl Future<Output = Result<U256>> + Send;
189    /// A page of the RAW origin ∪ operator address set (`blacklistedAddresses`).
190    fn blacklisted_addresses(
191        &self,
192        offset: U256,
193        limit: U256,
194        at: u64,
195    ) -> impl Future<Output = Result<Vec<Address>>> + Send;
196    /// Origin-level liveness (`isOriginBlacklisted`, honouring emergency
197    /// auto-expiry). `false` for an operator-only entry — must be OR-ed with
198    /// [`Self::is_operator_blacklisted`].
199    fn is_origin_blacklisted(
200        &self,
201        addr: Address,
202        at: u64,
203    ) -> impl Future<Output = Result<bool>> + Send;
204    /// Operator-level liveness (`isOperatorBlacklisted`, the `addOperator` leg).
205    fn is_operator_blacklisted(
206        &self,
207        addr: Address,
208        at: u64,
209    ) -> impl Future<Output = Result<bool>> + Send;
210}
211
212/// Production [`BlacklistChainReads`] over the live `ContentBlacklist` contract.
213/// Every read is bounded by the shared [`timed`] per-call timeout.
214#[derive(Clone)]
215struct ContractReads<P: Provider + Clone> {
216    contract: ContentBlacklist::ContentBlacklistInstance<P>,
217    /// The shared, TTL-cached single-flight head source every watcher reads
218    /// through (`chain_events::shared_head`), so the enumeration snapshot's block
219    /// pin does not cost its own per-boot `eth_blockNumber` call.
220    head: Arc<dyn HeadSource>,
221}
222
223impl<P: Provider + Clone> BlacklistChainReads for ContractReads<P> {
224    async fn snapshot_block(&self) -> Result<u64> {
225        snapshot_block(
226            self.contract.provider(),
227            &*self.head,
228            *self.contract.address(),
229        )
230        .await
231    }
232
233    async fn scope_regions(&self, operator: Address, at: u64) -> Result<Vec<B256>> {
234        timed(
235            None,
236            "getScopeRegions",
237            self.contract
238                .getScopeRegions(operator)
239                .block(BlockId::Number(at.into()))
240                .call(),
241        )
242        .await
243    }
244
245    async fn blacklisted_hash_count(&self, region: B256, at: u64) -> Result<U256> {
246        timed(
247            None,
248            "blacklistedHashCount",
249            self.contract
250                .blacklistedHashCount(region)
251                .block(BlockId::Number(at.into()))
252                .call(),
253        )
254        .await
255    }
256
257    async fn blacklisted_hashes(
258        &self,
259        region: B256,
260        offset: U256,
261        limit: U256,
262        at: u64,
263    ) -> Result<Vec<B256>> {
264        timed(
265            None,
266            "blacklistedHashes",
267            self.contract
268                .blacklistedHashes(region, offset, limit)
269                .block(BlockId::Number(at.into()))
270                .call(),
271        )
272        .await
273    }
274
275    async fn blacklisted_address_count(&self, at: u64) -> Result<U256> {
276        timed(
277            None,
278            "blacklistedAddressCount",
279            self.contract
280                .blacklistedAddressCount()
281                .block(BlockId::Number(at.into()))
282                .call(),
283        )
284        .await
285    }
286
287    async fn blacklisted_addresses(
288        &self,
289        offset: U256,
290        limit: U256,
291        at: u64,
292    ) -> Result<Vec<Address>> {
293        timed(
294            None,
295            "blacklistedAddresses",
296            self.contract
297                .blacklistedAddresses(offset, limit)
298                .block(BlockId::Number(at.into()))
299                .call(),
300        )
301        .await
302    }
303
304    async fn is_origin_blacklisted(&self, addr: Address, at: u64) -> Result<bool> {
305        timed(
306            None,
307            "isOriginBlacklisted",
308            self.contract
309                .isOriginBlacklisted(addr)
310                .block(BlockId::Number(at.into()))
311                .call(),
312        )
313        .await
314    }
315
316    async fn is_operator_blacklisted(&self, addr: Address, at: u64) -> Result<bool> {
317        timed(
318            None,
319            "isOperatorBlacklisted",
320            self.contract
321                .isOperatorBlacklisted(addr)
322                .block(BlockId::Number(at.into()))
323                .call(),
324        )
325        .await
326    }
327}
328
329/// The current on-chain deny-set as enumerated at one pinned block.
330struct BootstrapSnapshot {
331    /// The block every read below was pinned to; the tail is seeded here.
332    block: u64,
333    /// The origin ∪ operator address deny set, liveness-filtered by the union
334    /// predicate `isOriginBlacklisted || isOperatorBlacklisted`.
335    origins: HashSet<Address>,
336    /// Every `(region, hash)` entry across the in-scope regions.
337    known: HashSet<(B256, Hash)>,
338}
339
340/// Enumerate the on-chain deny-set at one pinned block, the lag margin below
341/// head: page the address union and the per-region hash sets against that
342/// height.
343async fn bootstrap_snapshot<R: BlacklistChainReads>(
344    reads: &R,
345    operator: Address,
346) -> Result<BootstrapSnapshot> {
347    let block = reads
348        .snapshot_block()
349        .await
350        .context("read the ContentBlacklist enumeration snapshot block")?;
351    let origins = enumerate_address_union(reads, operator, block).await?;
352    let known = enumerate_known(reads, operator, block).await?;
353    Ok(BootstrapSnapshot {
354        block,
355        origins,
356        known,
357    })
358}
359
360/// Page the origin ∪ operator address set at `at`, count-checked, then keep only
361/// the addresses that are LIVE by the UNION predicate.
362///
363/// The liveness filter is `isOriginBlacklisted(a) || isOperatorBlacklisted(a)` —
364/// the same disjunction `ContentBlacklist._syncAddr` / `OriginAssignment` use.
365/// Filtering by `isOriginBlacklisted` ALONE would drop every voted-out operator
366/// (`addOperator` sets only the operator mapping and never touches the origin
367/// mapping) = the #1499 hole, a restarted node serving a governance-ejected
368/// operator. The RAW enumeration also lists emergency origins past their
369/// auto-expiry, which the union drops (fail-open) — operators never auto-expire.
370async fn enumerate_address_union<R: BlacklistChainReads>(
371    reads: &R,
372    _operator: Address,
373    at: u64,
374) -> Result<HashSet<Address>> {
375    let count = reads
376        .blacklisted_address_count(at)
377        .await
378        .context("blacklistedAddressCount")?;
379    let mut raw: HashSet<Address> = HashSet::new();
380    let mut offset = U256::ZERO;
381    let page = U256::from(BLACKLIST_ENUM_PAGE_SIZE);
382    while offset < count {
383        let batch = reads
384            .blacklisted_addresses(offset, page, at)
385            .await
386            .with_context(|| format!("blacklistedAddresses(offset={offset}, limit={page})"))?;
387        if batch.is_empty() {
388            break;
389        }
390        offset = offset.saturating_add(U256::from(batch.len()));
391        raw.extend(batch);
392    }
393    let seen = U256::from(raw.len());
394    anyhow::ensure!(
395        seen == count,
396        "address deny-set enumeration read {seen} of {count} entries at block {at}; the pinned \
397         set was read inconsistently (a mid-page swap-and-pop removal, or an inconsistent/reorged \
398         RPC view of this block), so the snapshot would be missing an address — aborting rather \
399         than seating a partial set"
400    );
401
402    let mut live: HashSet<Address> = HashSet::with_capacity(raw.len());
403    for addr in raw {
404        let denied = reads
405            .is_origin_blacklisted(addr, at)
406            .await
407            .with_context(|| format!("isOriginBlacklisted({addr})"))?
408            || reads
409                .is_operator_blacklisted(addr, at)
410                .await
411                .with_context(|| format!("isOperatorBlacklisted({addr})"))?;
412        if denied {
413            live.insert(addr);
414        }
415    }
416    Ok(live)
417}
418
419/// Page `region`'s RAW hash set at `at`, count-checked. Liveness (emergency
420/// auto-expiry, region scope) is applied later by `isHashBlacklistedForOperator`
421/// during enforcement, matching the tail's `scope_check`.
422async fn enumerate_region_hashes<R: BlacklistChainReads>(
423    reads: &R,
424    region: B256,
425    at: u64,
426) -> Result<HashSet<Hash>> {
427    let count = reads
428        .blacklisted_hash_count(region, at)
429        .await
430        .with_context(|| format!("blacklistedHashCount({region})"))?;
431    let mut hashes: HashSet<Hash> = HashSet::new();
432    let mut offset = U256::ZERO;
433    let page = U256::from(BLACKLIST_ENUM_PAGE_SIZE);
434    while offset < count {
435        let batch = reads
436            .blacklisted_hashes(region, offset, page, at)
437            .await
438            .with_context(|| {
439                format!("blacklistedHashes(region={region}, offset={offset}, limit={page})")
440            })?;
441        if batch.is_empty() {
442            break;
443        }
444        offset = offset.saturating_add(U256::from(batch.len()));
445        hashes.extend(batch.into_iter().map(|h| Hash::from_bytes(h.0)));
446    }
447    let seen = U256::from(hashes.len());
448    anyhow::ensure!(
449        seen == count,
450        "region {region} hash enumeration read {seen} of {count} entries at block {at}; the \
451         pinned set was read inconsistently (a mid-page swap-and-pop removal, or an \
452         inconsistent/reorged RPC view of this block), so the snapshot would be missing a hash — \
453         aborting rather than seating a partial set"
454    );
455    Ok(hashes)
456}
457
458/// Build the `(region, hash)` deny-set across the operator's in-scope regions.
459async fn enumerate_known<R: BlacklistChainReads>(
460    reads: &R,
461    operator: Address,
462    at: u64,
463) -> Result<HashSet<(B256, Hash)>> {
464    let regions = reads
465        .scope_regions(operator, at)
466        .await
467        .with_context(|| format!("getScopeRegions({operator})"))?;
468    let mut known = HashSet::new();
469    for region in regions {
470        for hash in enumerate_region_hashes(reads, region, at).await? {
471            known.insert((region, hash));
472        }
473    }
474    Ok(known)
475}
476
477/// Mutable deny-set carried across poll ticks. The scan cursor lives on the
478/// resumable watcher (seeded at the boot snapshot); this holds only the
479/// re-scopable entry set.
480struct WatcherState {
481    /// Every blacklisted `(region, hash)` entry seen and not yet locally evicted —
482    /// including out-of-scope entries — re-scoped on each `on_tick_complete` pass
483    /// so a later region/ripening transition (which emits no `HashBlacklisted`)
484    /// still leads to eviction. Keyed like the contract's `_hashEntries`.
485    known: HashSet<(B256, Hash)>,
486    /// The live origin deny-set the delivery path reads (ADR 011 §On Blacklist
487    /// Event). Seeded from the enumerated address union on boot and fed by
488    /// `OriginBlacklistUpdated` / `OperatorBlacklisted` on the tail.
489    denylist: Arc<ContentDenylist>,
490    /// The latest tail change to each origin: its block and the blacklisted
491    /// state it set. A re-enumeration applies every change above its snapshot
492    /// block on top of the snapshot's origins, then forgets the rest.
493    origin_changes: HashMap<Address, (u64, bool)>,
494    /// The block of each `(region, hash)` entry the tail removed. A
495    /// re-enumeration skips a snapshot entry the tail removed above its snapshot
496    /// block, then forgets the rest.
497    removed_entries: HashMap<(B256, Hash), u64>,
498}
499
500impl WatcherState {
501    /// Record a `(region, hash)` entry in the re-scoping worklist.
502    fn add_entry(&mut self, region: B256, hash: Hash) {
503        self.known.insert((region, hash));
504    }
505
506    /// Fold a fresh re-enumeration snapshot into the projections. The asymmetry is
507    /// deliberate and correctness-critical: the `known` worklist is UNIONED so an
508    /// out-of-scope entry the tail learned (and retained for a future ripening
509    /// transition, which emits no event) is not dropped by an in-scope-only
510    /// enumeration, while the chain origins are authoritative and REPLACED
511    /// wholesale.
512    ///
513    /// The snapshot reads the lagged block, so it misses any change the tail
514    /// already applied above that block. Both halves keep those changes: every
515    /// origin change above the block is applied on top of the snapshot's origins,
516    /// and a snapshot entry the tail removed above the block is not re-added.
517    /// Without that, the fold would drop an origin the tail blacklisted inside the
518    /// margin.
519    fn fold_reenumeration(&mut self, snapshot: BootstrapSnapshot) {
520        let at = snapshot.block;
521        self.origin_changes.retain(|_, (block, _)| *block > at);
522        self.removed_entries.retain(|_, block| *block > at);
523        let mut origins = snapshot.origins;
524        for (origin, (_, blacklisted)) in &self.origin_changes {
525            if *blacklisted {
526                origins.insert(*origin);
527            } else {
528                origins.remove(origin);
529            }
530        }
531        let removed = &self.removed_entries;
532        self.known.extend(
533            snapshot
534                .known
535                .into_iter()
536                .filter(|entry| !removed.contains_key(entry)),
537        );
538        self.denylist.set_chain_origins(origins);
539    }
540
541    /// Drop exactly the `(region, hash)` entry — same-hash entries under other
542    /// regions stay retained for re-scoping — and note the block of the removal
543    /// for [`Self::fold_reenumeration`].
544    fn remove_entry(&mut self, region: B256, hash: Hash, block: u64) {
545        self.known.remove(&(region, hash));
546        self.removed_entries.insert((region, hash), block);
547    }
548
549    /// Drop every entry for `hash` (once locally evicted, the sticky eviction
550    /// covers all regions).
551    fn drop_hash(&mut self, hash: Hash) {
552        self.known.retain(|(_, known_hash)| *known_hash != hash);
553    }
554
555    /// Distinct hashes across all regions — the scope view
556    /// (`isHashBlacklistedForOperator`) is per `(operator, hash)`, so each hash
557    /// needs exactly one `eth_call` per pass regardless of how many regional
558    /// entries reference it.
559    fn distinct_hashes(&self) -> Vec<Hash> {
560        let unique: HashSet<Hash> = self.known.iter().map(|(_, hash)| *hash).collect();
561        unique.into_iter().collect()
562    }
563
564    /// Apply an origin/operator blacklist state change to the live deny-set the
565    /// delivery path and the origin-routing filter both read. Both on-chain lists
566    /// (`OriginBlacklistUpdated` and `OperatorBlacklisted`) feed this one set as a
567    /// union `isOriginBlacklisted(op) || isOperatorBlacklisted(op)` — the node does
568    /// not need to know which list an address came from, only that governance put
569    /// it on one. Notes the change and its block for
570    /// [`Self::fold_reenumeration`].
571    fn set_origin(&mut self, origin: Address, blacklisted: bool, block: u64) {
572        self.denylist.apply_chain_origin(origin, blacklisted);
573        self.origin_changes.insert(origin, (block, blacklisted));
574    }
575}
576
577/// Applies the five live event families to the deny-set, enforces compliance, and
578/// re-enumerates + re-scopes on the operator's cadence. `apply` records each entry
579/// and, for a `HashBlacklisted`, immediately re-checks scope + evicts (prompt live
580/// enforcement); an undecodable ORIGIN log aborts the tick (there is no per-tick
581/// backstop to reveal a skipped one until the next re-enumeration), while an
582/// undecodable HASH log is logged and skipped (re-scoped
583/// every pass, sticky eviction). [`Self::on_tick_complete`] runs the batched
584/// re-enumeration + re-scope.
585struct BlacklistSink<P: Provider + Clone> {
586    contract: ContentBlacklist::ContentBlacklistInstance<P>,
587    reads: ContractReads<P>,
588    operator: Address,
589    cache: CacheEngine,
590    warming: Arc<WarmingAllowance>,
591    state: WatcherState,
592    shutdown: CancellationToken,
593    /// How often the batched re-enumeration + re-scope runs (the operator's
594    /// `content_blacklist_poll_interval_sec`); the getLogs poll cadence itself is
595    /// faster so live entries enforce promptly.
596    rescan_interval: Duration,
597    /// When the last batched pass ran. Seeded to `Some(now)` at spawn (boot just
598    /// enumerated + enforced), so the first backstop is one interval out; cleared
599    /// by a failed live re-check to pull the next pass forward.
600    last_rescan: Option<Instant>,
601    /// Node metrics — feeds `blacklist_enforcement_failures` from a re-scope that
602    /// could not enforce every entry (#1319).
603    metrics: Arc<Metrics>,
604}
605
606impl<P: Provider + Clone> LogSink for BlacklistSink<P> {
607    async fn apply(&mut self, log: Log) -> Result<()> {
608        let failed = handle_log(
609            &self.contract,
610            self.operator,
611            &self.cache,
612            &self.warming,
613            &mut self.state,
614            log,
615        )
616        .await?;
617        // A live `HashBlacklisted` whose scope-check or eviction failed must retry
618        // at the poll cadence (seconds), not the operator's re-scope cadence
619        // (default 10 min) — serving the blob meanwhile is slashable. Clearing
620        // `last_rescan` forces the batched pass on this tick's `on_tick_complete`.
621        if failed {
622            self.last_rescan = None;
623        }
624        Ok(())
625    }
626
627    async fn on_tick_complete(&mut self) -> Result<()> {
628        // Re-enumerate + re-scope the whole deny-set on the operator's cadence (not
629        // every poll tick): catches a region/ripening transition that emits no
630        // event, and a tail event lost to a reorg or RPC backoff.
631        let due = self
632            .last_rescan
633            .is_none_or(|at| at.elapsed() >= self.rescan_interval);
634
635        // The re-enumeration backstop. Rebuild the address union + in-scope hash set
636        // at the lagged snapshot block and fold it into the projections: origins are
637        // authoritative (replace wholesale, with the tail's newer changes on top),
638        // the `known` worklist is UNIONed so an out-of-scope entry the tail learned
639        // (retained for future ripening) is not dropped by an in-scope-only
640        // enumeration. Build-then-fold; Ok-on-failure keeps the current set and
641        // counts `decdn_blacklist_reenumeration_failures_total`. The `rescan` below
642        // then re-scopes + evicts.
643        if due {
644            match bootstrap_snapshot(&self.reads, self.operator).await {
645                Ok(snapshot) => self.state.fold_reenumeration(snapshot),
646                Err(err) => {
647                    self.metrics.blacklist_reenumeration_failure();
648                    warn!(
649                        error = %sanitize_err_chain(&err),
650                        "blacklist watcher: periodic re-enumeration failed; keeping the current \
651                         deny-set (the live tail is still the primary path)"
652                    );
653                }
654            }
655        }
656
657        if due {
658            let RescanOutcome { clean, failed, .. } = rescan(
659                &self.contract,
660                self.operator,
661                &self.cache,
662                &self.warming,
663                &mut self.state,
664                &self.shutdown,
665                OnScopeFailure::Continue,
666            )
667            .await;
668            // Surface the slashable "deny-set not fully enforced" condition (#1319):
669            // a failed re-scope otherwise returns `Ok(())` while every downtime
670            // metric reads healthy. Separate from the down-family (chain-read
671            // outages only).
672            if failed > 0 {
673                self.metrics.blacklist_enforcement_failure(failed);
674            }
675            // A pass with any failed re-check retries at the poll cadence until it
676            // comes back clean; only a clean pass waits out the full cadence again.
677            self.last_rescan = clean.then(Instant::now);
678        }
679        Ok(())
680    }
681
682    /// Force the batched re-scope on the tick the watcher recovers.
683    ///
684    /// The same clock `apply` clears after a failed per-event re-scope, cleared
685    /// for the same reason: a stale deny set means serving content a takedown
686    /// covers, which is slashable. An outage is when the set is most likely to
687    /// have drifted and when the cadence helps least, since the reconcile does
688    /// not run at all while the route is errored.
689    fn on_recovered(&mut self) {
690        clear_cadence_on_recovery(&mut self.last_rescan, self.rescan_interval);
691    }
692}
693
694/// Outcome of one batched re-scope pass.
695struct RescanOutcome {
696    /// `true` iff every entry was re-verified with no failed re-check. A
697    /// shutdown-cancelled pass is unclean (`false`); it is decidedly **not** a
698    /// drift window, but that is enforced in `multiplexed_poller::run` (which
699    /// suppresses the backoff edge under a cancelled token), not here (#1321).
700    clean: bool,
701    /// Distinct hashes this pass could not enforce (`Recheck::Failed` — a cache
702    /// error or a scope `eth_call` failure). Feeds `blacklist_enforcement_failures`
703    /// so the slashable "deny-set not fully enforced" condition has a metric of its
704    /// own. A shutdown-cancelled pass reports the failures observed **before** the
705    /// cancel (not `0`).
706    failed: u64,
707    /// The failure that decides a boot attempt's fate: an eviction failure if
708    /// any (a local disk fault a retry does not repair), else the first scope
709    /// read failure. A boot attempt reports it as the cause, so it can tell a
710    /// deterministic fault from a transient one.
711    decisive_failure: Option<RecheckFailure>,
712}
713
714/// What a re-scope pass does after a scope read fails.
715#[derive(Clone, Copy, Debug, PartialEq, Eq)]
716enum OnScopeFailure {
717    /// Re-check every remaining hash: the periodic re-scope enforces every hash
718    /// it can, and retries the rest on the next tick.
719    Continue,
720    /// Stop the pass: a boot attempt with a failed scope read is lost and
721    /// retries whole, so its remaining reads only add load to a failing
722    /// provider.
723    Stop,
724}
725
726/// Re-scope every distinct hash in `known` (one scope `eth_call` per hash, not per
727/// regional entry) and evict those now in scope. Interruptible by shutdown between
728/// hashes; `on_scope_failure` decides whether a failed scope read ends the pass.
729async fn rescan<P>(
730    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
731    operator: Address,
732    cache: &CacheEngine,
733    warming: &Arc<WarmingAllowance>,
734    state: &mut WatcherState,
735    shutdown: &CancellationToken,
736    on_scope_failure: OnScopeFailure,
737) -> RescanOutcome
738where
739    P: Provider + Clone,
740{
741    let snapshot = state.distinct_hashes();
742    let mut evicted = 0usize;
743    let mut failed = 0u64;
744    let mut decisive_failure: Option<RecheckFailure> = None;
745    let mut clean = true;
746    for hash in snapshot {
747        if shutdown.is_cancelled() {
748            return RescanOutcome {
749                clean: false,
750                failed,
751                decisive_failure,
752            };
753        }
754        match recheck(contract, operator, cache, warming, state, hash).await {
755            Recheck::Evicted => evicted = evicted.saturating_add(1),
756            Recheck::NoAction => {}
757            Recheck::Failed(cause) => {
758                clean = false;
759                failed = failed.saturating_add(1);
760                let is_scope = matches!(cause, RecheckFailure::Scope(_));
761                let replaces = match &decisive_failure {
762                    None => true,
763                    Some(RecheckFailure::Scope(_)) => !is_scope,
764                    Some(RecheckFailure::Evict(_)) => false,
765                };
766                if replaces {
767                    decisive_failure = Some(cause);
768                }
769                if is_scope && on_scope_failure == OnScopeFailure::Stop {
770                    break;
771                }
772            }
773        }
774    }
775    if evicted > 0 {
776        info!(evicted, "blacklist watcher evicted blacklisted blobs");
777    }
778    if failed > 0 {
779        // The negative counterpart to the `evicted` line: the aggregate count of
780        // entries left unenforced this pass (#1319).
781        warn!(
782            unenforced = failed,
783            "blacklist re-scope could not enforce every entry"
784        );
785    }
786    RescanOutcome {
787        clean,
788        failed,
789        decisive_failure,
790    }
791}
792
793/// Handle one live log: `HashBlacklisted` records the `(region, hash)` entry and
794/// re-checks the hash; `HashRemoved` drops exactly that entry (hygiene — eviction
795/// stays sticky, and same-hash entries in other regions survive). `Ok(true)` iff a
796/// re-check failed and needs a prompt retry.
797///
798/// `Err` is reserved for an undecodable ORIGIN-class log: until the next
799/// re-enumeration there is nothing to sweep against, so
800/// skipping one is a silent deny-set gap — aborting the tick holds the scan cursor
801/// so the readiness gate keeps the router closed rather than opening on a set we
802/// know is incomplete. An undecodable HASH log is skipped (re-scoped every pass,
803/// sticky eviction).
804async fn handle_log<P>(
805    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
806    operator: Address,
807    cache: &CacheEngine,
808    warming: &Arc<WarmingAllowance>,
809    state: &mut WatcherState,
810    log: Log,
811) -> Result<bool>
812where
813    P: Provider + Clone,
814{
815    match log.topic0() {
816        Some(topic) if *topic == HashBlacklisted::SIGNATURE_HASH => Ok(on_blacklisted_log(
817            contract, operator, cache, warming, state, &log,
818        )
819        .await
820        .is_failed()),
821        Some(topic) if *topic == HashRemoved::SIGNATURE_HASH => {
822            on_removed_log(contract, operator, cache, state, &log).await;
823            Ok(false)
824        }
825        Some(topic) if *topic == OriginBlacklistUpdated::SIGNATURE_HASH => {
826            on_origin_log(state, &log)?;
827            Ok(false)
828        }
829        Some(topic) if *topic == OperatorBlacklisted::SIGNATURE_HASH => {
830            on_operator_log(state, &log, true)?;
831            Ok(false)
832        }
833        Some(topic) if *topic == OperatorBlacklistCleared::SIGNATURE_HASH => {
834            on_operator_log(state, &log, false)?;
835            Ok(false)
836        }
837        _ => Ok(false),
838    }
839}
840
841/// Decode an `OperatorBlacklisted` / `OperatorBlacklistCleared` log and apply it to
842/// the same origin deny-set `OriginBlacklistUpdated` feeds.
843fn on_operator_log(state: &mut WatcherState, log: &Log, blacklisted: bool) -> Result<()> {
844    let operator = if blacklisted {
845        match OperatorBlacklisted::decode_log_data(&log.inner.data) {
846            Ok(event) => event.operator,
847            Err(err) => return Err(undecodable_origin_log(&err, log, "OperatorBlacklisted")),
848        }
849    } else {
850        match OperatorBlacklistCleared::decode_log_data(&log.inner.data) {
851            Ok(event) => event.operator,
852            Err(err) => {
853                return Err(undecodable_origin_log(
854                    &err,
855                    log,
856                    "OperatorBlacklistCleared",
857                ));
858            }
859        }
860    };
861    state.set_origin(operator, blacklisted, log_block(log));
862    debug!(%operator, blacklisted, "blacklist watcher: operator blacklist updated");
863    Ok(())
864}
865
866/// The block a tail log was mined in. A log without one never comes back from
867/// `eth_getLogs`; one that did would count as the newest change, which a
868/// re-enumeration keeps.
869fn log_block(log: &Log) -> u64 {
870    log.block_number.unwrap_or(u64::MAX)
871}
872
873/// An origin-class log we cannot decode is an ENFORCEMENT failure, not a parse
874/// curiosity, so it aborts the tick and holds the scan cursor.
875///
876/// The hash events can afford to skip-and-continue: they are re-scoped every pass
877/// and eviction is sticky. The origin events have no such per-tick backstop, so a
878/// skipped log is gone until the next
879/// re-enumeration and the deny-set is silently short an entry meanwhile. Holding
880/// the cursor lets the readiness gate keep the router closed rather than opening on
881/// a set we know is incomplete.
882fn undecodable_origin_log(err: &alloy::sol_types::Error, log: &Log, event: &str) -> anyhow::Error {
883    anyhow::anyhow!("{err}").context(format!(
884        "undecodable {event} log at block {:?} tx {:?}; refusing to advance the scan cursor \
885         past an unreadable takedown event",
886        log.block_number, log.transaction_hash
887    ))
888}
889
890fn on_origin_log(state: &mut WatcherState, log: &Log) -> Result<()> {
891    let event = match OriginBlacklistUpdated::decode_log_data(&log.inner.data) {
892        Ok(event) => event,
893        Err(err) => return Err(undecodable_origin_log(&err, log, "OriginBlacklistUpdated")),
894    };
895    state.set_origin(event.origin, event.blacklisted, log_block(log));
896    debug!(
897        origin = %event.origin,
898        blacklisted = event.blacklisted,
899        "blacklist watcher: origin blacklist updated"
900    );
901    Ok(())
902}
903
904/// Decode a `HashBlacklisted` log, record its `(region, hash)` entry, and re-check
905/// the hash. An undecodable log is [`Recheck::NoAction`] (skipped, per the
906/// `LogSink` contract).
907async fn on_blacklisted_log<P>(
908    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
909    operator: Address,
910    cache: &CacheEngine,
911    warming: &Arc<WarmingAllowance>,
912    state: &mut WatcherState,
913    log: &Log,
914) -> Recheck
915where
916    P: Provider + Clone,
917{
918    match HashBlacklisted::decode_log_data(&log.inner.data) {
919        Ok(event) => {
920            let hash = Hash::from_bytes(event.hash.0);
921            state.add_entry(event.region, hash);
922            recheck(contract, operator, cache, warming, state, hash).await
923        }
924        Err(err) => {
925            warn!(error = %err, "blacklist watcher: undecodable HashBlacklisted log");
926            Recheck::NoAction
927        }
928    }
929}
930
931/// Decode a `HashRemoved` log and drop exactly that `(region, hash)` entry.
932///
933/// Also retires the hash from the governance deny-set, but only on a definitive
934/// out-of-scope read: `HashRemoved` is per-region, and a same-hash entry under
935/// another region can still cover this operator. `isHashBlacklistedForOperator` is
936/// the authoritative union, so it — not the event — decides. An RPC failure keeps
937/// the hash denied, the conservative direction (the hash stays evicted regardless,
938/// so the cost is a stale refusal *code*, not a stale refusal).
939async fn on_removed_log<P: Provider + Clone>(
940    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
941    operator: Address,
942    cache: &CacheEngine,
943    state: &mut WatcherState,
944    log: &Log,
945) {
946    match HashRemoved::decode_log_data(&log.inner.data) {
947        Ok(event) => {
948            let hash = Hash::from_bytes(event.hash.0);
949            state.remove_entry(event.region, hash, log_block(log));
950            if cache.is_chain_denied(hash) {
951                lift_deny_if_out_of_scope(contract, operator, cache, hash).await;
952            }
953            debug!(
954                region = %event.region,
955                %hash,
956                "blacklist entry removed on-chain (local eviction stays sticky)"
957            );
958        }
959        Err(err) => warn!(error = %err, "blacklist watcher: undecodable HashRemoved log"),
960    }
961}
962
963/// Lift the governance deny on `hash` only on a definitive out-of-scope read. A
964/// failed read keeps the deny: over-denying is the safe direction.
965async fn lift_deny_if_out_of_scope<P: Provider + Clone>(
966    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
967    operator: Address,
968    cache: &CacheEngine,
969    hash: Hash,
970) {
971    match scope_check(contract, operator, hash).await {
972        Ok(false) => undeny_hash(cache, hash),
973        Ok(true) => {}
974        Err(err) => warn!(
975            %hash,
976            error = %sanitize_err_chain(&err),
977            "blacklist watcher: isHashBlacklistedForOperator failed after HashRemoved; \
978             keeping the hash governance-denied"
979        ),
980    }
981}
982
983/// Outcome of one scope re-check, so callers can distinguish an enforcement
984/// *failure* (retry promptly — the entry may be live and slashable) from a
985/// legitimately out-of-scope entry (the periodic re-scope keeps watching it).
986#[derive(Debug)]
987enum Recheck {
988    /// The hash was in scope and its eviction succeeded.
989    Evicted,
990    /// Nothing to do: already evicted, or currently out of scope.
991    NoAction,
992    /// The scope read or the eviction failed — the entry is retained and must be
993    /// re-checked promptly.
994    Failed(RecheckFailure),
995}
996
997impl Recheck {
998    const fn is_failed(&self) -> bool {
999        matches!(self, Self::Failed(_))
1000    }
1001}
1002
1003/// Why a re-check failed.
1004#[derive(Debug)]
1005enum RecheckFailure {
1006    /// The `isHashBlacklistedForOperator` read failed (RPC error or timeout).
1007    Scope(anyhow::Error),
1008    /// The local cache could not evict the hash (a disk error).
1009    Evict(anyhow::Error),
1010}
1011
1012/// Record that `hash` is refused because *governance* blacklisted it, and publish
1013/// that to the live deny-set the delivery path reads. This is what selects the
1014/// `chain_hash_denied` refusal reason over the local-eviction one; both sign
1015/// `Declined`.
1016fn deny_hash(cache: &CacheEngine, hash: Hash) {
1017    cache.set_chain_denied_one(hash, true);
1018}
1019
1020/// Stop treating `hash` as governance-denied. The hash stays *refused* — eviction
1021/// is sticky and one-way — so this only moves its refusal reason from
1022/// `chain_hash_denied` back to the eviction, which is what any other evicted hash
1023/// answers. The wire class stays `Declined`.
1024fn undeny_hash(cache: &CacheEngine, hash: Hash) {
1025    cache.set_chain_denied_one(hash, false);
1026}
1027
1028/// Evict `hash` if in scope. Out-of-scope (`Ok(false)`) and RPC-error (`Err`)
1029/// hashes keep their `known` entries for the next re-scope (callers insert before
1030/// calling); evicted hashes drop *all* their regional entries — eviction is sticky
1031/// and region-independent.
1032async fn recheck<P>(
1033    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1034    operator: Address,
1035    cache: &CacheEngine,
1036    warming: &Arc<WarmingAllowance>,
1037    state: &mut WatcherState,
1038    hash: Hash,
1039) -> Recheck
1040where
1041    P: Provider + Clone,
1042{
1043    if cache.is_evicted(hash) {
1044        // Back-fill the governance deny-set for a hash that is already evicted but
1045        // not yet recorded as governance-denied. `evicted.log` persists the eviction
1046        // but records no cause, and the deny-set lives only in memory, so after a
1047        // restart every takedown evicted on a prior boot would otherwise keep
1048        // answering `EvictedSinceProbe` forever.
1049        if !cache.is_chain_denied(hash) {
1050            match scope_check(contract, operator, hash).await {
1051                Ok(true) => deny_hash(cache, hash),
1052                Ok(false) => {}
1053                Err(err) => warn!(
1054                    %hash,
1055                    error = %sanitize_err_chain(&err),
1056                    "blacklist watcher: isHashBlacklistedForOperator failed for an evicted \
1057                     hash; its governance deny waits for the next re-enumeration"
1058                ),
1059            }
1060        }
1061        state.drop_hash(hash);
1062        return Recheck::NoAction;
1063    }
1064    match scope_check(contract, operator, hash).await {
1065        Ok(true) => {
1066            // Deny before evicting: eviction is what retires the hash from `known`,
1067            // and `known` is the retry backstop. Denying first also means an
1068            // eviction that fails on a disk error still stops the serving, since
1069            // `CacheEngine::refuses` honors this set too.
1070            deny_hash(cache, hash);
1071            match evict(cache, warming, hash).await {
1072                Ok(()) => {
1073                    state.drop_hash(hash);
1074                    Recheck::Evicted
1075                }
1076                Err(err) => Recheck::Failed(RecheckFailure::Evict(err)),
1077            }
1078        }
1079        Ok(false) => Recheck::NoAction,
1080        Err(err) => {
1081            warn!(
1082                %hash,
1083                error = %sanitize_err_chain(&err),
1084                "blacklist watcher: isHashBlacklistedForOperator failed; keeping for re-scope"
1085            );
1086            Recheck::Failed(RecheckFailure::Scope(err))
1087        }
1088    }
1089}
1090
1091/// `isHashBlacklistedForOperator` bounded by the shared
1092/// [`chain_events::DEFAULT_RPC_CALL_TIMEOUT`]. `Err` on timeout or RPC error;
1093/// each caller logs it with what the failure means on its path.
1094///
1095/// [`chain_events::DEFAULT_RPC_CALL_TIMEOUT`]: crate::chain_events::DEFAULT_RPC_CALL_TIMEOUT
1096async fn scope_check<P>(
1097    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1098    operator: Address,
1099    hash: Hash,
1100) -> Result<bool>
1101where
1102    P: Provider + Clone,
1103{
1104    let hash_key = B256::from(*hash.as_bytes());
1105    timed(
1106        None,
1107        "isHashBlacklistedForOperator",
1108        contract
1109            .isHashBlacklistedForOperator(hash_key, operator)
1110            .call(),
1111    )
1112    .await
1113    .with_context(|| format!("isHashBlacklistedForOperator({hash})"))
1114}
1115
1116/// Evict `hash` from the cache (durable + sticky). A failure is logged here and
1117/// returned.
1118async fn evict(cache: &CacheEngine, warming: &Arc<WarmingAllowance>, hash: Hash) -> Result<()> {
1119    match cache.evict(hash).await {
1120        Ok(()) => {
1121            info!(%hash, "evicted blacklisted blob (ADR 011 compliance)");
1122            // ADR 041: drop the warming tag for the evicted hash, so a later
1123            // reuse of this slot can never credit a stale source's allowance.
1124            warming.forget(hash);
1125            Ok(())
1126        }
1127        Err(err) => {
1128            // Through `anyhow` so the log carries the whole chain down to the I/O
1129            // cause, not only the cache error's top line.
1130            let err = anyhow::Error::new(err).context(format!("evict {hash}"));
1131            warn!(%hash, error = %sanitize_err_chain(&err), "blacklist watcher: evict failed; will retry");
1132            Err(err)
1133        }
1134    }
1135}
1136
1137/// One boot attempt: enumerate the current on-chain deny-set at one pinned
1138/// block, seed the live origin deny-set the delivery path reads, then enforce —
1139/// deny + evict every enumerated hash that is in scope right now, decided by the
1140/// same `isHashBlacklistedForOperator` liveness the tail uses (so a lapsed
1141/// emergency entry is not enforced).
1142///
1143/// The pass stops at the first failed scope read: the attempt is lost and
1144/// retries whole, so the remaining reads only add load to a failing provider.
1145/// An unclean pass fails the attempt with its decisive failure as the cause: a
1146/// local eviction error is a [`BootFault`], since a retry does not repair a
1147/// disk, and otherwise the scope read keeps its typed RPC error for the retry
1148/// classifier. The inline pass is not shutdown-interruptible; the periodic
1149/// re-scope on the sink is.
1150#[allow(clippy::too_many_arguments)]
1151async fn enumerate_and_enforce<P>(
1152    reads: &ContractReads<P>,
1153    contract: &ContentBlacklist::ContentBlacklistInstance<P>,
1154    operator: Address,
1155    cache: &CacheEngine,
1156    warming: &Arc<WarmingAllowance>,
1157    denylist: &Arc<ContentDenylist>,
1158    metrics: &Arc<Metrics>,
1159    shutdown: &CancellationToken,
1160) -> Result<BootEnforced>
1161where
1162    P: Provider + Clone,
1163{
1164    let snapshot = bootstrap_snapshot(reads, operator).await?;
1165    denylist.set_chain_origins(snapshot.origins.clone());
1166    let mut state = WatcherState {
1167        known: snapshot.known,
1168        denylist: Arc::clone(denylist),
1169        origin_changes: HashMap::new(),
1170        removed_entries: HashMap::new(),
1171    };
1172    let RescanOutcome {
1173        clean,
1174        failed,
1175        decisive_failure,
1176    } = rescan(
1177        contract,
1178        operator,
1179        cache,
1180        warming,
1181        &mut state,
1182        shutdown,
1183        OnScopeFailure::Stop,
1184    )
1185    .await;
1186    if failed > 0 {
1187        metrics.blacklist_enforcement_failure(failed);
1188    }
1189    if !clean {
1190        let cause = match decisive_failure {
1191            Some(RecheckFailure::Scope(err)) => err,
1192            Some(RecheckFailure::Evict(err)) => BootFault(format!("{err:#}")).into(),
1193            // Only a cancelled pass is unclean with no failure, and nothing
1194            // cancels the boot pass's token.
1195            None => anyhow::anyhow!("enforcement pass was cancelled"),
1196        };
1197        return Err(cause.context(format!(
1198            "initial ContentBlacklist enforcement could not enforce every entry ({failed} failed)"
1199        )));
1200    }
1201    Ok(BootEnforced {
1202        block: snapshot.block,
1203        origin_count: snapshot.origins.len(),
1204        state,
1205    })
1206}
1207
1208/// A clean boot attempt: the deny-set enumerated and enforced.
1209struct BootEnforced {
1210    /// The pinned block the enumeration read, where the live tail starts.
1211    block: u64,
1212    /// Live origin and operator addresses in the seeded origin deny-set.
1213    origin_count: usize,
1214    /// The enforced deny-set the sink carries forward.
1215    state: WatcherState,
1216}
1217
1218/// Enumerate the on-chain deny-set at one pinned block, enforce it, then return
1219/// the [`Route`] that follows the live tail seeded at that block on the shared
1220/// multiplexed poller.
1221///
1222/// The pinned block sits `SNAPSHOT_LAG_MARGIN_BLOCKS` below head (or at head for
1223/// a contract deployed inside the margin), so the readiness gate opens on the
1224/// deny-set as of that block. A takedown inside the margin is enforced by the
1225/// first tail tick, which replays the margin.
1226///
1227/// One boot attempt is the enumeration plus the enforcement pass. An attempt that
1228/// cannot read the chain (block or enumeration RPC error) or cannot enforce
1229/// every entry retries on `boot`'s budget. `initial_sync_tx` fires once, when
1230/// the attempts resolve: `Ok` after a clean pass, so the runtime can gate the
1231/// ALPN router on blacklist enforcement being live, or `Err` on a deterministic
1232/// fault or an exhausted budget. That failure also returns `Err`, so the router
1233/// never opens on an un-vetted deny-set. `rescan_interval` is the batched
1234/// re-enumeration + re-scope cadence.
1235///
1236/// The returned route carries a [`SinkSource::Factory`]: the blacklist sink must
1237/// observe the poller's own shutdown token (its re-scope polls it between per-hash
1238/// `eth_call`s), which the poller mints only at spawn — so the sink is built
1239/// inside that spawn from the freshly-minted token.
1240#[allow(clippy::too_many_arguments)]
1241pub(crate) async fn bootstrap<P>(
1242    provider: P,
1243    contract_addr: Address,
1244    operator: Address,
1245    cache: CacheEngine,
1246    warming: Arc<WarmingAllowance>,
1247    head: Arc<dyn HeadSource>,
1248    rescan_interval: Duration,
1249    initial_sync_tx: oneshot::Sender<InitialSyncResult>,
1250    metrics: &Arc<Metrics>,
1251    denylist: Arc<ContentDenylist>,
1252    chain_freshness: ChainFreshness,
1253    boot: &BootRetry,
1254) -> Result<Route>
1255where
1256    P: Provider + Clone + 'static,
1257{
1258    let contract = ContentBlacklist::new(contract_addr, provider.clone());
1259    let reads = ContractReads {
1260        contract: contract.clone(),
1261        head: Arc::clone(&head),
1262    };
1263    let initial_sync = InitialSyncGate::new(initial_sync_tx);
1264
1265    // An attempt that cannot enforce every entry is retried whole, unless its
1266    // first failure is deterministic; re-seeding and re-enforcing are
1267    // idempotent.
1268    let boot_shutdown = CancellationToken::new();
1269    let attempt = boot
1270        .run("ContentBlacklist boot snapshot", || {
1271            enumerate_and_enforce(
1272                &reads,
1273                &contract,
1274                operator,
1275                &cache,
1276                &warming,
1277                &denylist,
1278                metrics,
1279                &boot_shutdown,
1280            )
1281        })
1282        .await;
1283    let BootEnforced {
1284        block: snapshot_block,
1285        origin_count,
1286        state,
1287    } = match attempt {
1288        Ok(done) => done,
1289        Err(err) => {
1290            // Fail CLOSED: signal the readiness gate so the runtime keeps every ALPN
1291            // listener shut, then abort startup — a node that cannot vet the
1292            // takedown set must not serve.
1293            initial_sync.signal(Err(format!(
1294                "initial ContentBlacklist sync failed: {}",
1295                sanitize_err_chain(&err)
1296            )));
1297            return Err(err).context("enumerate and enforce the ContentBlacklist boot snapshot");
1298        }
1299    };
1300    // The boot enumeration + enforcement read the chain successfully, so seed the
1301    // freshness clock: a node that has just vetted its deny-set must not read as
1302    // stale in the window before the first poll tick stamps it.
1303    chain_freshness.stamp();
1304    initial_sync.signal(Ok(()));
1305
1306    info!(
1307        %contract_addr,
1308        %operator,
1309        snapshot_block,
1310        known_hashes = state.known.len(),
1311        origins = origin_count,
1312        "blacklist compliance watcher enumerated its boot snapshot"
1313    );
1314
1315    let sink_metrics = Arc::clone(metrics);
1316    // Unlike the flush-only sinks, `BlacklistSink` must observe the *same* token
1317    // the poller cancels: `rescan` polls it between per-hash `eth_call`s so a
1318    // large deny-set re-scope yields promptly to shutdown. The poller mints one
1319    // token at spawn and hands it to this factory, so sink and loop share it
1320    // (#1236).
1321    let sink_factory: SinkSource = SinkSource::Factory(Box::new(move |shutdown| {
1322        Box::new(BlacklistSink {
1323            contract,
1324            reads,
1325            operator,
1326            cache,
1327            warming,
1328            state,
1329            shutdown: shutdown.clone(),
1330            rescan_interval: rescan_interval.max(Duration::from_secs(1)),
1331            // Boot enumeration + enforcement just ran; first backstop is one
1332            // interval out.
1333            last_rescan: Some(Instant::now()),
1334            metrics: sink_metrics,
1335        }) as Box<dyn crate::chain_events::multiplexed_poller::ErasedSink>
1336    }));
1337
1338    Ok(Route {
1339        addresses: vec![contract_addr],
1340        topic0s: blacklist_route_topic0s(),
1341        start: blacklist_cursor_start(snapshot_block),
1342        sink: sink_factory,
1343        label: "blacklist",
1344        on_established: Some(metric_hook(
1345            metrics,
1346            Metrics::blacklist_watcher_cycle_established,
1347        )),
1348        on_backoff: Some(metric_hook(
1349            metrics,
1350            Metrics::blacklist_watcher_backoff_started,
1351        )),
1352        // Every successful poll tick (including idle ones) both stamps the
1353        // liveness gauge and refreshes the serve-path staleness clock — the two
1354        // are the same signal, so they fire from one hook.
1355        on_tick_success: Some({
1356            let metrics = Arc::clone(metrics);
1357            let freshness = chain_freshness.clone();
1358            Box::new(move || {
1359                metrics.blacklist_watcher_tick();
1360                freshness.stamp();
1361            })
1362        }),
1363        on_task_panic: Some(metric_hook(
1364            metrics,
1365            Metrics::blacklist_watcher_task_panicked,
1366        )),
1367    })
1368}
1369
1370/// The blacklist route's demux key: hash takedowns plus both origin-blacklist
1371/// paths. Split out from [`bootstrap`] so the exact topic0 set is
1372/// unit-testable without a provider.
1373///
1374/// Origin blacklisting rides the same scan (ADR 011 § Hash Evasion). Like the
1375/// hash events, it has no version counter to detect a missed one — the
1376/// re-enumeration is the backstop. `addOperator` is the PRIMARY governance
1377/// origin-blacklist path — it writes a SEPARATE mapping and emits
1378/// `OperatorBlacklisted`/`OperatorBlacklistCleared`, never
1379/// `OriginBlacklistUpdated`. `OriginAssignment` unions the two mappings
1380/// on-chain; watching only the first would leave the delivery gate enforcing
1381/// the softer list and missing the voted one.
1382fn blacklist_route_topic0s() -> Vec<B256> {
1383    vec![
1384        HashBlacklisted::SIGNATURE_HASH,
1385        HashRemoved::SIGNATURE_HASH,
1386        OriginBlacklistUpdated::SIGNATURE_HASH,
1387        OperatorBlacklisted::SIGNATURE_HASH,
1388        OperatorBlacklistCleared::SIGNATURE_HASH,
1389    ]
1390}
1391
1392/// The blacklist route's cursor start: seed the tail at the enumeration
1393/// snapshot block. No durable cursor and no historical replay — the boot
1394/// enumeration rebuilt the whole deny-set, so the tail only follows forward
1395/// from the snapshot. Split out from [`bootstrap`] so the cursor shape is
1396/// unit-testable without a provider.
1397const fn blacklist_cursor_start(snapshot_block: u64) -> CursorStart {
1398    CursorStart::Seeded { at: snapshot_block }
1399}
1400
1401#[cfg(test)]
1402// Test-only: the assertion style below intentionally panics on the negative
1403// branch, and unwraps/expects on results the fixtures guarantee. Matches the
1404// convention in `content_deny.rs` / `config/mod.rs`.
1405#[allow(
1406    clippy::panic,
1407    clippy::unwrap_used,
1408    clippy::expect_used,
1409    clippy::indexing_slicing
1410)]
1411mod tests;