Module bulk_storage

Expand description

Bulk storage extraction over eth_call state overrides.

The default StorageBatchFetchFn issues one eth_getStorageAt per slot (JSON-RPC-batched, but still one billed request per slot — 20 CU each on Alchemy). This module implements the “bulk storage extraction” technique described by Dedaub, which packs thousands of slot reads into a single eth_call (26 CU on Alchemy, flat):

blog: https://dedaub.com/blog/bulk-storage-extraction/
reference implementation: https://github.com/Dedaub/storage-extractor

§Mechanism

eth_call accepts a state-override set that can replace the code at any address while leaving its storage intact. We override the target contract with a 23-byte handwritten extractor (STORAGE_EXTRACTOR_CODE, Dedaub’s bytecode, credited above) that treats calldata as a raw array of 32-byte slot keys, SLOADs each one, and returns the packed values — no function selector, no ABI:

[00] PUSH0            counter = 0
[01] JUMPDEST         loop:
[02] DUP1 CALLDATASIZE EQ
[05] PUSH1 0x13 JUMPI   -> exit when counter == calldatasize
[08] DUP1 CALLDATALOAD  slot key at calldata[counter]
[0a] SLOAD
[0b] DUP2 MSTORE        mem[counter] = value (counter doubles as mem offset)
[0d] PUSH1 0x20 ADD     counter += 32
[10] PUSH1 0x01 JUMP
[13] JUMPDEST CALLDATASIZE PUSH0 RETURN

Marginal cost is ~2,664 gas per slot (cold SLOAD 2,100 + calldata ~510 + loop ~30 + memory), so a default 50M-gas eth_call fits ~18,500 slots. BulkCallConfig::max_slots_per_call defaults to a conservative 10,000 (~27M gas), splitting larger requests across concurrent calls.

§Multi-contract batches

SLOAD reads the storage of the executing contract, so each target must run the extractor at its own address. To read many contracts in one round trip we additionally override MULTICALL3_ADDRESS with the canonical Multicall3 runtime (multicall3_runtime_code) and dispatch one aggregate3 call whose subcalls hit each overridden target. Overriding the dispatcher code unconditionally makes the scheme work on chains — and at historical blocks — where Multicall3 is not deployed.

§Semantics & caveats

Results are identical to eth_getStorageAt at the same block: absent slots (and slots of code-less accounts) read as zero.
The provider must support the state-override parameter of eth_call (Geth-lineage nodes, Reth, Erigon, and the major hosted providers all do). Providers that reject it surface a per-slot error; install a fallback via bulk_call_storage_fetcher_with_fallback to repair those with classic point reads.
True precompile addresses (0x01..=0x11 on mainnet) execute the precompile regardless of a code override; slots requested there fail the response-length check and surface as errors (repaired by the fallback when configured) rather than silently returning garbage.
STORAGE_EXTRACTOR_CODE uses PUSH0 (Shanghai). For pre-Shanghai chains set BulkCallConfig::pre_shanghai_extractor to use the equivalent STORAGE_EXTRACTOR_CODE_PRE_SHANGHAI.

§Wiring it into a cache

Since 0.2.0 this is every provider-backed cache’s default storage fetcher — no wiring needed. Tune it with EvmCacheBuilder::bulk_call_config, opt out with StorageFetchStrategy::PointRead, or compose it manually as below (e.g. over a custom fallback):

let provider = Arc::new(
    ProviderBuilder::new()
        .network::<AnyNetwork>()
        .connect_http("https://example-rpc.invalid".parse()?),
);
let mut cache = EvmCache::builder(provider.clone()).build().await;

// Keep the default point-read fetcher as a repair path, then route all
// batch storage fetches through call-override bulk extraction.
let fallback = cache
    .storage_batch_fetcher()
    .cloned()
    .expect("provider-backed cache has a default fetcher");
cache.set_storage_batch_fetcher(bulk_call_storage_fetcher_with_fallback(
    provider,
    BulkCallConfig::default(),
    fallback,
));

Structs§

AccountFieldsSample: Balance + code hash of one account, as sampled in-EVM by ACCOUNT_FIELDS_EXTRACTOR_CODE.
BlockContextSample: One block’s EVM-visible context, as sampled by BLOCK_CONTEXT_EXTRACTOR_CODE.
BulkCallConfig: Tuning knobs for the call-override bulk storage fetcher.
BulkFetcherStatus: A cheap, cloneable handle to a running bulk fetcher’s fallback state.
StorageProgram: A caller-supplied extraction program: arbitrary bytecode injected at target through a code override and executed by one eth_call.

Enums§

CallDispatch: How planned extraction chunks are shipped to the provider.

Constants§

ACCOUNT_FIELDS_EXTRACTOR_CODE: Account-fields extractor: calldata is a contiguous array of 32-byte left-padded addresses; the return data is [balance, extcodehash] (two words) per address, via the BALANCE and EXTCODEHASH opcodes.
BLOCK_CONTEXT_EXTRACTOR_CODE: Block-context extractor: no calldata; returns seven words — NUMBER, TIMESTAMP, BASEFEE, COINBASE, PREVRANDAO, GASLIMIT, CHAINID — straight from the EVM environment of the queried block. Piggybacks block-header context onto the same transport as slot loads without an eth_getBlockByNumber. Requires PUSH0 (Shanghai).
STORAGE_EXTRACTOR_CODE: Dedaub’s 23-byte storage extractor (see the module docs for the annotated disassembly). Calldata is a contiguous array of 32-byte slot keys; the return data is the corresponding array of 32-byte values. Requires PUSH0 (Shanghai).
STORAGE_EXTRACTOR_CODE_PRE_SHANGHAI: STORAGE_EXTRACTOR_CODE with both PUSH0s replaced by PUSH1 0x00 (jump targets re-pointed), for chains that have not activated Shanghai.

Functions§

bulk_call_storage_fetcher: Build a StorageBatchFetchFn backed by call-override bulk extraction.
bulk_call_storage_fetcher_with_fallback: bulk_call_storage_fetcher with a repair path.
bulk_call_storage_fetcher_with_status: bulk_call_storage_fetcher_with_fallback that also returns a BulkFetcherStatus handle for observing the fallback latch at runtime.
decode_multi_target_response: Decode an aggregate3 response produced by encode_multi_target_calldata back into one result tuple per requested (target, slot) pair.
decode_packed_values: Decode extractor return data (packed 32-byte words) into values.
encode_multi_target_calldata: ABI-encode one aggregate3 dispatch whose subcalls run the extractor at each (target, slots) pair. Subcalls use allowFailure = true so one failing target degrades to per-target errors instead of reverting the whole batch.
fetch_account_fields_bulk: Fetch balance + code hash for many accounts in one eth_call.
fetch_block_context: Sample a block’s EVM context in one eth_call (see BLOCK_CONTEXT_EXTRACTOR_CODE).
fetch_slots_bulk: Fetch storage slots in bulk via eth_call code overrides (async core).
multicall3_runtime_code: Runtime bytecode of Multicall3 (0xcA11bde05977b3631167028862bE2a173976CA11), as deployed on Ethereum mainnet. Injected as a code override at MULTICALL3_ADDRESS for multi-contract extraction so the dispatcher exists on every chain and at every historical block.
pack_slots_calldata: Pack slot keys into extractor calldata: the raw concatenation of each key’s 32-byte big-endian representation (no selector, no ABI).
planned_call_count: Number of eth_calls a request set will be split into under config.
run_storage_program: Execute one StorageProgram via eth_call and return its raw output.
run_storage_programs: Execute several StoragePrograms, batching programs with distinct targets into a single Multicall3-dispatched eth_call.