ethl

WIP

ethl is a powerful Ethereum log event ETL (Extract, Transform, Load) tool designed for efficient and reliable log processing. It provides:

• RPC utilities for streaming logs with support for multiple providers, fallback mechanisms, and retries.
• Tools to archive events into an opinionated Arrow + Parquet format for optimized storage and querying.
• Features enabling fast replay of specific events and continuous indexing for real-time use cases.

This tool was built to address the need for high-performance event processing and storage in Ethereum-based applications.

Getting Started

To get started with ethl, add it to your Cargo.toml via cargo install ethl:

[dependencies]
ethl = "0.1"

Then, include it in your project:

use ethl;

For detailed examples and usage, check the documentation.

Development nodes (anvil, hardhat, ganache)

ethl treats RPC provider responses as authoritative. Local development nodes expose two failure modes that can silently corrupt an event cache:

1. Silent empty responses past tip. Some nodes return {"result": []} for eth_getLogs on block ranges past their actual tip rather than erroring. ethl guards against this by calling eth_blockNumber whenever a batch returns no logs and refusing to advance the cursor when to_block exceeds the reported tip (yields RpcError::CursorPastTip). The overhead is one extra RPC per empty batch; batches that return logs are unaffected.

2. Synthetic blocks from local mining. Dev nodes mine new blocks in response to transactions you send, including simulations. Those blocks share a number with real chain blocks but contain only your local transactions — they are well-formed blocks as far as the API is concerned, so ethl cannot distinguish them from fork-proxied real blocks. If you index against a node that has been used for writes, treat the cache as poisoned: roll block_height back to a value from before the node was first used and rebuild from a real RPC.