Blocks iterator

Iterates over Bitcoin blocks, decoding data inside Bitcoin Core's blocks directory.

Features:

Blocks are returned in height order, it avoids following reorgs (see max_reorg parameter)
Blocks come with metadata like all block's previous outputs, it allows computing transactions fee or verifying scripts in the blockchain.

Iteration modes

In rust programs

Used as a library blocks could be iterated:

via the blocks_iterator::iter method like done in outputs_versions example
via the blocks_iterator::par_iter method like done in verify example. par_iter is preferred when the task to be perforemd is computational costly.

Through Pipes

Other than inside Rust programs, ordered blocks with prevouts could be iterated using Unix pipes.

$ cargo build --release 
$ cargo build --release --examples
$ ./target/release/blocks_iterator --blocks-dir /Volumes/Transcend/bitcoin-testnet/testnet3/blocks --network testnet --max-reorg 40 | ./target/release/examples/most_output_pipe | ./target/release/examples/heaviest_pipe >/dev/null
...
[2021-10-21T10:10:24Z INFO  most_output_pipe] most_output tx is d28305817238ee92e5d9ac0d81c3bf5ecf7399528e6d87226d726e4070c7e665 with #outputs: 30001
[2021-10-21T10:10:24Z INFO  heaviest_pipe] heaviest tx is 73e64e38faea386c88a578fd1919bcdba3d0b3af7b6302bf6ee1b423dc4e4333 with weight: 3999608

Memory requirements and performance

Running (cargo run --release -- --network X --blocks-dir Y >/dev/null) on threadripper 1950X, Testnet @ 2130k, Mainnet @ 705k. Spinning disk.

Network	`--skip--prevout`	`--max-reorg`	`utxo-db`	Memory	Time
Mainnet	true	6	no	33MB	1h:00m
Mainnet	false	6	no	5.3GB	1h:29m
Mainnet	false	6	1 run	201MB	9h:42m
Mainnet	false	6	2 run	113MB	1h:05m
Testnet	true	40	no	123MB	3m:03s
Testnet	false	40	no	1.4GB	8m:02s
Testnet	false	40	1 run	247MB	16m:12s
Testnet	false	40	2 run	221MB	8m:32s

Examples

Run examples with:

cargo run --release --example verify

heaviest find the transaction with greatest weight
most_output find the transaction with most output
outputs_versions Count outputs witness version
signatures_in_witness Count signatures in witness
verify verify transactions in blocks using libbitcoin-consensus

Similar projects

bitcoin-iterate this project inspired blocks_iterator, the differences are:
- bitcoin-iterate is C-based
- bitcoin-iterate it's more suited for shell piping, while blocks_itearator is intended to use as a rust library
- bitcoin-iterate doesn't give previous outputs information
- bitcoin-iterate is making two passage over blocks*.dat while blocks_iterator is doing one pass keeping out-order-blocks in memory (the latter is faster at the expense of higher memory usage)
rust-bitcoin-indexer this project requires longer setup times (about 9 hours indexing) and a postgre database, once the indexing is finished it allows fast queries on relational database.
electrs specifically intended for the Electrum protocol