Bitcoin Block Parser
Fast optimized parser for the bitcoin blocks
data with UTXO tracking.
Features
- Parses blocks into the Rust bitcoin crate
Block
format for easier manipulation - Tracks whether any
TxOut
in aTransaction
is spent or unspent - Tracks the
Amount
of everyTxIn
for calculating metrics such as fee rates - Uses memory-optimizations, multithreading, and cuckoo filters to provide the best performance
Requirements / Benchmarks
- You must be running a non-pruning bitcoin node (this is the default configuration)
- You should look at the table below to understand how much RAM you need (increases with # of blocks)
- We recommend using fast storage and a multithreaded CPU for best performance
Our benchmarks were run on NVMe storage with a 32-thread processor on 850,000 blocks:
Function | Time | Memory |
---|---|---|
parse() | 6 min | 2.5 GB |
parse_o() | 21 min | 3.4 GB |
parse_i() | 47 min | 35.4 GB |
parse_io() | 47 min | 10.9 GB |
write_filter() | 23 min | 6.6 GB |
Beware of running out-of-memory when using parse_i()
because unspent UTXOs are stored in RAM instead of a filter.
Usage
To parse blocks pass in the blocks
directory of your bitcoin node and call BlockParser::parse()
// Load all the block locations from the headers of the block files
let locations = parse?;
// Create a parser from a slice of the first 100K blocks
let parser = new;
// Iterates over all the blocks in height order
for parsed in parser.parse
If you need the input Amount
for every transaction you can use BlockParser::parse_i()
// `parse_i()` provides the input amounts for all transactions
for parsed in parser.parse_i
If you need to know if a transaction output was spent or unspent use BlockParser::parse_o()
// We have to write filter before getting any output information
parser.write_filter?;
// `parse_o()` provides whether the output was spent or unspent for all transactions
for parsed in parser.parse_o
If you want both input amounts and output status use BlockParser::parse_io()
// If we already wrote the filter file we don't need to write it again for the same blocks
for parsed in parser.parse_io