Bitcoin slices

ZERO allocations parse library for Bitcoin data structures.

Data is accessed by providing visitor structs for the data the user is interested in.

Data structures are read-only and parsed data must be in memory, no streaming API.

Tradeoffs

Check the CONS before using this library, use rust-bitcoin if they are too restrictive for your case.

Pros

• Deserialization is amazingly fast, since no allocation is made during parsing.
• Serialization is incredibly fast, since a slice of the serialized data is kept in the structure.
• hashing a little faster because slice are ready without the need of re-serializing data.
• No mandatory dependency.
• No standard.
• Calculate txid and block hash via optional dep bitcoin_hashes or sha2.
• Visitor pattern to visit just what you are interested in.

Cons

• Full data must be in memory, there is no streaming (Read/Write) API.
• Data structure are read-only, cannot be modified.
• Visitor pattern requires user-built data structure for visiting.

Test

cargo test

Bench

RUSTFLAGS='--cfg=bench' cargo +nightly bench --all-features

test bsl::block::bench::block_deserialize            ... bench:     289,421 ns/iter (+/- 46,179)
test bsl::block::bench::block_deserialize_bitcoin    ... bench:   2,719,666 ns/iter (+/- 459,186)
test bsl::block::bench::block_sum_outputs            ... bench:     288,248 ns/iter (+/- 39,013)
test bsl::block::bench::block_sum_outputs_bitcoin    ... bench:   2,607,791 ns/iter (+/- 321,212)
test bsl::block::bench::hash_block_txs               ... bench:   8,406,341 ns/iter (+/- 938,119)
test bsl::block::bench::hash_block_txs_bitcoin       ... bench:  11,843,590 ns/iter (+/- 1,052,109)
test bsl::block::bench::hash_block_txs_sha2          ... bench:   7,891,956 ns/iter (+/- 1,047,439)
test bsl::block_header::bench::block_hash            ... bench:       1,399 ns/iter (+/- 205)
test bsl::block_header::bench::block_hash_bitcoin    ... bench:       1,510 ns/iter (+/- 222)
test bsl::transaction::bench::tx_deserialize         ... bench:          38 ns/iter (+/- 8)
test bsl::transaction::bench::tx_deserialize_bitcoin ... bench:         219 ns/iter (+/- 30)
test bsl::transaction::bench::txid                   ... bench:       2,185 ns/iter (+/- 166)
test bsl::transaction::bench::txid_bitcoin           ... bench:       2,416 ns/iter (+/- 213)
test bsl::transaction::bench::txid_sha2              ... bench:       2,085 ns/iter (+/- 216)

• benches ending with _bitcoin use rust-bitcoin
• benches ending with _sha2 use sha2 lib instead of bitcoin_hashes

Comparison against rust-bitcoin

block_deserialize is almost 10 times faster then block_deserialize_bitcoin. It may see unfair comparison since you can't for example iterate transactions from the resulted object in case of block_deserialize, but looking at the sum_outputs example where a visitor is used to access every outputs in a block we se there isn't noticeable difference.

Hashing

block_hash and block_hash_bitcoin use the same code to hash, however bitcoin_slice is about 7% faster because use a slice already available instead of serializing back data. Similar results apply between txid and txid_bitcoin. The performance increase is more notable (30%) between hash_block_txs and hash_block_txs_bitcoin.

*_sha2 are not really representative on virtual CI machines since they are not hardware-accellerated.

Fuzz

Use cargo fuzz Run fuzzing with transaction as target.

cargo +nightly fuzz run transaction

Other target available in fuzz/fuzz_targets

Miniminze corpus:

cargo +nightly fuzz cmin transaction

Previous work and credits

• Bitiodine use similar visitor pattern (parser credited to Mathias Svensson)
• Some previous work on the idea to parse while reducing allocations in this PR
• Matt Corallo mentioned something like this in a comment in that PR

TODO

• create rotating buffer that consume and produce keeping a linear memory (rotate back when it can't append), this would overcome a bit the absence of streaming API
• implements network types
• add limited time fuzzing in CI