Crate firewood_ffi

Expand description

§Firewood Golang FFI

The FFI package provides a golang FFI layer for Firewood.

§Usage

§Basic Usage

import (
    "context"
    "time"

    "github.com/ava-labs/firewood/ffi"
)

// Open a database with default configuration
db, err := ffi.New("/path/to/database_dir")
if err != nil {
    log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
defer db.Close(ctx)

§Configuration Options

Firewood uses the functional options pattern for configuration. You can customize the database by passing option functions:

db, err := ffi.New("/path/to/database_dir",
    ffi.WithTruncate(true),                    // Clear the database if it exists
    ffi.WithNodeCacheSizeInBytes(256_000_000), // Set node cache memory limit
    ffi.WithFreeListCacheEntries(50_000),      // Set freelist cache size
    ffi.WithRevisions(200),                    // Keep 200 historical revisions
    ffi.WithReadCacheStrategy(ffi.CacheAllReads), // Cache all reads
    ffi.WithRootStoreDir("/path/to/roots"),    // Store roots on disk
)

For detailed information about each configuration option, see the godoc for the With* functions

§Building Firewood Golang FFI

The Golang FFI layer uses a CGO directive to locate a C-API compatible binary built from Firewood. Firewood supports both seamless local development and a single-step compilation process for Go projects that depend or transitively depend on Firewood.

To do this, firewood.go includes CGO directives to include multiple search paths for the Firewood binary in the local target/ build directory and ffi/libs. For the latter, attach-static-libs GitHub Action pushes an FFI package for ethhash with static libraries attached for the following supported architectures:

x86_64-unknown-linux-gnu
aarch64-unknown-linux-gnu
aarch64-apple-darwin
x86_64-apple-darwin

to a separate repo firewood-go-ethhash (to avoid including binaries in the Firewood repo).

§Local Development

firewood.go includes CGO directives to include builds in the target/ directory.

Firewood prioritizes builds in the following order:

maxperf
release
debug

To use and test the Firewood FFI locally, you can run:

cargo build --profile maxperf
cd ffi
go test

To use a local build of Firewood for a project that depends on Firewood, you must redirect the go.mod to use the local version of Firewood FFI, for example:

go mod edit -replace github.com/ava-labs/firewood-go-ethhash/ffi=/path/to/firewood/ffi
go mod tidy

§Production Development Flow

Firewood pushes the FFI source code and attached static libraries to firewood-go-ethhash via attach-static-libs.

This enables consumers to utilize it directly without forcing them to compile Firewood locally. Go programs running on supported architectures can utilize firewood-go-ethhash/ffi just like any other dependency.

To trigger this build, attach-static-libs supports triggers for both manual GitHub Actions and tags, so you can create a mirror branch/tag on firewood-go-ethhash by either trigger a manual GitHub Action and selecting your branch or pushing a tag to Firewood.

§Hash Mode

Firewood implemented its own optimized merkle trie structure. To support Ethereum Merkle Trie hash compatibility, it also provides a feature flag ethhash.

This is an optional feature (disabled by default). To enable it for a local build, compile with:

cargo build -p firewood-ffi --features ethhash

To support development in Coreth, Firewood pushes static libraries for Ethereum-compatible hashing to firewood-go-ethhash with ethhash enabled by default. To use Firewood’s native hashing structure, you must still build the static library separately.

§Development

Iterative building is unintuitive for the ffi and some common sources of confusion are listed below.

§CGO Regeneration

As you edit any Rust code and save the file in VS Code, the firewood.h file is automatically updated with edited function and struct definitions. However, the Go linter will not recognize these changes until you manually regenerate the cgo wrappers. To do this, you can run go tool cgo firewood.go. Alternatively, in VS Code, right above the import "C" definition, you can click on the small letters saying “regenerate CGO definitions”. This will allow the linter to use the altered definitions.

Because the C header file is autogenerated from the Rust code, the naming matches exactly (due to the no_mangle macro). However, the C definitions imported in Go do not match exactly, and are prefixed with struct_. Function naming is the same as the header file. These names are generated by the go tool cgo command above.

It is possible that your editor does not properly recognize the C bindings, due to the nature of multi-module workspaces. If the Go code still compiles, this is likely the issue. To fix this, you can just ./scripts/run_just.sh setup-go-workspace from the repository root and refresh your editor. Changes to the go workspace should not be checked in.

§Testing

Although the VS Code testing feature does work, there are some quirks in ensuring proper building. The Rust code must be compiled separated, and sometimes the go test command continues to use a cached result. Whenever testing after making changes to the Rust/C builds, the cache should be cleared if results don’t seem correct. The Go testing suite can determine dynamically whether ethhash is enabled or not, so it can be run with either configuration. For each individual testing module, ensure the hashing method you compiled with matches the test.

To ensure there are no memory leaks, the easiest way is to use your preferred CLI tool (e.g. valgrind for Linux, leaks for macOS) and compile the tests into a binary. You must not compile a release binary to ensure all memory can be managed. An example flow is given below.

cd ffi
cargo build # use debug
go test -a -c -o binary_file # ignore cache
leaks --nostacks --atExit -- ./binary_file

Structs§

BorrowedSlice: A borrowed byte slice. Used to represent data that was passed in from C callers and will not be freed or retained by Rust code.
ChangeProofContext: FFI context for a parsed or generated change proof. This change proof has not been verified. Calling verify on it will generate a VerifiedChangeProofContext and consume the proof and replacing it with None.
CodeIteratorHandle
CommittedChangeProofArgs
CreateChangeProofArgs: Arguments for creating a change proof.
CreateIteratorResult
CreateProposalResult
CreateRangeProofArgs: Arguments for creating a range proof.
DatabaseHandle: A handle to the database, returned by fwd_open_db.
DatabaseHandleArgs: Arguments for creating or opening a database. These are passed to fwd_open_db
GetRevisionResult
HashKey: A database hash key, used in FFI functions that require hashes. This type requires no allocation and can be copied freely and dropped without any additional overhead.
IteratorHandle: An opaque wrapper around a BoxKeyValueIter and a reference to the ArcDynDbView backing it, preventing the view from being dropped while iteration is in progress.
LogArgs: Arguments for initializing logging for the Firewood FFI.
NextKeyRange: A key range that should be fetched to continue iterating through a range or change proof that was truncated. Represents a half-open range [start_key, end_key). If end_key is None, the range is unbounded and continues to the end of the keyspace.
OwnedKeyValuePair: Owned version of KeyValuePair, returned to ffi callers.
OwnedSlice: A Rust-owned vector of bytes that can be passed to C code.
ProposalHandle: An opaque wrapper around a Proposal that also retains a reference to the database handle it was created from.
ProposedChangeProofArgs
ProposedChangeProofContext: FFI context for a proposed change proof. It is created from calling propose on a VerifiedChangeProofContext and stores the database, proposal handle, and other parameters need to implement find_next_key. Calling commit on it will consume the proof, but find_next_key can still be called on it.
RangeProofContext: FFI context for for a parsed or generated range proof.
ReconstructedHandle: An opaque wrapper around a reconstructed view.
RevisionHandle
VerifiedChangeProofContext: FFI context for a verified change proof. It is created from calling verify on a ChangeProofContext and stores the parameters of that call in params. Calling propose on it will consume the proof to create a ProposedChangeProofContext.
VerifyChangeProofArgs: Arguments for verifying a change proof.
VerifyRangeProofArgs: Arguments for verifying a range proof.

Enums§

BatchOp: A batch operation passed to the FFI.
ChangeProofResult: A result type returned from FFI functions that create or parse change proofs.
CodeIteratorResult: A result type returned from FFI functions that create an code hash iterator
HandleResult: The result type returned from the open or create database functions.
HashResult: A result type returned from FFI functions return the database root hash. This may or may not be after a mutation.
IteratorResult: A result type returned from FFI functions that create an iterator
KeyValueBatchResult: A result type returned from iterator FFI functions
KeyValueResult: A result type returned from iterator FFI functions
Maybe: Maybe is a C-compatible optional type using a tagged union pattern.
NextKeyRangeResult
NodeHashAlgorithm: The hashing mode to use for the database.
ProposalResult: A result type returned from FFI functions that create a proposal but do not commit it to the database.
ProposedChangeProofResult
RangeProofResult: A result type returned from FFI functions that create or parse range proofs.
ReconstructedResult: A result type returned from FFI functions that create a reconstructed view.
RevisionResult: A result type returned from FFI functions that get a revision
ValueResult: A result type returned from FFI functions that retrieve a single value.
VerifiedChangeProofResult
VoidResult: The result type returned from an FFI function that returns no value but may return an error.

Traits§

CView: A trait that abstracts over database handles and proposal handles for creating proposals.

Functions§

fwd_batch: Puts the given key-value pairs into the database.
fwd_block_replay_flush: Flushes buffered block replay operations to disk.
fwd_change_proof_find_next_key_proposed: Returns the next key range that should be fetched after processing the current set of operations in a change proof that was truncated.
fwd_change_proof_from_bytes: Deserialize a ChangeProof from bytes.
fwd_change_proof_to_bytes: Serialize a ChangeProof to bytes.
fwd_close_db: Close and free the memory for a database handle
fwd_code_hash_iter_free: Frees the memory associated with a CodeIteratorHandle.
fwd_code_hash_iter_next: Advances the code hash iterator and returns the next code hash.
fwd_commit_proposal: Commits a proposal to the database.
fwd_db_change_proof: Create a change proof for the given range of keys between two roots.
fwd_db_commit_change_proof: Commit a change proof to the database.
fwd_db_dump: Dumps the Trie structure of the latest revision of the database to a DOT (Graphviz) format string for debugging.
fwd_db_propose_change_proof: Create a proposal from a change proof and return a ProposedChangeProofResult.
fwd_db_range_proof: Generate a range proof for the given range of keys for the latest revision.
fwd_db_verify_and_commit_range_proof: Verify and commit a range proof to the database.
fwd_db_verify_range_proof: Verify a range proof and prepare a proposal to later commit or drop. If the proof has already been verified, the cached validation context will be used to avoid re-verifying the proof.
fwd_free_change_proof: Frees the memory associated with a ChangeProofContext.
fwd_free_iterator: Consumes the IteratorHandle, destroys the iterator, and frees the memory.
fwd_free_owned_bytes: Consumes the OwnedBytes and frees the memory associated with it.
fwd_free_owned_key_value_batch: Consumes the OwnedKeyValueBatch and frees the memory associated with it.
fwd_free_owned_kv_pair: Consumes the OwnedKeyValuePair and frees the memory associated with it.
fwd_free_proposal: Consumes the ProposalHandle, cancels the proposal, and frees the memory.
fwd_free_proposed_change_proof: Frees the memory associated with a ProposedChangeProofContext.
fwd_free_range_proof: Frees the memory associated with a RangeProofContext.
fwd_free_reconstructed: Consumes the ReconstructedHandle and frees the memory associated with it.
fwd_free_revision: Consumes the RevisionHandle and frees the memory associated with it.
fwd_free_verified_change_proof: Frees the memory associated with a VerifiedChangeProofContext.
fwd_gather: Gather latest metrics for this process.
fwd_get_from_proposal: Gets the value associated with the given key from the proposal provided.
fwd_get_from_reconstructed: Gets the value associated with the given key from the reconstructed view provided.
fwd_get_from_revision: Gets the value associated with the given key from the provided revision handle.
fwd_get_latest: Gets the value associated with the given key from the database for the latest revision.
fwd_get_revision: Gets a handle to the revision identified by the provided root hash.
fwd_iter_next: Retrieves the next item from the iterator.
fwd_iter_next_n: Retrieves the next batch of items from the iterator.
fwd_iter_on_proposal: Returns an iterator on the provided proposal optionally starting from a key
fwd_iter_on_reconstructed: Returns an iterator on the provided reconstructed view optionally starting from a key.
fwd_iter_on_revision: Returns an iterator optionally starting from a key in the provided revision.
fwd_open_db: Open a database with the given arguments.
fwd_proposal_dump: Dumps the Trie structure of a proposal to a DOT (Graphviz) format string for debugging.
fwd_propose_on_db: Proposes a batch of operations to the database.
fwd_propose_on_proposal: Proposes a batch of operations to the database on top of an existing proposal.
fwd_range_proof_code_hash_iter: Returns an iterator over the code hashes contained in the range proof. The iterator must be freed after use.
fwd_range_proof_find_next_key: Returns the next key range that should be fetched after processing the current set of key-value pairs in a range proof that was truncated.
fwd_range_proof_from_bytes: Deserialize a RangeProof from bytes.
fwd_range_proof_to_bytes: Serialize a RangeProof to bytes.
fwd_range_proof_verify: Verify a range proof against the given start and end keys and root hash. The proof will be updated with the validation context if the proof is valid to avoid re-verifying it during commit.
fwd_reconstruct_on_reconstructed: Reconstructs a batch of operations on top of an existing reconstructed view.
fwd_reconstruct_on_revision: Reconstructs a batch of operations on top of a historical revision.
fwd_reconstructed_dump: Dumps the Trie structure of a reconstructed view to a DOT (Graphviz) format string for debugging.
fwd_reconstructed_root_hash: Get the root hash of the reconstructed view.
fwd_revision_dump: Dumps the Trie structure of a revision to a DOT (Graphviz) format string for debugging.
fwd_root_hash: Get the root hash of the latest version of the database
fwd_start_logs: Start logs for this process.
fwd_start_metrics: Start metrics recorder for this process.
fwd_start_metrics_with_exporter: Start metrics recorder and exporter for this process.
fwd_verify_change_proof: Verify a change proof and return a VerifiedChangeProofResult.

Type Aliases§

BorrowedBatchOps: A type alias for a borrowed slice of BatchOps.
BorrowedBytes: A type alias for a borrowed byte slice.
KeyRange: A key range represented by a start key and an optional end key.
OwnedBytes: A type alias for a rust-owned byte slice.
OwnedKeyValueBatch: A type alias for a rust-owned byte slice.