Crate arloader[−][src]
Expand description
SDK for uploading files in bulk to Arweave.
CLI
See README.md for usage instructions.
The main cli application is all in main.rs
following a pattern of specifying arguments,
matching them, and then in turn passing them to commands, which are all broken out
separately in the commands
module to facilitate re-use in other command
line applications or as library functions. There shouldn’t be anything called by the cli
application directly from the library. Everything gets composed into a command in the commands
module.
Library
Overview
The library is mostly focused on uploading files as efficiently as possible. Arweave has two different transaction formats and two different upload formats. Transactions can either be normal, single data item transactions (see transaction format for details), or bundle transactions (see bundle format for details). The bundle format, introduced mid-2021, bundles together individual data items into larger transactions, making uploading much more efficient and reducing network congestion. The library supports both formats, with the recommended approach being to use the bundle format.
There are also two upload formats, whole transactions and in chunks. Whole transaction can up uploaded
to the tx/
endpoint if they are less than 12 MB in total. Otherwise, you have to use the chunk/
endpoint
and upload chunk sizes that are less than 256 KB. Arloader includes functions for both.
Transactions and DataItems
Both formats start with chunking file data and creating merkle trees from the chunks. The merkle
tree logic can be found in the merkle
module. All of the hashing functions and other crypto
operations are in the crypto
module. Once the data is chunked, hashed, and a merkle root
calculated for it, it gets incorporated into either a Transaction
, which can be found in the
transaction
module, or a DataItem
(if it is going to be included in a bundle format transaction),
which can be found in the bundle
module.
Tags
Tags are structs with name
and value
properties that can be included with either Transaction
s or
DataItem
s. One subtlety is that for Transaction
s, Arweave expects the content at each key to be Base64 Url
encoded string, whereas for DataItems, Arweave expects utf8-encoded strings. Arloader includes two types of
tags to account for this, Tag<Base64>
and Tag<String>
, used for Transaction
and DataItem
,
respectively.
The Content-Type
tag is especially important as it is used by the Arweave gateways to communicate the content
type to browsers. Arloader includes a mime-type database that includes the appropriate content type
tag based on file extension for both Transaction
s and DataItem
s.
Bytes and Base64Url Data
The library takes advantage of Rust’s strong typing and trait model to store all data, signatures and
addresses as a Base64
struct with implementations for serialization and deserialization that automatically
convert the underlying bytes to and from the Base64Url format required for submission to Arweave.
Signing
A key part of constructing transactions is signing them. Arweave has a specific algorithm for generating the
digest that gets signed and then hashed to serve as a transaction id, called deep hash. It takes various elements
of either the Transaction
or DataItem
, including nested arrays of tags, and successively concatenates and
hashes them. The required elements are assembled using the ToItems
trait, implemented separately for Transaction
and DataItem
. Arloader’s implementation of the deep hash algorithm can be found in crypto::Provider::deep_hash
.
Higher Level Functions
The functions for creating transactions, data items and bundles are all consolidated on the Arweave
struct.
In general there are lower level functions for creating the items from data, that are then used in successively
higher level functions to create the items from a single file path and ultimately uploading streams of items in
parallel from collections of file paths.
Status Tracking
A key added functionality is the tracking and reporting on transaction statuses. There are two status structs,
Status
and BundleStatus
used for these purposes. They are essentially the same format, except that
BundleStatus
is modified to include references to all of the included DataItem
instead of just a
single Transaction
for Status
.
Solana
The functions for allowing payment to be made in SOL can be found in the solana
module.
Modules
Functions for Cli commands composed from library functions.
Functionality for creating and verifying signatures and hashing.
Errors propagated by library functions.
Functionality for chunking file data and calculating and verifying root ids.
Includes functionality for paying for transaction in SOL.
Data structures for reporting transaction statuses.
Data structures for serializing and deserializing Transaction
s and Tag
s.
Structs
Struct with methods for interacting with the Arweave network.
Tuple struct includes two elements: chunk of paths and aggregatge data size of paths.
Constants
Block size used for pricing calculations = 256 KB
Winstons are a sub unit of the native Arweave network token, AR. There are 1012 Winstons per AR.
Functions
Used when updating to determine wether files in a directory are BundleStatus
s.
Queries network and updates locally stored BundleStatus
structs.
Queries network and updates locally stored Status
structs.
Uploads a stream of bundles from Vec<PathsChunk>
s.
Uploads a stream of bundles from Vec<PathsChunk>
s, paying with SOL.
Uploads files matching glob pattern, returning a stream of Status
structs.
Uploads files matching glob pattern, returning a stream of Status
structs, paying with SOL.