w3s

A Rust crate to the easily upload file or directory to Web3.Storage with optional encryption and compression.

Usage

Add the following line to your Cargo.toml file:

w3s = { version = "0.2", features = ["all"] }

Example

To upload a single file:

 let cid_result = w3s::helper::upload(
    path,  // the file path 
    auth_token,  // the api token created in web3.storage
    2,  // max concurrent upload threads
    Some(Arc::new(Mutex::new(|name, part, pos, total| {  // the progress listener
        println!("name: {name} part:{part} {pos}/{total}");
    }))),
    Some(None),  // if packed in CAR with custom block size, `Some(None)` means packed in CAR with default 256K block size
    Some(&mut b"abcd1234".to_owned()),  // if use encryption with password
    Some(None),  // if use compression with zstd level, `Some(None)` means uses compression with zstd level at 10
)
.await?;

To upload a directory:

let cid_result = w3s::helper::upload_dir(
    path,  // the folder path
    None,  // file filter which can bypass specific files
    auth_token,  // the api token created in web3.storage
    2,  // max concurrent upload threads
    Some(Arc::new(Mutex::new(|name, part, pos, total| {  // the progress listener
        println!("name: {name} part:{part} {pos}/{total}");
    }))),
    None,  // if use encryption with password
    None,  // if use compression with zstd level
)
.await?;

To download a compressed and encrypted file from IPFS gateway:

w3s::helper::download(
    url,  // the whole url pointing to the file under the IPFS geteway
    name,  // just a label that will later be passed to the progress listener
    &mut file,  // file to written
    Some(Arc::new(Mutex::new(|name, _, pos, total| {  // the progress listener
        println!("name: {name} {pos}/{total}");
    }))),
    None,  // start offset which should be `None` for compressed or encrypted file
    Some(b"abcd1234".to_vec()),  // use decryption with password
    true,  // use decompression
)
.await?;

Details about how to use

Please check the examples/ folder for different usage examples.

TODO

• Composable chain writer
• Gateways availability checker
• Single file upload though api.web3.storage/upload
• Single CAR upload though api.web3.storage/car
• Single file/CAR concurrent uploads
• Single file/CAR upload with compression
• Single file/CAR upload with encryption
• Auto split for >100MB single file upload
• CAR generation from single file
• Single file/car download
• Single file/car download with decompression and decryption
• Directory upload by CAR writer
• Directory upload with compression and encryption
• Code comments
• Documentation

Chain writer

The w3s crate contains several writers for the upload tasks. You can put writers according to your needs.

For example, if you'd like to compress and encrypt your data before upload:

Get the file -> compression writer -> encryption writer -> split writer -> upload writer

If you'd like to upload a CAR file:

Get the file -> CAR writer -> upload writer

If you'd like to encrypt the file before CAR it and upload:

Get the file -> encryption writer -> CAR writer -> upload writer