Reader Writer Builder

• Reader Writer Builder
  • Introduction
  • Usage
  • Warning
  • Example
  • Sources and Sinks
  • Features
  • Contributing
  • License

Introduction

This crate provides a convenient way to build std::io::Readers and std::io::Writers by chaining transformations. Since readers and writers are defined simultaneously through the same builder they can be used as inverses of each other.

Usage

Add this to your Cargo.toml:

[dependencies]
rw-builder = "0.2.0"

Warning

This crate is provided "as is" without warranty of any kind. Furthermore, this crate is still in its infancy and lacks significant real world exposure. We try our best to ensure the quality of this crate, but we should warn you that it's more a proof of concept than something to be used in production at the moment.

Example

Let's say you have some application state you want to encrypt and store on disk. Once the application starts up you want to read that state back into memory. A good practice when encrypting is to compress the data beforehand so you may feel the desire to chain some readers and writers together.

use flate2::Compression;
use rw_builder::{FileBuilder, Result, RwBuilder, RwBuilderExt};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct ApplicationState {
    ...
}

fn main() -> Result<()> {
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let builder = FileBuilder::new("/some/file".into())
        .buffered()
        .chacha20(key.into(), nonce.into())
        .deflate(Compression::fast())
        .wincode();
    let mut state: ApplicationState = builder.load()?;
    // Change the state
    builder.save(&state)
}

The builder ensures the order of the readers will match the order of the writers so there's no opportunity for mistakes.

Writing something similar in the usual way is much more verbose and error prone.

use anyhow::Result;
use cipher::{KeyIvInit, StreamCipher};
use flate2::{Compression, Status};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct ApplicationState {
    ...
}

fn main() -> Result<()> {
    let mut state = load_application_state()?;
    // Change the state
    save_application_state(&state)
}

fn save_application_state(state: &ApplicationState) -> Result<()> {
    let serialized = wincode::encode(&state)?;
    let mut compressor = flate2::Compress::new(Compression::fast(), false);
    let mut output = vec![];
    assert_eq!(
        compressor.compress_vec(&serialized, &mut output, flate2::FlushCompress::Sync)?,
        Status::StreamEnd
    );
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let mut decoder = chacha20::ChaCha20::new(&key.into(), &nonce.into());
    decoder.apply_keystream(&mut output);
    Ok(std::fs::write("/some/file", output)?)
}

fn load_application_state() -> Result<ApplicationState> {
    let mut buffer = std::fs::read("/some/file")?;
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let mut encoder = chacha20::ChaCha20::new(&key.into(), &nonce.into());
    encoder.apply_keystream(&mut buffer);
    let mut decompress = flate2::Decompress::new(false);
    let mut output = vec![];
    assert_eq!(
        decompress.decompress_vec(
            buffer.as_slice(),
            &mut output,
            flate2::FlushDecompress::Sync,
        )?,
        Status::StreamEnd
    );
    Ok(wincode::decode(output.as_slice())?)
}

This second example doesn't support streaming, which is necessary in the case of large files. Notice how the save and load functionality is described in reverse order from each other and that configuration options like the file location and the encryption key need to be made available in multiple locations. This is needlessly challenging to maintain when compared to the former example.

Sources and Sinks

You may have noticed that the FileBuilder struct and the wincode function have a special role. They are examples of a source and a sink respectively. Sources are a typical starting point for chaining builders, since they can be constructed without an inner builder. Sinks are a typical ending point for chaining builders, since they can interface with other types than &[u8] which Read and Write are restricted to.

Features

To provide the functionality of many different readers and writers this crate has many optional dependencies which are enabled through a predefined set of features. The example above requires the wincode, chacha20 and flate2 features. Currently, the following features are available:

• wincode: includes the serde and wincode crates and enables the wincode sink on the RwBuilderExt trait.
• rmp_serde: includes the serde and rmp-serde crates and enables the rmp_serde MessagePack sink on the RwBuilderExt trait.
• flate2: includes the flate2 crate and enables the crc, deflate, gz and zlib compressions.
• zstd: includes the zstd crate and enables zstd compression.
• bzip2: includes the bzip2 crate and enables bzip2 compression.
• lz4_flex: includes the lz4_flex crate and enables lz4_flex compression.
• chacha20: includes the cipher and chacha20 crates and enables chacha20 symmetric encryption.
• salsa20: includes the cipher and salsa20 crates and enables salsa20 symmetric encryption.
• aes_ctr: includes the aes and ctr crates and enables aes128_ctr and aes256_ctr symmetric encryption.
• digest: includes the digest crate and enables the generic hash checksumming combinator.
• sha2: includes the sha2 crate and enables sha256 and sha512 hashing.
• crc32fast: includes the crc32fast crate and enables crc32fast checksumming.
• base64: includes the base64 crate and enables the base64 encoding/decoding modifier on the RwBuilderExt trait.
• serde_json: includes the serde and serde_json crates and enables the serde_json JSON sink on the RwBuilderExt trait.
• postcard: includes the serde and postcard crates and enables the postcard binary sink on the RwBuilderExt trait.

Development Environment

Although not required, it is recommended to use the provided Nix flake to set up your development environment. This avoids polluting your host machine and ensures you are using the correct version of Rust and other dependencies.

To activate the environment:

nix develop

Alternatively, you can build the project directly with:

nix build

Examples & Integration Tests

The tests/ directory contains highly-documented integration tests that demonstrate how to construct complex pipelines for real-world scenarios. We encourage developers to look at these tests to understand how to use this crate:

• tests/secure_archival.rs: Demonstrates chaining File -> AES-256 -> Deflate -> Postcard to securely archive sensitive Rust structures directly to disk.
• tests/json_streaming.rs: Demonstrates base64 encoding a JSON stream on the fly.
• tests/combinations.rs: Showcases complex combinations like roundtripping, dealing with sink limitations, and proper Compression / Encryption ordering.

You can run these integration tests natively via cargo:

cargo test --test secure_archival --all-features
cargo test --test json_streaming --all-features

Benchmarks

Because rw-builder constructs a transparent proxy of std::io readers and writers, the overhead of the builder pattern is practically zero.

A criterion benchmark suite is included in benches/rw_benchmark.rs to measure the overhead of chaining transformations.

Hardware Details:

• CPU: AMD Ryzen 9 3900X 12-Core Processor
• RAM: 32 GB
• OS: Linux

Throughput (1MB Payload):

• Raw Vec::write_all: ~22.5µs (43.2 GiB/s)
• VecBuilder: ~20.9µs (46.5 GiB/s) (Zero overhead)
• Raw DeflateEncoder: ~137.1µs (7.12 GiB/s)
• Deflate via Builder: ~145.0µs (6.73 GiB/s) (< 5% overhead)

You can run the benchmarks yourself with:

cargo bench --all-features

Contributing

See CONTRIBUTING.md.

License

The rw-builder crate is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.

See LICENSE-APACHE.txt, LICENSE-MIT.txt, and COPYRIGHT.txt for details.