rw-builder 0.2.0

Provides a convenient way to build `std::io::Read`ers and `std::io::Write`rs by chaining transformations
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201

# Reader Writer Builder


[![Test Status](https://github.com/rlangendam/rw-builder/workflows/Tests/badge.svg?event=push)](https://github.com/rlangendam/rw-builder/actions)
[![Crate](https://img.shields.io/crates/v/rw-builder.svg)](https://crates.io/crates/rw-builder)
[![API](https://docs.rs/rw-builder/badge.svg)](https://docs.rs/rw-builder)

- [Reader Writer Builder](#reader-writer-builder)
  - [Introduction](#introduction)
  - [Usage](#usage)
  - [Warning](#warning)
  - [Example](#example)
  - [Sources and Sinks](#sources-and-sinks)
  - [Features](#features)
  - [Contributing](#contributing)
  - [License](#license)

## Introduction

This crate provides a convenient way to build `std::io::Read`ers and `std::io::Write`rs by chaining transformations. Since readers and writers are defined simultaneously through the same builder they can be used as inverses of each other.

## Usage

Add this to your `Cargo.toml`:

```toml
[dependencies]
rw-builder = "0.2.0"
```

## Warning

This crate is provided "as is" without warranty of any kind. Furthermore, this crate is still in its infancy and lacks significant real world exposure. We try our best to ensure the quality of this crate, but we should warn you that it's more a proof of concept than something to be used in production at the moment.

## Example

Let's say you have some application state you want to encrypt and store on disk. Once the application starts up you want to read that state back into memory. A good practice when encrypting is to compress the data beforehand so you may feel the desire to chain some readers and writers together.

```rust
use flate2::Compression;
use rw_builder::{FileBuilder, Result, RwBuilder, RwBuilderExt};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct ApplicationState {
    ...
}

fn main() -> Result<()> {
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let builder = FileBuilder::new("/some/file".into())
        .buffered()
        .chacha20(key.into(), nonce.into())
        .deflate(Compression::fast())
        .wincode();
    let mut state: ApplicationState = builder.load()?;
    // Change the state
    builder.save(&state)
}
```
The builder ensures the order of the readers will match the order of the writers so there's no opportunity for mistakes.

Writing something similar in the usual way is much more verbose and error prone.
```rust
use anyhow::Result;
use cipher::{KeyIvInit, StreamCipher};
use flate2::{Compression, Status};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct ApplicationState {
    ...
}

fn main() -> Result<()> {
    let mut state = load_application_state()?;
    // Change the state
    save_application_state(&state)
}

fn save_application_state(state: &ApplicationState) -> Result<()> {
    let serialized = wincode::encode(&state)?;
    let mut compressor = flate2::Compress::new(Compression::fast(), false);
    let mut output = vec![];
    assert_eq!(
        compressor.compress_vec(&serialized, &mut output, flate2::FlushCompress::Sync)?,
        Status::StreamEnd
    );
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let mut decoder = chacha20::ChaCha20::new(&key.into(), &nonce.into());
    decoder.apply_keystream(&mut output);
    Ok(std::fs::write("/some/file", output)?)
}

fn load_application_state() -> Result<ApplicationState> {
    let mut buffer = std::fs::read("/some/file")?;
    let key = [0x42; 32];
    let nonce = [0x24; 12];
    let mut encoder = chacha20::ChaCha20::new(&key.into(), &nonce.into());
    encoder.apply_keystream(&mut buffer);
    let mut decompress = flate2::Decompress::new(false);
    let mut output = vec![];
    assert_eq!(
        decompress.decompress_vec(
            buffer.as_slice(),
            &mut output,
            flate2::FlushDecompress::Sync,
        )?,
        Status::StreamEnd
    );
    Ok(wincode::decode(output.as_slice())?)
}
```
This second example doesn't support streaming, which is necessary in the case of large files. Notice how the save and load functionality is described in reverse order from each other and that configuration options like the file location and the encryption key need to be made available in multiple locations. This is needlessly challenging to maintain when compared to the former example.

## Sources and Sinks

You may have noticed that the `FileBuilder` struct and the `wincode` function have a special role. They are examples of a source and a sink respectively. Sources are a typical starting point for chaining builders, since they can be constructed without an inner builder. Sinks are a typical ending point for chaining builders, since they can interface with other types than `&[u8]` which `Read` and `Write` are restricted to.

## Features

To provide the functionality of many different readers and writers this crate has many optional dependencies which are enabled through a predefined set of features. The example above requires the `wincode`, `chacha20` and `flate2` features. Currently, the following features are available:
* `wincode`: includes the `serde` and `wincode` crates and enables the `wincode` sink on the `RwBuilderExt` trait.
* `rmp_serde`: includes the `serde` and `rmp-serde` crates and enables the `rmp_serde` MessagePack sink on the `RwBuilderExt` trait.
* `flate2`: includes the `flate2` crate and enables the `crc`, `deflate`, `gz` and `zlib` compressions.
* `zstd`: includes the `zstd` crate and enables `zstd` compression.
* `bzip2`: includes the `bzip2` crate and enables `bzip2` compression.
* `lz4_flex`: includes the `lz4_flex` crate and enables `lz4_flex` compression.
* `chacha20`: includes the `cipher` and `chacha20` crates and enables `chacha20` symmetric encryption.
* `salsa20`: includes the `cipher` and `salsa20` crates and enables `salsa20` symmetric encryption.
* `aes_ctr`: includes the `aes` and `ctr` crates and enables `aes128_ctr` and `aes256_ctr` symmetric encryption.
* `digest`: includes the `digest` crate and enables the generic `hash` checksumming combinator.
* `sha2`: includes the `sha2` crate and enables `sha256` and `sha512` hashing.
* `crc32fast`: includes the `crc32fast` crate and enables `crc32fast` checksumming.
* `base64`: includes the `base64` crate and enables the `base64` encoding/decoding modifier on the `RwBuilderExt` trait.
* `serde_json`: includes the `serde` and `serde_json` crates and enables the `serde_json` JSON sink on the `RwBuilderExt` trait.
* `postcard`: includes the `serde` and `postcard` crates and enables the `postcard` binary sink on the `RwBuilderExt` trait.

## Development Environment

Although not required, it is recommended to use the provided Nix flake to set up your development environment. This avoids polluting your host machine and ensures you are using the correct version of Rust and other dependencies.

To activate the environment:

```bash
nix develop
```

Alternatively, you can build the project directly with:

```bash
nix build
```

## Examples & Integration Tests

The `tests/` directory contains highly-documented integration tests that demonstrate how to construct complex pipelines for real-world scenarios. We encourage developers to look at these tests to understand how to use this crate:

- **`tests/secure_archival.rs`**: Demonstrates chaining `File` -> `AES-256` -> `Deflate` -> `Postcard` to securely archive sensitive Rust structures directly to disk.
- **`tests/json_streaming.rs`**: Demonstrates base64 encoding a JSON stream on the fly.
- **`tests/combinations.rs`**: Showcases complex combinations like roundtripping, dealing with sink limitations, and proper `Compression` / `Encryption` ordering.

You can run these integration tests natively via cargo:
```bash
cargo test --test secure_archival --all-features
cargo test --test json_streaming --all-features
```

## Benchmarks

Because `rw-builder` constructs a transparent proxy of `std::io` readers and writers, the overhead of the builder pattern is practically zero. 

A `criterion` benchmark suite is included in `benches/rw_benchmark.rs` to measure the overhead of chaining transformations.

**Hardware Details:**
- **CPU**: AMD Ryzen 9 3900X 12-Core Processor
- **RAM**: 32 GB
- **OS**: Linux

**Throughput (1MB Payload):**
- **Raw `Vec::write_all`**: ~22.5µs (43.2 GiB/s)
- **`VecBuilder`**: ~20.9µs (46.5 GiB/s) *(Zero overhead)*
- **Raw `DeflateEncoder`**: ~137.1µs (7.12 GiB/s)
- **`Deflate` via Builder**: ~145.0µs (6.73 GiB/s) *(< 5% overhead)*

You can run the benchmarks yourself with:
```bash
cargo bench --all-features
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

The rw-builder crate is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), with portions covered by various BSD-like licenses.

See [LICENSE-APACHE.txt](LICENSE-APACHE.txt), [LICENSE-MIT.txt](LICENSE-MIT.txt), and [COPYRIGHT.txt](COPYRIGHT.txt) for details.