s-zip

███████╗      ███████╗██╗██████╗ 
██╔════╝      ╚══███╔╝██║██╔══██╗
███████╗█████╗  ███╔╝ ██║██████╔╝
╚════██║╚════╝ ███╔╝  ██║██╔═══╝ 
███████║      ███████╗██║██║     
╚══════╝      ╚══════╝╚═╝╚═╝     

High-performance streaming ZIP library for Rust backends. Process multi-gigabyte archives with constant ~5MB memory usage.

Features

• 🚀 Streaming I/O - Constant memory regardless of archive size
• 🔐 AES-256 Encryption - WinZip-compatible password protection (sync + async)
• ⚡ Async/Await - Full Tokio support with encryption
• 🌩️ Cloud Storage - Direct streaming to/from S3, GCS, MinIO
• 💪 Parallel Compression - 2-4x speedup on multi-core CPUs
• 📦 ZIP64 - Files >4GB supported
• 🗜️ Multiple Codecs - DEFLATE, Zstd (3x faster compression)

Quick Start

[dependencies]
s-zip = "0.11"

# With all features
s-zip = { version = "0.11", features = ["async", "encryption", "async-zstd", "cloud-all"] }

Basic Usage

use s_zip::{StreamingZipWriter, StreamingZipReader};

// Write
let mut writer = StreamingZipWriter::new("output.zip")?;
writer.start_entry("file.txt")?;
writer.write_data(b"Hello, World!")?;
writer.finish()?;

// Read
let mut reader = StreamingZipReader::open("output.zip")?;
let data = reader.read_entry_by_name("file.txt")?;

Async with Encryption

use s_zip::AsyncStreamingZipWriter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = AsyncStreamingZipWriter::new("encrypted.zip").await?;
    writer.set_password("my_password");
    
    writer.start_entry("secret.txt").await?;
    writer.write_data(b"Confidential data").await?;
    writer.finish().await?;
    Ok(())
}

Cloud Storage (S3/GCS)

use s_zip::{AsyncStreamingZipWriter, cloud::S3ZipWriter};
use aws_sdk_s3::Client;

let config = aws_config::load_from_env().await;
let s3_client = Client::new(&config);

let writer = S3ZipWriter::builder()
    .client(s3_client)
    .bucket("my-bucket")
    .key("archive.zip")
    .build()
    .await?;

let mut zip = AsyncStreamingZipWriter::from_writer(writer);
zip.start_entry("data.json").await?;
zip.write_data(br#"{"status": "ok"}"#).await?;
zip.finish().await?;

What's New in v0.11.3

⚡ Performance & DRY refactor — zero-copy parallel compression, streaming decrypt, proptest coverage:

• True zero-copy parallel compression — compress_file_deflate in parallel.rs previously buffered the entire file into RAM before compressing. New CrcReader wrapper computes CRC32 on-the-fly through a File → BufReader(64KB) → CrcReader → DeflateEncoder pipeline. Peak RAM per task is now ~96 KB regardless of file size (was full file size × concurrency). Measured: 20 files × 5MB with 8 threads → ~16 MB peak (was ~320 MB).

• Streaming decryption — read_entry_streaming() on both sync and async readers now supports encrypted entries via new DecryptingReader<R> / AsyncDecryptingReader<R> wrappers. Call .finish() after reading to verify the HMAC-SHA1 tag.

• Shared format.rs module — ZIP constants, ZipEntry, and pure parsing helpers extracted from duplicated reader.rs/async_reader.rs. ~300 lines of duplicate code removed.

• Proptest fuzz coverage — 6 property tests in tests/proptest_zip_parsing.rs verify find_eocd_in_buffer, find_zip64_eocd_offset, and parse_zip64_extra_field never panic on arbitrary input.

Breaking Changes: None.

Migration from v0.11.2:

s-zip = { version = "0.11.3", features = ["async", "encryption"] }

Performance

Single-threaded (1MB compressible data, DEFLATE level 6):

• Write: ~1.5 ms → ~680 MB/s
• Async write: ~2.2 ms (~1.5× overhead vs sync at 1MB, converges at larger sizes)

Parallel compression (20 files × 5MB = 100MB):

• 2 threads: 731 MB/s, peak RAM 3 MB
• 4 threads: 2484 MB/s, peak RAM 2 MB
• 8 threads: 1434 MB/s, peak RAM 3 MB
• Process total peak RSS: 16 MB (bounded regardless of file size)

Memory: ~16 MB process peak even compressing 100MB in parallel (8 threads).

See CHANGELOG.md for full details.

Optional Features

Examples

Encryption (including streaming decrypt):

// Write encrypted
let mut writer = StreamingZipWriter::new("secure.zip")?;
writer.set_password("password123");
writer.start_entry("secret.txt")?;
writer.write_data(b"Confidential")?;
writer.finish()?;

// Read encrypted — full (HMAC verified before any bytes returned)
let mut reader = StreamingZipReader::open("secure.zip")?;
reader.set_password("password123");
let data = reader.read_entry_by_name("secret.txt")?;

// Read encrypted — streaming (decrypt on-the-fly, call finish() to verify HMAC)
let entry = reader.find_entry("secret.txt").unwrap().clone();
let mut stream = reader.read_entry_streaming(&entry)?;
std::io::copy(&mut stream, &mut output)?;
// stream is dropped here; HMAC is verified in DecryptingReader::finish()

Zstd Compression:

let mut writer = StreamingZipWriter::with_zstd("output.zip", 3)?;
writer.start_entry("data.bin")?;
writer.write_data(&large_data)?;
writer.finish()?;

Parallel Compression:

use s_zip::{AsyncStreamingZipWriter, ParallelConfig, ParallelEntry};

let entries = vec![
    ParallelEntry::new("file1.txt", "path/to/file1.txt"),
    ParallelEntry::new("file2.txt", "path/to/file2.txt"),
];

let config = ParallelConfig::balanced(); // 4 threads
let mut writer = AsyncStreamingZipWriter::new("output.zip").await?;
writer.write_entries_parallel(entries, config).await?;
writer.finish().await?;

In-Memory ZIP:

let buffer = Vec::new();
let cursor = std::io::Cursor::new(buffer);
let mut writer = StreamingZipWriter::from_writer(cursor)?;

writer.start_entry("data.txt")?;
writer.write_data(b"In-memory content")?;

let cursor = writer.finish()?;
let zip_bytes = cursor.into_inner();

More examples in examples/ directory.

Use Cases

• Web APIs - Generate ZIPs on-demand (Axum, Actix, Rocket)
• Cloud Pipelines - Stream directly to S3/GCS without local disk
• Data Exports - Large dataset exports with encryption
• ETL Jobs - Batch processing with bounded memory
• Microservices - Streaming responses over HTTP

Documentation

• API Docs: https://docs.rs/s-zip
• Performance: BENCHMARK_RESULTS.md
• Examples: examples/
• Changelog: CHANGELOG.md

Non-Goals

• Not a CLI tool (use zip/unzip for that)
• Not optimized for small files (<1KB)
• Not focused on desktop/GUI usage

License

MIT License - see LICENSE

Contributing

Contributions welcome! Please feel free to submit a Pull Request.

Author

Ton That Vu - @KSD-CO