Module streaming_compression 

Streaming compression and decompression support

This module provides streaming compression and decompression capabilities for large data that cannot or should not be loaded entirely into memory. It integrates with the async streaming infrastructure and supports all compression algorithms.

Features

• Async streaming compression - Compress data on-the-fly as it’s streamed
• Async streaming decompression - Decompress data on-the-fly as it’s streamed
• All algorithms supported - Zstd, LZ4, and passthrough (None)
• Configurable buffer sizes - Tune memory usage vs performance
• Statistics tracking - Monitor bytes processed and compression ratios

Example

use ipfrs_core::streaming_compression::{CompressingStream, CompressionAlgorithm};
use tokio::io::AsyncReadExt;
use bytes::Bytes;

// Create data to compress
let data = Bytes::from(b"Hello, world! ".repeat(1000));
let cursor = std::io::Cursor::new(data.to_vec());

// Create a compressing stream
let mut stream = CompressingStream::new(cursor, CompressionAlgorithm::Zstd, 3)?;

// Read compressed data
let mut compressed = Vec::new();
stream.read_to_end(&mut compressed).await?;

println!("Original: {} bytes", data.len());
println!("Compressed: {} bytes", compressed.len());
println!("Ratio: {:.2}%", (compressed.len() as f64 / data.len() as f64) * 100.0);

Re-exports

pub use crate::compression::CompressionAlgorithm;

Structs

CompressingStream

A streaming compressor that compresses data on-the-fly

DecompressingStream

A streaming decompressor that decompresses data on-the-fly

StreamingStats

Statistics for streaming compression/decompression