Module streaming

Expand description

Streaming decompression API for memory-efficient extraction.

This module provides a streaming API for extracting 7z archives with bounded memory usage. It’s designed for processing large archives that shouldn’t be fully loaded into memory.

§Overview

The streaming API provides:

StreamingArchive: High-level streaming archive reader
StreamingConfig: Configuration for memory bounds and behavior
StreamingEntry: Individual entry with streaming data access
EntryIterator: Iterator over archive entries
MemoryTracker: Memory usage monitoring and limits
RandomAccessReader: Random access for non-solid archives
DecoderPool: Stream pooling for efficient solid archive access
Various sinks: For extracting to different Write implementations

§Example

use zesven::streaming::{StreamingArchive, StreamingConfig};

// Open with custom configuration
let config = StreamingConfig::new()
    .max_memory_buffer(32 * 1024 * 1024)  // 32 MiB
    .verify_crc(true);

let mut archive = StreamingArchive::open_with_config(file, password, config)?;

// Process entries one at a time
for entry_result in archive.entries()? {
    let mut entry = entry_result?;

    if entry.is_directory() {
        std::fs::create_dir_all(entry.name())?;
        continue;
    }

    let mut file = std::fs::File::create(entry.name())?;
    entry.extract_to(&mut file)?;
}

§Solid Archives

7z archives can be “solid”, meaning multiple files are compressed together as a single stream. This achieves better compression but requires sequential decompression.

For solid archives:

Use StreamingArchive::entries() for sequential access
Entries must be processed in order
Skipping an entry still decompresses it (but discards the data)

For non-solid archives:

RandomAccessReader enables direct access to specific entries
Entries can be accessed in any order

§Memory Management

Use StreamingConfig to control memory usage:

// Low memory configuration
let config = StreamingConfig::low_memory();

// High performance configuration (more memory, better throughput)
let config = StreamingConfig::high_performance();

// Custom configuration
let config = StreamingConfig::new()
    .max_memory_buffer(16 * 1024 * 1024)
    .read_buffer_size(32 * 1024);

Use MemoryTracker for fine-grained allocation control:

let tracker = MemoryTracker::new(64 * 1024 * 1024);

// Allocate with RAII guard
let guard = tracker.allocate(1024)?;
// Memory automatically released when guard drops

§Write Sinks

The sink module provides various Write implementations:

BoundedVecSink: In-memory buffer with size limit
Crc32Sink: Computes CRC while discarding data
NullSink: Discards data (for skipping)
CountingSink: Wraps another writer and counts bytes
ProgressSink: Wraps another writer with progress callbacks
TeeSink: Writes to two destinations simultaneously

Structs§

BoundedReader: A bounded reader that limits the number of bytes read.
BoundedVecSink: In-memory Vec sink with size limit.
CancellationToken: Request cancellation of ongoing parallel extraction.
ChainedReader: A chained reader that reads from multiple readers in sequence.
CountingSink: Counting sink that wraps another Write and counts bytes.
Crc32Sink: Streaming hash computation sink.
DecoderPool: Stream pool for caching decompression streams.
EntryIterator: Iterator that yields archive entries one at a time with streaming decompression.
EntryLocation: Location of an entry within a solid block.
EntryLocator: Entry locator for finding entries by various criteria.
ExtractAllResult: Result of extracting all entries.
MemoryEstimate: Memory usage estimate for an operation.
MemoryGuard: RAII guard that releases memory when dropped.
MemoryTracker: Memory usage tracker for streaming operations.
NullSink: Null sink for skip operations.
ParallelExtractionOptions: Options for parallel extraction.
ParallelExtractionResult: Result of parallel extraction.
ParallelFolderExtractor: Parallel folder extractor for non-solid archives.
PoolStats: Statistics for pool usage.
PooledDecoder: A decoder borrowed from the pool.
ProgressSink: Progress-reporting sink that calls a callback periodically.
ProgressiveReader: A reader wrapper that tracks read progress.
ProgressiveReaderWithCallback: A progressive reader with a callback for progress updates.
RandomAccessReader: Random access reader for non-solid archives.
RandomEntryReader: Reader for a single entry accessed randomly.
SkippedEntry: Information about an entry that was skipped during archive parsing.
SolidBlockInfo: Information about a solid block.
SolidBlockStreamReader: Sequential reader for solid archive blocks.
SolidEntryLocator: Helper to determine entry positions within solid blocks.
StreamingArchive: High-level streaming archive reader.
StreamingConfig: Configuration for streaming decompression with memory bounds.
StreamingEntry: Represents a single entry during streaming iteration.
SystemMemoryInfo: Information about system memory.
TeeSink: Tee sink that writes to two writers simultaneously.
TrackedBuffer: A tracked allocation that owns a Vec<u8>.

Enums§

CompressionMethod: Compression method identifier for memory estimation.
SkipReason: The reason why an archive entry was skipped during parsing.

Traits§

ExtractToSink: Trait for extracting entries to Write sinks.