Module parallel_pack 

Parallel packing pipeline for multi-threaded compression.

This module implements a producer-consumer architecture where:

• Main thread reads and chunks input
• Worker threads compress chunks in parallel
• Single dedup thread writes unique blocks

Architecture

                    ┌─→ Worker 1: Compress + Hash ─┐
                    │                                │
Main: Read chunks ──┼─→ Worker 2: Compress + Hash ──┼─→ Dedup + Write
                    │                                │   (sequential)
                    └─→ Worker N: Compress + Hash ─┘

Parallel (N cores)                                   Sequential

This design maximizes CPU utilization while keeping dedup single-threaded to avoid hash table locking overhead.

Structs

CompressedChunk

Compressed chunk with hash for deduplication

ParallelPackConfig

Configuration for parallel packing

RawChunk

Raw chunk data from input

Functions

process_chunks_parallel

Process chunks in parallel with multiple worker threads