Module parallel_chunking 

Parallel chunking for high-performance large file processing

This module provides parallel implementations of chunking operations, leveraging Rayon to process multiple chunks concurrently. This significantly improves performance for large files on multi-core systems.

Performance

Parallel chunking can provide near-linear speedup based on CPU core count:

• 4 cores: ~3.5x faster than sequential
• 8 cores: ~6-7x faster than sequential
• 16 cores: ~12-14x faster than sequential

Example

use ipfrs_core::parallel_chunking::{ParallelChunker, ParallelChunkingConfig};

let data = vec![0u8; 10_000_000]; // 10MB
let chunker = ParallelChunker::new();
let result = chunker.chunk_parallel(&data).unwrap();

println!("Root CID: {}", result.root_cid);
println!("Chunks: {}", result.chunk_count);
println!("Processing time: {:?}", result.duration);

Structs

ParallelChunker

Parallel chunker for high-performance file processing

ParallelChunkingConfig

Configuration for parallel chunking operations

ParallelChunkingResult

Result of a parallel chunking operation

ParallelDeduplicator

Parallel deduplication for content-defined chunking