Struct BatchProcessor 

pub struct BatchProcessor { /* private fields */ }

High-performance batch processor for parallel operations

Provides parallel processing of multiple blocks, CIDs, and hash computations using Rayon’s thread pool.

Implementations

impl BatchProcessor

pub fn new() -> Self

Create a new batch processor with default settings (SHA2-256)

pub fn with_hash_algorithm(hash_algorithm: HashAlgorithm) -> Self

Create a batch processor with a specific hash algorithm

pub fn create_blocks_parallel( &self, data_chunks: Vec<Bytes>, ) -> Result<Vec<Block>>

Create multiple blocks in parallel from data chunks

This is significantly faster than creating blocks sequentially when processing many chunks.

pub fn generate_cids_parallel( &self, data_chunks: Vec<Bytes>, ) -> Result<Vec<(Bytes, Cid)>>

Generate CIDs in parallel for multiple data chunks

Returns a vector of (data, CID) pairs.

pub fn verify_blocks_parallel(&self, blocks: &[Block]) -> Result<()>

Verify multiple blocks in parallel

Returns Ok(()) if all blocks are valid, or an error for the first invalid block.

pub fn compute_hashes_parallel( &self, data_chunks: &[&[u8]], ) -> Result<Vec<Vec<u8>>>

Compute hashes in parallel for multiple data chunks

Returns a vector of hash digests.

pub fn total_bytes_parallel(&self, blocks: &[Block]) -> usize

Count total bytes across multiple blocks in parallel

pub fn filter_blocks_parallel<F>( &self, blocks: Vec<Block>, predicate: F, ) -> Vec<Block>

where F: Fn(&Block) -> bool + Sync + Send,

Find blocks matching a predicate in parallel

pub fn unique_cids_parallel(&self, blocks: &[Block]) -> Vec<Cid>

Collect unique CIDs from blocks in parallel

pub fn compress_data_parallel( &self, data_chunks: Vec<Bytes>, algorithm: CompressionAlgorithm, level: u8, ) -> Result<Vec<Bytes>>

Compress multiple data chunks in parallel

Compresses each data chunk using the specified algorithm and level. Returns a vector of compressed data, maintaining the same order as input.

Arguments

• data_chunks - Vector of data chunks to compress
• algorithm - Compression algorithm to use
• level - Compression level (0-9)

Returns

Vector of compressed data chunks

Example

use ipfrs_core::batch::BatchProcessor;
use ipfrs_core::CompressionAlgorithm;
use bytes::Bytes;

let processor = BatchProcessor::new();
let data = vec![
    Bytes::from(vec![0u8; 1000]),
    Bytes::from(vec![1u8; 1000]),
];

let compressed = processor.compress_data_parallel(
    data,
    CompressionAlgorithm::Zstd,
    3
).unwrap();
assert_eq!(compressed.len(), 2);

pub fn decompress_data_parallel( &self, compressed_chunks: Vec<Bytes>, algorithm: CompressionAlgorithm, ) -> Result<Vec<Bytes>>

Decompress multiple compressed chunks in parallel

Decompresses each chunk using the specified algorithm. Returns a vector of decompressed data, maintaining the same order as input.

Arguments

• compressed_chunks - Vector of compressed data chunks
• algorithm - Compression algorithm that was used

Returns

Vector of decompressed data chunks

Example

use ipfrs_core::batch::BatchProcessor;
use ipfrs_core::CompressionAlgorithm;
use bytes::Bytes;

let processor = BatchProcessor::new();
let data = vec![Bytes::from(vec![0u8; 1000])];

let compressed = processor.compress_data_parallel(
    data.clone(),
    CompressionAlgorithm::Lz4,
    3
).unwrap();

let decompressed = processor.decompress_data_parallel(
    compressed,
    CompressionAlgorithm::Lz4
).unwrap();
assert_eq!(decompressed, data);

pub fn analyze_compression_ratios_parallel( &self, data_chunks: &[Bytes], algorithm: CompressionAlgorithm, level: u8, ) -> Result<Vec<f64>>

Analyze compression ratios for multiple data chunks in parallel

Computes compression ratio estimates for each chunk. Returns a vector of ratios (compressed_size / original_size), where lower is better.

Arguments

• data_chunks - Vector of data chunks to analyze
• algorithm - Compression algorithm to use for estimation
• level - Compression level (0-9)

Returns

Vector of compression ratios (0.0 to 1.0, where 0.5 means 50% size reduction)

Example

use ipfrs_core::batch::BatchProcessor;
use ipfrs_core::CompressionAlgorithm;
use bytes::Bytes;

let processor = BatchProcessor::new();
let data = vec![
    Bytes::from(vec![0u8; 1000]), // Highly compressible
    Bytes::from(vec![1u8; 1000]), // Highly compressible
];

let ratios = processor.analyze_compression_ratios_parallel(
    &data,
    CompressionAlgorithm::Zstd,
    3
).unwrap();
assert_eq!(ratios.len(), 2);
// Repetitive data should compress well (ratio < 0.5)
assert!(ratios[0] < 0.5);

Trait Implementations

impl Default for BatchProcessor

fn default() -> Self

Returns the “default value” for a type.