Skip to main content

Module batch

thread_flow

Module batch 

Source
Expand description

Batch file processing with optional parallel execution

This module provides utilities for processing multiple files efficiently:

  • CLI builds (default): Uses rayon for CPU parallelism across cores
  • Worker builds: Falls back to sequential processing (no threads in edge)

§Feature Gating

Parallel processing is controlled by the parallel feature flag:

  • Enabled (default): Multi-core parallel processing via rayon
  • Disabled (worker): Single-threaded sequential processing

§Usage

ⓘ
use thread_flow::batch::process_files_batch;

let results = process_files_batch(&file_paths, |path| {
    // Process each file
    analyze_file(path)
});

§Performance Characteristics

TargetConcurrency100 Files1000 Files
CLI (4 cores)Parallel~0.4s~4s
CLI (1 core)Sequential~1.6s~16s
WorkerSequential~1.6s~16s

Speedup: 2-4x on multi-core systems (linear with core count)

Functions§

process_batch
Process multiple items in batch with optional parallelism
process_files_batch
Process multiple files in batch with optional parallelism
try_process_files_batch
Try to process multiple files in batch, collecting errors