Struct BatchExecutor 

pub struct BatchExecutor { /* private fields */ }

High-performance batch processor for HEDL files.

Orchestrates parallel or serial processing based on configuration and workload. Provides progress tracking and comprehensive error collection.

Thread Safety

BatchExecutor is thread-safe and can be shared across threads via Arc.

Examples

use hedl_cli::batch::{BatchExecutor, BatchConfig, ValidationOperation};
use std::path::PathBuf;

let processor = BatchExecutor::new(BatchConfig {
    parallel_threshold: 5,
    verbose: true,
    ..Default::default()
});

let files: Vec<PathBuf> = vec![
    "file1.hedl".into(),
    "file2.hedl".into(),
];

let results = processor.process(
    &files,
    ValidationOperation { strict: false },
    true,
)?;

if results.has_failures() {
    eprintln!("Some files failed validation");
    for failure in results.failures() {
        eprintln!("  - {}: {:?}", failure.path.display(), failure.result);
    }
}

Implementations

impl BatchExecutor

pub fn new(config: BatchConfig) -> Self

Create a new batch processor with the given configuration.

pub fn default_config() -> Self

Create a batch processor with default configuration.

pub fn process<O>( &self, files: &[PathBuf], operation: O, show_progress: bool, ) -> Result<BatchResults<O::Output>, CliError>

where O: BatchOperation,

Process multiple files with the given operation.

Automatically selects parallel or serial processing based on configuration and file count. Provides progress reporting and collects all results.

Arguments

• files - Slice of file paths to process
• operation - The operation to perform on each file
• show_progress - Whether to show progress updates

Returns

• Ok(BatchResults) - Successfully processed all files (individual failures collected in results)
• Err(CliError::ThreadPoolError) - Failed to create thread pool with requested configuration

Thread Pool Selection

The method uses different thread pool strategies based on configuration:

1. Serial Processing: If files.len() < parallel_threshold, processes serially (no thread pool)
2. Local Thread Pool: If max_threads is Some(n), creates isolated pool with n threads
3. Global Thread Pool: If max_threads is None, uses Rayon’s global pool

Error Handling

Thread pool creation can fail if:

• max_threads is 0 (invalid configuration)
• System cannot allocate thread resources
• Thread stack allocation fails

Individual file processing errors are collected in BatchResults, not returned as errors.

Performance

• Serial processing for small batches to avoid thread pool overhead
• Local thread pool: ~0.5-1ms creation overhead, ~2-8MB per thread
• Global thread pool: zero overhead
• Lock-free progress tracking using atomic counters

Examples

use hedl_cli::batch::{BatchExecutor, BatchConfig, FormatOperation};
use hedl_cli::error::CliError;
use std::path::PathBuf;

let processor = BatchExecutor::new(BatchConfig {
    max_threads: Some(4),
    ..Default::default()
});

let files = vec![PathBuf::from("a.hedl"), PathBuf::from("b.hedl")];

match processor.process(
    &files,
    FormatOperation {
        check: false,
        ditto: true,
        with_counts: false,
    },
    true,
) {
    Ok(results) => {
        println!("Formatted {} files", results.success_count());
        if results.has_failures() {
            // Handle individual file failures
        }
    }
    Err(CliError::ThreadPoolError { message, requested_threads }) => {
        eprintln!("Failed to create thread pool: {}", message);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}

pub fn process_streaming<O>( &self, files: &[PathBuf], operation: O, show_progress: bool, ) -> Result<BatchResults<O::Output>, CliError>

where O: StreamingBatchOperation,

Process files using streaming operations for memory efficiency.

This method uses the streaming parser from hedl-stream to process files with constant memory usage regardless of file size. Ideal for:

• Files larger than 100MB
• Memory-constrained environments
• Processing thousands of files

Arguments

• files - Slice of file paths to process
• operation - The streaming operation to perform
• show_progress - Whether to show progress updates

Returns

• Ok(BatchResults) - Always succeeds and collects all individual results
• Err(CliError) - Only on catastrophic failures

Memory Usage

Peak memory = buffer_size (8KB) × num_threads + ID tracking set

Examples

use hedl_cli::batch::{BatchExecutor, StreamingValidationOperation, BatchConfig};
use std::path::PathBuf;

let processor = BatchExecutor::default_config();
let files = vec![PathBuf::from("large-file.hedl")];
let operation = StreamingValidationOperation { strict: false };

let results = processor.process_streaming(&files, operation, true)?;
println!("Processed {} files with constant memory", results.success_count());

pub fn process_auto<O, SO>( &self, files: &[PathBuf], standard_op: O, streaming_op: SO, show_progress: bool, ) -> Result<BatchResults<O::Output>, CliError>

where O: BatchOperation<Output = SO::Output>, SO: StreamingBatchOperation,

Automatically choose between standard and streaming based on file size.

Files larger than 100MB use streaming mode for memory efficiency, while smaller files use standard mode for better performance.

Arguments

• files - Slice of file paths to process
• standard_op - Standard operation for small files
• streaming_op - Streaming operation for large files
• show_progress - Whether to show progress updates

Returns

• Ok(BatchResults) - Combined results from both modes
• Err(CliError) - On catastrophic failures

Examples

use hedl_cli::batch::{BatchExecutor, ValidationOperation, StreamingValidationOperation};
use std::path::PathBuf;

let processor = BatchExecutor::default_config();
let files = vec![
    PathBuf::from("small.hedl"),
    PathBuf::from("large-200mb.hedl"),
];

let results = processor.process_auto(
    &files,
    ValidationOperation { strict: false },
    StreamingValidationOperation { strict: false },
    true,
)?;
println!("Processed {} files", results.results.len());

Trait Implementations

impl Clone for BatchExecutor

fn clone(&self) -> BatchExecutor

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BatchExecutor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.