Struct StreamingFileProcessor 

pub struct StreamingFileProcessor<T: FileIOService> { /* private fields */ }

Implementation of FileProcessorService

This struct provides a high-performance implementation of the file processor service interface, handling file chunking, parallel processing, and result aggregation with comprehensive error handling and statistics collection.

Key Features

• Parallel Processing: Concurrent processing of file chunks
• Adaptive Chunking: Dynamic chunk size optimization
• Progress Tracking: Real-time progress monitoring
• Error Resilience: Comprehensive error handling and recovery
• Statistics Collection: Detailed processing metrics

Architecture

The service is built around several core components:

• File I/O Service: Handles efficient file reading and writing
• Chunk Processors: Pluggable processing logic for file chunks
• Configuration Management: Runtime configuration with thread-safe updates
• Statistics Tracking: Comprehensive processing statistics

Examples

Implementations

impl<T: FileIOService> StreamingFileProcessor<T>

pub fn new(file_io_service: Arc<T>, config: FileProcessorConfig) -> Self

Creates a new FileProcessorService instance

pub fn new_default(file_io_service: Arc<T>) -> Self

Creates a new FileProcessorService with default configuration

Trait Implementations

impl<T: FileIOService> FileProcessorService for StreamingFileProcessor<T>

fn process_file<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, input_path: &'life1 Path, output_path: Option<&'life2 Path>, processor: Box<dyn ChunkProcessor>, ) -> Pin<Box<dyn Future<Output = Result<FileProcessingResult, PipelineError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Processes a single file through the pipeline

fn process_files_batch<'life0, 'async_trait>( &'life0 self, file_pairs: Vec<(PathBuf, Option<PathBuf>)>, processor: Box<dyn ChunkProcessor>, ) -> Pin<Box<dyn Future<Output = Result<Vec<FileProcessingResult>, PipelineError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Processes multiple files concurrently

fn process_file_in_place<'life0, 'life1, 'async_trait>( &'life0 self, file_path: &'life1 Path, processor: Box<dyn ChunkProcessor>, ) -> Pin<Box<dyn Future<Output = Result<FileProcessingResult, PipelineError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Processes a file in-place (modifying the original)

fn validate_file_before_processing<'life0, 'life1, 'async_trait>( &'life0 self, file_path: &'life1 Path, ) -> Pin<Box<dyn Future<Output = Result<bool, PipelineError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Validates file integrity before processing

fn get_processing_stats(&self) -> FileProcessingStats

Gets processing statistics

fn reset_processing_stats(&mut self)

Resets processing statistics

fn get_config(&self) -> FileProcessorConfig

Gets the current configuration

fn update_config(&mut self, config: FileProcessorConfig)

Updates the configuration