Module worker_count

Expand description

§Worker Count Value Object - Parallel Processing Optimization Infrastructure

This module provides a comprehensive worker count value object that implements adaptive parallel processing optimization, resource-aware worker allocation, and performance-optimized concurrency management for the adaptive pipeline system’s parallel processing infrastructure.

§Overview

The worker count system provides:

Adaptive Parallel Processing: Dynamic worker allocation based on file characteristics
Resource-Aware Optimization: System resource consideration for optimal performance
Performance-Optimized Concurrency: Empirically-validated worker count strategies
Cross-Platform Compatibility: Consistent representation across languages and systems
Serialization: Comprehensive serialization across storage backends and APIs
Validation: Comprehensive worker count validation and constraint enforcement

§Key Features

§1. Adaptive Parallel Processing

Dynamic worker allocation with comprehensive optimization:

File Size Optimization: Empirically-validated strategies for different file sizes
System Resource Awareness: CPU core consideration for optimal performance
Processing Type Adaptation: CPU-intensive vs I/O-intensive processing optimization
Performance Validation: Benchmark-driven optimization strategies

§2. Resource-Aware Optimization

System resource consideration for optimal performance:

CPU Core Management: Optimal worker allocation based on available cores
Memory Consideration: Worker count limits to prevent resource exhaustion
Oversubscription Control: Balanced oversubscription for optimal throughput
Resource Validation: System capability validation and constraint enforcement

§3. Cross-Platform Compatibility

Consistent worker count management across platforms:

JSON Serialization: Standard JSON representation
Database Storage: Optimized database storage patterns
API Integration: RESTful API compatibility
Multi-Language: Consistent interface across languages

§Usage Examples

§Basic Worker Count Creation and Optimization

§System Resource-Aware Optimization

§Worker Count Validation and Suitability

§Performance Strategy and Description

§Conversion and Integration

§Worker Count Features

§Adaptive Optimization Strategies

Worker count optimization based on empirical benchmark results:

Tiny Files (< 1MB): 1-2 workers (minimize overhead)
Small Files (1-50MB): 6-14 workers (aggressive parallelism, +102% performance gain)
Medium Files (50-500MB): 5-12 workers (balanced approach)
Large Files (500MB-2GB): 8-12 workers (moderate parallelism)
Huge Files (> 2GB): 3-6 workers (conservative, +76% performance gain)

§Resource-Aware Features

CPU Core Consideration: Optimal worker allocation based on available cores
Memory Management: Worker count limits to prevent resource exhaustion
Oversubscription Control: Balanced oversubscription for optimal throughput
System Validation: Comprehensive system capability validation

§Performance Optimization

Empirical Validation: Benchmark-driven optimization strategies
Processing Type Adaptation: CPU-intensive vs I/O-intensive optimization
Throughput Balancing: Optimal balance between throughput and coordination overhead
Suitability Checking: Validation of worker count suitability for specific workloads

§Performance Characteristics

Creation Time: ~1μs for new worker count creation with bounds checking
Optimization Time: ~5μs for file size-based optimization calculation
Validation Time: ~3μs for comprehensive user input validation
Suitability Check: ~2μs for worker count suitability validation
Memory Usage: ~8 bytes for worker count storage (single usize)
Thread Safety: Immutable access patterns are thread-safe

§Cross-Platform Compatibility

Rust: WorkerCount newtype wrapper with full optimization
Go: WorkerCount struct with equivalent interface
JSON: Numeric representation for API compatibility
Database: INTEGER column with validation constraints

Structs§

WorkerCount: Worker count value object for adaptive parallel processing optimization