Module chunk_size

adaptive_pipeline_domain::value_objects

Module chunk_size 

Source
Expand description

§Chunk Size Value Object

This module provides a type-safe representation of chunk sizes used throughout the adaptive pipeline system. It ensures chunk sizes are within valid bounds and provides convenient methods for working with chunk sizes.

§Overview

The chunk size value object provides:

  • Validation: Ensures chunk sizes are within acceptable bounds
  • Type Safety: Prevents invalid chunk sizes at compile time
  • Convenience Methods: Easy creation and manipulation of chunk sizes
  • Serialization: JSON and binary serialization support
  • Performance: Optimized for frequent use in processing pipelines

§Design Principles

The chunk size follows Domain-Driven Design value object principles:

  • Immutability: Once created, chunk sizes cannot be modified
  • Validation: All chunk sizes are validated at creation time
  • Equality: Two chunk sizes are equal if they have the same byte count
  • Value Semantics: Chunk sizes are compared by value, not identity

§Chunk Size Constraints

§Minimum Size (1 byte)

  • Purpose: Ensures chunks contain at least some data
  • Rationale: Zero-byte chunks would be meaningless in processing
  • Impact: Prevents degenerate cases in processing algorithms

§Maximum Size (512MB)

  • Purpose: Prevents memory exhaustion and performance issues
  • Rationale: Very large chunks can cause memory pressure
  • Impact: Ensures predictable memory usage patterns

§Default Size (1MB)

  • Purpose: Provides a balanced default for most use cases
  • Rationale: Good balance between memory usage and processing efficiency
  • Impact: Optimal performance for typical file processing scenarios

§Usage Examples

§Basic Chunk Size Creation

use adaptive_pipeline_domain::value_objects::ChunkSize;

// Create from bytes
let chunk = ChunkSize::new(1024 * 1024).unwrap(); // 1MB
assert_eq!(chunk.bytes(), 1024 * 1024);

// Create from kilobytes
let chunk_kb = ChunkSize::from_kb(512).unwrap(); // 512KB
assert_eq!(chunk_kb.bytes(), 512 * 1024);

// Create from megabytes
let chunk_mb = ChunkSize::from_mb(16).unwrap(); // 16MB
assert_eq!(chunk_mb.megabytes(), 16.0);

// Use default (1MB)
let default_chunk = ChunkSize::default();
assert_eq!(default_chunk.bytes(), 1024 * 1024);

§Chunk Size Validation

use adaptive_pipeline_domain::value_objects::ChunkSize;

// Valid chunk sizes
let valid = ChunkSize::new(64 * 1024).unwrap(); // 64KB - valid
assert_eq!(valid.bytes(), 64 * 1024);

// Invalid: too small
let too_small = ChunkSize::new(0); // Must be at least 1 byte
assert!(too_small.is_err());

// Invalid: too large
let too_large = ChunkSize::new(600 * 1024 * 1024); // Max is 512MB
assert!(too_large.is_err());

// Optimal sizing for file
let optimal = ChunkSize::optimal_for_file_size(100 * 1024 * 1024); // 100MB file
assert!(optimal.bytes() >= ChunkSize::MIN_SIZE);
assert!(optimal.bytes() <= ChunkSize::MAX_SIZE);

§Chunk Size Arithmetic

use adaptive_pipeline_domain::value_objects::ChunkSize;

let chunk = ChunkSize::from_mb(2).unwrap(); // 2MB chunk

// Calculate chunks needed for a file
let file_size = 10 * 1024 * 1024; // 10MB file
let chunks_needed = chunk.chunks_needed_for_file(file_size);
assert_eq!(chunks_needed, 5); // 10MB / 2MB = 5 chunks

// Check if optimal for file size
let is_optimal = chunk.is_optimal_for_file(file_size);
println!("Chunk is optimal: {}", is_optimal);

// Display formatting
assert_eq!(format!("{}", chunk), "2.0MB");

§Performance Considerations

§Memory Usage

  • Small Chunks: Lower memory usage but higher processing overhead
  • Large Chunks: Higher memory usage but lower processing overhead
  • Optimal Range: 64KB to 4MB for most applications

§Processing Efficiency

  • I/O Operations: Larger chunks reduce I/O overhead
  • CPU Processing: Moderate chunks balance CPU cache efficiency
  • Parallelism: Smaller chunks enable better parallel processing

§Adaptive Sizing

The chunk size can be dynamically adjusted based on:

  • File Size: Larger files may benefit from larger chunks
  • Available Memory: Adjust chunk size based on system resources
  • Processing Type: Different algorithms may prefer different chunk sizes
  • Network Conditions: Streaming scenarios may require smaller chunks

§Integration

The chunk size value object integrates with:

  • File Processing: Determines how files are divided for processing
  • Memory Management: Influences memory allocation patterns
  • Performance Tuning: Enables performance optimization strategies
  • Configuration: Allows runtime configuration of chunk sizes

§Thread Safety

The chunk size value object is fully thread-safe:

  • Immutable: Once created, chunk sizes cannot be modified
  • Copy Semantics: Cheap to copy and pass between threads
  • No Shared State: No mutable shared state to synchronize

§Future Enhancements

Planned enhancements include:

  • Adaptive Sizing: Automatic chunk size optimization
  • Profile-Based Sizing: Chunk size profiles for different use cases
  • Dynamic Adjustment: Runtime chunk size adjustment based on performance
  • Compression-Aware Sizing: Chunk sizes optimized for compression algorithms

Structs§

ChunkSize
Value object representing a chunk size with validation