Struct ChunkSize 

pub struct ChunkSize { /* private fields */ }

Value object representing a chunk size with validation

This struct provides a type-safe representation of chunk sizes used throughout the adaptive pipeline system. It ensures chunk sizes are within valid bounds and provides convenient methods for working with chunk sizes.

Key Features

• Validation: Ensures chunk sizes are within acceptable bounds (1 byte to 512MB)
• Type Safety: Prevents invalid chunk sizes at compile time
• Immutability: Once created, chunk sizes cannot be modified
• Serialization: Full JSON and binary serialization support
• Performance: Optimized for frequent use in processing pipelines

Constraints

• Minimum Size: 1 byte (prevents degenerate cases)
• Maximum Size: 512MB (prevents memory exhaustion)
• Default Size: 1MB (balanced for most use cases)

Examples

Implementations

impl ChunkSize

pub const MIN_SIZE: usize = 1usize

Minimum chunk size (1 byte) - must be at least 1 byte for processing

pub const MAX_SIZE: usize = 536_870_912usize

Maximum chunk size (512MB) - prevents memory exhaustion

pub const DEFAULT_SIZE: usize = 1_048_576usize

Default chunk size (1MB)

pub fn new(bytes: usize) -> Result<Self, PipelineError>

Creates a new chunk size with validation

Validates that the specified size is within acceptable bounds before creating the chunk size instance.

Arguments

• bytes - Size in bytes (must be between 1 byte and 512MB)

Returns

• Ok(ChunkSize) - Valid chunk size
• Err(PipelineError::InvalidConfiguration) - If size is out of bounds

Errors

Returns an error if:

• Size is less than MIN_SIZE (1 byte)
• Size exceeds MAX_SIZE (512MB)

Examples

pub fn from_kb(kb: usize) -> Result<Self, PipelineError>

Creates a chunk size from kilobytes

Convenience method for creating chunk sizes in KB units.

Arguments

• kb - Size in kilobytes

Returns

• Ok(ChunkSize) - Valid chunk size
• Err(PipelineError) - If resulting size is out of bounds

Examples

pub fn from_mb(mb: usize) -> Result<Self, PipelineError>

Creates a chunk size from megabytes

Convenience method for creating chunk sizes in MB units.

Arguments

• mb - Size in megabytes

Returns

• Ok(ChunkSize) - Valid chunk size
• Err(PipelineError) - If resulting size is out of bounds

Examples

pub fn bytes(&self) -> usize

Gets the chunk size in bytes

Returns

The size in bytes as a usize

pub fn as_bytes(&self) -> usize

Gets the size in bytes (alias for test framework compatibility)

pub fn kilobytes(&self) -> f64

Gets the size in kilobytes

pub fn megabytes(&self) -> f64

Gets the size in megabytes

pub fn optimal_for_file_size(file_size: u64) -> Self

Calculates the optimal chunk size based on file size

This method implements an empirically-optimized strategy based on comprehensive benchmark results across file sizes from 5MB to 2GB.

Empirical Optimization Results

• 100MB files: 16MB chunks optimal (vs 2MB adaptive = +43.7% performance)
• 500MB files: 16MB chunks optimal (vs 4MB adaptive = +56.2% performance)
• 2GB files: 128MB chunks optimal (current algorithm validated)
• Small files: Current algorithm performing reasonably well

pub fn chunks_needed_for_file(&self, file_size: u64) -> u64

Calculates the number of chunks needed for a given file size

pub fn is_optimal_for_file(&self, file_size: u64) -> bool

Checks if this chunk size is optimal for the given file size

pub fn adjust_for_memory( &self, available_memory: usize, max_parallel_chunks: usize, ) -> Result<Self, PipelineError>

Adjusts the chunk size based on available memory

pub fn validate_user_input( user_chunk_size_mb: usize, file_size: u64, ) -> Result<usize, String>

Validates user-provided chunk size input with sanity checks Returns validated chunk size in bytes or error message

pub fn strategy_description(file_size: u64) -> &'static str

Returns a description of the chunk size strategy for the given file size

Trait Implementations

impl Clone for ChunkSize

fn clone(&self) -> ChunkSize

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkSize

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkSize

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for ChunkSize

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for ChunkSize

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for ChunkSize

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for ChunkSize

fn cmp(&self, other: &ChunkSize) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for ChunkSize

fn eq(&self, other: &ChunkSize) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for ChunkSize

fn partial_cmp(&self, other: &ChunkSize) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Serialize for ChunkSize

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for ChunkSize

impl Eq for ChunkSize

impl StructuralPartialEq for ChunkSize