Enum BufferSizeHint

Source

pub enum BufferSizeHint {
    Small,
    Medium,
    Large,
    Huge,
}

Expand description

Buffer size hints for different workload profiles.

These size classes provide pre-configured buffer sizes optimized for common use cases, from embedded systems to high-throughput data processing.

§Size Classes

Small (8KB): Embedded systems, small config files, memory-constrained environments
Medium (64KB): Default for general use, good balance of performance and memory
Large (256KB): Large files, high-throughput scenarios, server workloads
Huge (1MB): Multi-GB files, maximum performance, minimal syscall overhead

§Performance Characteristics

Larger buffers reduce system call overhead but use more memory. The optimal size depends on:

File size (larger files benefit from larger buffers)
Available memory (constrained systems need smaller buffers)
I/O characteristics (fast storage benefits more from large buffers)
Access pattern (sequential vs. random)

§Examples

§Automatic Selection

use hedl_stream::{StreamingParserConfig, BufferSizeHint};

let config = StreamingParserConfig::default()
    .with_buffer_hint(BufferSizeHint::Large);

assert_eq!(config.buffer_size, 256 * 1024);

§Custom Configuration

use hedl_stream::{StreamingParserConfig, BufferSizeHint};

// Small embedded device
let embedded_config = StreamingParserConfig::default()
    .with_buffer_hint(BufferSizeHint::Small);

// High-throughput server
let server_config = StreamingParserConfig::default()
    .with_buffer_hint(BufferSizeHint::Huge);

Variants§

§

Small

8KB buffer - for embedded systems and small files.

Use when:

Parsing config files (<1MB)
Running on embedded systems
Memory is very limited (<10MB available)
Processing many small files concurrently

Trade-offs:

Minimal memory footprint
More system calls for large files
Lower throughput on fast storage

§

Medium

64KB buffer - default for general use.

Use when:

General-purpose parsing
No specific performance requirements
Mixed file sizes
Standard development environments

Trade-offs:

Good balance of memory and performance
Suitable for most workloads
May not be optimal for extremes

§

Large

256KB buffer - for large files and high throughput.

Use when:

Parsing large files (>100MB)
High-throughput ETL pipelines
Fast storage (NVMe SSD)
Server environments with available memory

Trade-offs:

Reduced syscall overhead
Better throughput on large files
Higher memory usage per parser

§

Huge

1MB buffer - maximum performance for huge files.

Use when:

Parsing multi-GB files
Maximum throughput required
Abundant memory available
Single-threaded processing

Trade-offs:

Minimal syscall overhead
Maximum throughput
Significant memory per parser (limits concurrency)

Implementations§

Source §

impl BufferSizeHint

Source

pub const fn size(self) -> usize

Get the buffer size in bytes for this hint.

§Examples

use hedl_stream::BufferSizeHint;

assert_eq!(BufferSizeHint::Small.size(), 8 * 1024);
assert_eq!(BufferSizeHint::Medium.size(), 64 * 1024);
assert_eq!(BufferSizeHint::Large.size(), 256 * 1024);
assert_eq!(BufferSizeHint::Huge.size(), 1024 * 1024);

Source

pub fn for_file_size(size_bytes: u64) -> Self

Get a buffer size hint based on file size.

Automatically selects an appropriate buffer size based on the total size of the file being parsed.

§Heuristics

Files <1MB: Small (8KB)
Files 1-100MB: Medium (64KB)
Files 100MB-1GB: Large (256KB)
Files >1GB: Huge (1MB)

§Examples

use hedl_stream::BufferSizeHint;

let hint = BufferSizeHint::for_file_size(500 * 1024); // 500KB
assert_eq!(hint, BufferSizeHint::Small);

let hint = BufferSizeHint::for_file_size(50 * 1024 * 1024); // 50MB
assert_eq!(hint, BufferSizeHint::Medium);

let hint = BufferSizeHint::for_file_size(500 * 1024 * 1024); // 500MB
assert_eq!(hint, BufferSizeHint::Large);

let hint = BufferSizeHint::for_file_size(2 * 1024 * 1024 * 1024); // 2GB
assert_eq!(hint, BufferSizeHint::Huge);

Source

pub fn for_memory_budget( available_memory: usize, concurrent_parsers: usize, ) -> Self

Get a buffer size hint for memory-constrained environments.

Recommends a buffer size that won’t exceed the given memory budget when running concurrent_parsers simultaneously.

§Examples

use hedl_stream::BufferSizeHint;

// 10MB available, running 10 parsers concurrently
let hint = BufferSizeHint::for_memory_budget(10 * 1024 * 1024, 10);
// Should suggest Small (8KB) since 10 * 64KB = 640KB is reasonable