Struct StreamConfig 

pub struct StreamConfig {
    pub buffer_size: usize,
    pub max_object_bytes: Option<usize>,
    pub from_json: FromJsonConfig,
    pub use_size_estimation: bool,
    pub true_streaming: bool,
}

Configuration for streaming JSON parsing

Controls memory limits and parsing behavior for streaming operations.

Memory Safety

Streaming parsers process data incrementally to avoid loading entire files into memory. However, individual objects can still be large. Configure max_object_bytes to limit memory per object.

Examples

use hedl_json::streaming::StreamConfig;
use hedl_json::FromJsonConfig;

// Default configuration - suitable for trusted input
let config = StreamConfig::default();

// Conservative configuration for untrusted input
let strict = StreamConfig {
    buffer_size: 8 * 1024,           // 8 KB buffer
    max_object_bytes: 1024 * 1024,   // 1 MB per object
    from_json: FromJsonConfig::builder()
        .max_depth(100)
        .max_array_size(10_000)
        .build(),
};

// High-throughput configuration for large ML datasets
let ml_config = StreamConfig {
    buffer_size: 256 * 1024,         // 256 KB buffer
    max_object_bytes: 100 * 1024 * 1024, // 100 MB per object
    from_json: FromJsonConfig::default(),
};

Fields

buffer_size: usize

Size of internal read buffer in bytes (default: 64 KB)

Larger buffers improve throughput for network I/O but use more memory. Smaller buffers reduce memory overhead for many concurrent streams.

max_object_bytes: Option<usize>

Maximum bytes per JSON object (default: 10 MB)

Prevents memory exhaustion from individual oversized objects. Set to None to disable (not recommended for untrusted input).

from_json: FromJsonConfig

Configuration for JSON to HEDL conversion

Controls limits and behavior when converting each parsed JSON object to a HEDL document.

use_size_estimation: bool

Enable efficient size estimation instead of serialization for size checks. Default: true

true_streaming: bool

Enable true streaming for JSON arrays (constant memory usage). When true, uses incremental parsing instead of loading entire array. Default: true

Implementations

impl StreamConfig

pub fn large_file() -> Self

Configuration optimized for large files (GB+)

Uses larger buffers and object limits while maintaining constant memory.

pub fn low_memory() -> Self

Configuration for memory-constrained environments

Minimizes memory usage at the cost of some throughput.

impl StreamConfig

pub fn builder() -> StreamConfigBuilder

Create a new builder for configuring stream parsing

Examples

use hedl_json::streaming::StreamConfig;

let config = StreamConfig::builder()
    .buffer_size(128 * 1024)
    .max_object_bytes(50 * 1024 * 1024)
    .build();

Trait Implementations

impl Clone for StreamConfig

fn clone(&self) -> StreamConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StreamConfig

fn default() -> Self

Returns the “default value” for a type.