Struct ChunkConfig 

pub struct ChunkConfig {
    pub chunk_size: usize,
    pub skip_invalid_lines: bool,
}

Configuration for chunked NDJSON processing.

Controls how NDJSON files are read and processed in chunks to reduce memory usage when handling large files.

Examples

use helios_sof::ChunkConfig;

// Default configuration (1000 resources per chunk)
let config = ChunkConfig::default();

// Custom configuration for memory-constrained environments
let config = ChunkConfig {
    chunk_size: 100,
    skip_invalid_lines: true,
};

Fields

chunk_size: usize

Number of resources to process per chunk. Default: 1000 (approximately 10MB memory usage per chunk)

skip_invalid_lines: bool

If true, skip lines that fail to parse as valid JSON. If false (default), return an error on the first invalid line.

Trait Implementations

impl Clone for ChunkConfig

fn clone(&self) -> ChunkConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChunkConfig

fn default() -> Self

Returns the “default value” for a type.