Struct StreamingLoader

pub struct StreamingLoader {
    pub loaded_count: usize,
    pub duplicate_count: usize,
    pub total_processed: usize,
}

Ultra-fast streaming loader for massive datasets Optimized for loading 100k-1M+ positions efficiently

Fields

loaded_count: usize

duplicate_count: usize

total_processed: usize

Implementations

impl StreamingLoader

pub fn new() -> Self

pub fn stream_load_json<P: AsRef<Path>>( &mut self, path: P, engine: &mut ChessVectorEngine, batch_size: usize, ) -> Result<(), Box<dyn Error>>

Stream-load massive JSON files with minimal memory usage Uses streaming JSON parser and batched processing

pub fn stream_load_binary<P: AsRef<Path>>( &mut self, path: P, engine: &mut ChessVectorEngine, ) -> Result<(), Box<dyn Error>>

Ultra-fast binary format streaming loader For pre-processed binary training data

Trait Implementations

impl Default for StreamingLoader

fn default() -> Self

Returns the “default value” for a type.