Skip to main content

stream_binary_corpus

bbpe::corpus

Function stream_binary_corpus 

Source 
pub fn stream_binary_corpus<P: AsRef<Path>>(
    inputs: &[P],
    cfg: &IngestConfig,
) -> Result<BinaryChunkStream>
Expand description

Streams binary corpus chunks from disk, avoiding materialising the entire corpus at once.

The returned iterator yields Vec<u8> buffers according to IngestConfig::chunk_size, skipping empty results. File discovery honours the same semantics as load_binary_corpus. Metadata about the total number of chunks and bytes is precomputed from filesystem information so callers can set up progress reporting without forcing eager loading.