Struct DatasetLoader 

pub struct DatasetLoader;

Dataset loader that provides lazy iteration over entries

Implementations

impl DatasetLoader

pub fn count_entries<P: AsRef<Path>>(path: P) -> Result<usize, Error>

Count the number of non-empty lines in a JSONL file (fast approximation of entry count)

pub fn from_file<P: AsRef<Path>>( path: P, ) -> Result<DatasetIterator<BufReader<File>>, Error>

Create an iterator from a JSONL file in OpenAI batch API format Returns unparsed entries (without tokenization)

pub fn from_string(data: String) -> DatasetIterator<BufReader<Cursor<String>>>

Create an iterator from a string (useful for testing or WASM) Returns unparsed entries (without tokenization)