rullama-datasets 0.12.0

Training data pipelines for rullama: JSONL I/O, tokenization, dedup, format conversion (Alpaca / ChatML / OpenAI / ShareGPT / Together), and quality stats. Shared by cloud (`rullama-finetune`) and local (`rullama-lora`, in the rullama workspace) fine-tuning.
Documentation 
1
2
3
4
5
6
7

/// Streaming JSONL reader.
pub mod reader;
/// Buffered JSONL writer.
pub mod writer;

pub use reader::{JsonlReader, read_jsonl, read_jsonl_preferences};
pub use writer::{JsonlWriter, write_jsonl, write_jsonl_preferences};