rullama-datasets 0.12.0

Training data pipelines for rullama: JSONL I/O, tokenization, dedup, format conversion (Alpaca / ChatML / OpenAI / ShareGPT / Together), and quality stats. Shared by cloud (`rullama-finetune`) and local (`rullama-lora`, in the rullama workspace) fine-tuning.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

/// Dataset statistics computation.
pub mod stats;
/// Dataset validation rules and reporting.
pub mod validator;

/// MinHash-based and exact deduplication.
#[cfg(feature = "datasets-dedup")]
pub mod dedup;

pub use stats::{
    DatasetStats, HistogramBucket, PreferenceStats, RoleCounts, compute_preference_stats,
    compute_stats,
};
pub use validator::{
    DataValidator, IssueSeverity, ValidationIssue, ValidationReport, ValidatorConfig,
};

#[cfg(feature = "datasets-dedup")]
pub use dedup::{Deduplicator, exact_dedup, exact_dedup_preferences};