brainwires-datasets

Training data pipelines for the Brainwires Agent Framework — JSONL I/O, tokenization, deduplication, format conversion.

Overview

brainwires-datasets handles every step between raw training data and model-ready datasets. It reads and writes JSONL files, converts between popular fine-tuning formats (OpenAI, Together, Alpaca, ShareGPT, ChatML), validates data quality, deduplicates examples, tokenizes text, and splits into train/eval sets.

Design principles:

• Format-agnostic — a single TrainingExample type normalizes all formats; convert freely between them
• Streaming I/O — JsonlReader/JsonlWriter stream line-by-line to handle datasets larger than memory
• Quality-first — DataValidator catches missing fields, empty messages, and role violations before training
• Pluggable tokenizers — HuggingFace Tokenizers and tiktoken via feature flags
• Deduplication — exact hash-based dedup to remove repeated examples

  ┌──────────────────────────────────────────────────────────────┐
  │                     brainwires-datasets                       │
  │                                                              │
  │  ┌──────┐    ┌───────────┐    ┌──────────┐    ┌──────────┐  │
  │  │ JSONL │───▶│  Format   │───▶│ Dataset  │───▶│ Quality  │  │
  │  │ Reader│    │ Converter │    │ Instruct │    │ Validator│  │
  │  │ Writer│    │ OpenAI    │    │Preference│    │ Dedup    │  │
  │  └──────┘    │ Together  │    └──────────┘    └────┬─────┘  │
  │              │ Alpaca    │                         │        │
  │              │ ShareGPT  │                         ▼        │
  │              │ ChatML    │                  ┌───────────┐   │
  │              └───────────┘                  │ Tokenizer │   │
  │                                             │ HF / Tik  │   │
  │                                             └─────┬─────┘   │
  │                                                   │         │
  │                                             ┌─────▼─────┐   │
  │                                             │Train/Eval │   │
  │                                             │  Split     │   │
  │                                             └───────────┘   │
  └──────────────────────────────────────────────────────────────┘

  Flow: JSONL → Format Converter → Dataset → Quality/Dedup → Tokenizer → Split

Quick Start

Add to your Cargo.toml:

[dependencies]
brainwires-datasets = "0.6"

Load a JSONL dataset and validate it:

use brainwires_datasets::{
    JsonlReader, DataValidator, ValidatorConfig,
    InstructDataset, TrainingExample, compute_stats,
};

// Read training examples from JSONL
let examples: Vec<TrainingExample> = JsonlReader::read("data/train.jsonl")?;

// Validate data quality
let validator = DataValidator::new(ValidatorConfig::default());
let report = validator.validate(&examples)?;
println!("Issues: {}", report.issues.len());

// Compute statistics
let stats = compute_stats(&examples);
println!("Examples: {}, avg tokens: {:.0}", stats.total_examples, stats.avg_tokens);

// Build a dataset
let dataset = InstructDataset::from_examples(examples);

Features

# With tiktoken for OpenAI token counting
[dependencies]
brainwires-datasets = { version = "0.6", features = ["tiktoken"] }

# Full feature set
[dependencies]
brainwires-datasets = { version = "0.6", features = ["full"] }

# Minimal — no tokenizer, just I/O and format conversion
[dependencies]
brainwires-datasets = { version = "0.6", default-features = false }

Architecture

Core Types

JSONL I/O

Dataset Abstractions

Format Converters

All converters implement the FormatConverter trait with to_format() and from_format() methods.

Quality Tools

Tokenizers

Sampling

Usage Examples

Format Conversion

use brainwires_datasets::{
    read_jsonl, write_jsonl,
    OpenAiFormat, AlpacaFormat, FormatConverter,
    TrainingExample,
};

// Read Alpaca-format data
let examples = AlpacaFormat::from_file("data/alpaca.jsonl")?;

// Convert to OpenAI format
let openai_lines: Vec<String> = examples
    .iter()
    .map(|ex| OpenAiFormat::to_string(ex))
    .collect::<Result<Vec<_>>>()?;

write_jsonl(&openai_lines, "data/openai-format.jsonl")?;

Token Counting

use brainwires_datasets::{Tokenizer, TrainingExample};

#[cfg(feature = "hf-tokenizer")]
{
    use brainwires_datasets::HfTokenizer;
    let tokenizer = HfTokenizer::from_pretrained("bert-base-uncased")?;
    let count = tokenizer.count_tokens("Hello, world!")?;
    println!("Tokens: {count}");
}

#[cfg(feature = "tiktoken")]
{
    use brainwires_datasets::TiktokenTokenizer;
    let tokenizer = TiktokenTokenizer::new("cl100k_base")?;
    let count = tokenizer.count_tokens("Hello, world!")?;
    println!("Tokens: {count}");
}

Deduplication

#[cfg(feature = "dedup")]
{
    use brainwires_datasets::{Deduplicator, exact_dedup, TrainingExample};

    let examples: Vec<TrainingExample> = load_examples()?;
    let deduped = exact_dedup(&examples);
    println!("Removed {} duplicates", examples.len() - deduped.len());
}

Train/Eval Split

use brainwires_datasets::{train_eval_split, SplitConfig};

let config = SplitConfig {
    eval_ratio: 0.1,
    shuffle: true,
    seed: Some(42),
};
let split = train_eval_split(&examples, config)?;
println!("Train: {}, Eval: {}", split.train.len(), split.eval.len());

Preference Datasets

use brainwires_datasets::{PreferencePair, PreferenceDataset};

let pairs = vec![
    PreferencePair {
        prompt: "Explain recursion".into(),
        chosen: "Recursion is when a function calls itself...".into(),
        rejected: "It's a loop thing".into(),
        ..Default::default()
    },
];

let dataset = PreferenceDataset::from_pairs(pairs);

Data Validation

use brainwires_datasets::{DataValidator, ValidatorConfig, IssueSeverity};

let validator = DataValidator::new(ValidatorConfig {
    max_tokens: Some(4096),
    require_system_message: false,
    ..Default::default()
});

let report = validator.validate(&examples)?;
for issue in &report.issues {
    if issue.severity == IssueSeverity::Error {
        eprintln!("Error in example {}: {}", issue.example_index, issue.message);
    }
}

Integration with Brainwires

Use via the brainwires facade crate:

[dependencies]
brainwires = { version = "0.6", features = ["datasets"] }

Or depend on brainwires-datasets directly for standalone dataset tooling without the rest of the framework.

The brainwires-training crate consumes brainwires-datasets types directly — datasets flow seamlessly into both cloud and local training pipelines.

References

Papers

• FED: GPU-Accelerated Deduplication Framework (Jan 2025) — high-throughput dedup strategies
• LSHBloom: Internet-Scale Deduplication (Nov 2024) — locality-sensitive hashing for massive datasets
• Linguistic Laws & Subword Tokenization (Nov 2024) — analysis of tokenizer behavior
• DPO: Direct Preference Optimization (2023) — the preference pair format consumed by PreferenceDataset
• ORPO: Monolithic Preference Optimization (2024) — single-stage alignment data format
• SLM-Bench: Small Language Model Benchmark (EMNLP 2025) — evaluation datasets for small models

Technical Blogs & Guides

• Modern Tokenization Techniques — CodeSignal — BPE, WordPiece, and SentencePiece
• Tokenization Deep Dive — Let's Data Science — why tokenization matters
• Diffusion Curriculum (DisCL) — ICCV 2025 — curriculum learning strategies (cf. curriculum_order)
• Synthetic Data for ML 2025 — generating training data

Data Tools

• Duplodocus — Allen AI — large-scale deduplication
• fastdedup — fast exact dedup
• DataTrove — HuggingFace — data processing pipelines

License

Licensed under the MIT License. See LICENSE for details.