rskit-dataset — Streaming Dataset Collection Framework

Streaming dataset collection framework: source, transform, target, schema validation, and collector orchestration.

Features

• ETL pipeline: streaming Source → fallible Transform → Target orchestrated by Collector
• DataItem with checked in-memory payload construction for small samples and file-backed streaming payloads for large samples
• Parallel fetching with configurable concurrency
• Manifest — incremental build cache for resumable collection
• DatasetLimits — configurable memory threshold and bounded stream buffers
• Schema validation delegated to rskit-schema
• Stream adapters for rskit-pipeline
• Streaming DatasetRecord readers/writers with JSON Lines and CSV support plus filter/column-selection operators
• Bounded JSON record size, nesting depth, field count, array length, and string length for untrusted records
• Progress callback support via ProgressCallback

Usage

[dependencies]
rskit-dataset = "0.1.0-alpha.1"

use rskit_dataset::{CollectorConfig, DataItem, DatasetLimits, Label, MediaType};
use std::path::PathBuf;

let config = CollectorConfig {
    output_dir: PathBuf::from("dataset_out"),
    concurrency: 4,
    source_timeout_secs: 300.0,
    force: false,
    limits: DatasetLimits::default(),
};

let item = DataItem::new_bytes(vec![1, 2, 3], Label::Real, MediaType::Image, "local")?
    .with_extension(".bin");

// Implement Source, Transform, and Target traits, then construct Collector::new(...)

See Also

Main repository README