Crate rskit_dataset 

Dataset collection framework — streaming sources, transforms, and schema validation on a generic collection engine.

One generic Collector<T> drives collection for every item family: it is generic over any DatasetItem, so DataItem blobs and DatasetRecord rows share the same worker pool, cancellation, and event loop. The engine stays item-agnostic — it never writes items itself. Per-item materialization lives behind an injected ItemSink<T> (LocalBlobSink writes DataItem samples to real/ and ai/), and per-item validation is a pluggable Validator<T> that callers opt into (for example a schema-backed validator for tabular records).

Re-exports

pub use collector::Collector;

pub use collector::CollectorConfig;

pub use collector::CollectorResult;

pub use collector::NullProgress;

pub use collector::ProgressCallback;

pub use manifest::CacheStatus;

pub use manifest::Manifest;

pub use manifest::SourceEntry;

pub use manifest::SourceStats;

pub use record::BoxRecordStream;

pub use record::CsvReader;

pub use record::CsvWriter;

pub use record::DatasetFormat;

pub use record::DatasetReader;

pub use record::DatasetRecord;

pub use record::DatasetWriter;

pub use record::JsonArrayReader;

pub use record::JsonArrayWriter;

pub use record::JsonLinesReader;

pub use record::JsonLinesWriter;

pub use record::RecordSink;

pub use record::RecordSource;

pub use record::SchemaValidator;

pub use record::filter_records;

pub use record::select_columns;

pub use schema::DatasetSchema;

pub use schema::validate_record;

pub use source::BoxDataStream;

pub use source::BoxItemStream;

pub use source::Source;

pub use stream::DatasetStreamExt;

pub use target::PublishResult;

pub use target::Target;

pub use transform::ResizeTransform;

pub use transform::Transform;

Modules

collector

Collector — orchestrates source → transform → target pipelines.

manifest

.manifest.json cache logic — skip re-downloading completed sources.

record

Streaming row/record dataset abstractions.

schema

Schema validation delegated to rskit-schema.

source

Source trait — pull data from any origin.

stream

Stream adapters for composing dataset items with rskit-stream.

target

Target trait — publish collected data to a destination.

transform

Transform trait and built-in transforms.

Structs

DataItem

A single data sample flowing through the dataset pipeline.

DataPayload

Payload for a single dataset item.

DatasetLimits

Runtime limits for dataset streaming and bounded materialization.

LocalBlobSink

Local filesystem sink for DataItem samples.

Enums

Label

Binary classification label.

MediaType

Supported media types.

Constants

DEFAULT_MAX_IN_MEMORY_BYTES

Default threshold above which payloads should be represented as files.

Traits

DatasetItem

Capability an item must provide to flow through the generic collection engine.

ItemSink

Destination that materializes each collected item as the engine streams it.

Validator

Rejects invalid items before they are materialized by a sink.