Skip to main content

Crate lerobot_dataset_io

Crate lerobot_dataset_io 

Source
Expand description

§lerobot-dataset-io

Rust I/O for inspecting, accessing, and creating subsets of packed LeRobot datasets.

The crate owns the LeRobot v3 format boundary. It parses meta/info.json, reads packed episode metadata and per-frame Parquet columns, resolves episode video segments, materializes remote files through a provider-neutral trait, and writes reindexed subset datasets.

For an example of a public robotics-data application where this crate can be used, see Pareto.

§What it provides

  • Local synchronous opens and async provider-backed opens.
  • Ordered feature metadata with unknown fields preserved for round-trip export.
  • RGB, depth-video, and unknown-video classification.
  • Episode metadata and packed video-segment resolution.
  • Projected and streaming reads of per-frame vector columns.
  • Re-encoded or byte-preserving packed-video subset exports.
  • Extension hooks for namespaced metadata and sidecars.
  • Storage injection through DatasetFileProvider; the crate does not depend on a cloud vendor SDK.

The reader accepts LeRobot v3.x datasets. It rejects older major versions and proceeds optimistically for newer v3 minor versions.

§Local dataset example

use lerobot_dataset_io::Dataset;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let dataset = Dataset::open("/datasets/example")?;

    println!(
        "{} episodes at {} fps",
        dataset.episodes().len(),
        dataset.frames_per_second().get()
    );
    for episode in dataset.episodes() {
        println!(
            "episode {}: {} frames, {} video streams",
            episode.episode_index(),
            episode.length_frames(),
            episode.video_segments().len()
        );
    }

    Ok(())
}

§Remote storage

Implement DatasetFileProvider to list dataset-relative files and atomically materialize one file at a requested local path. Construct a DatasetSource with DatasetSource::from_file_provider, then call Dataset::open_source. This supports object stores, HTTP services, content-addressed storage, and application-specific backends without adding them to the format crate.

The default object-store feature provides an adapter for Apache Arrow’s object_store crate, including local filesystem and in-memory stores. Enable aws, gcp, azure, or http for URL construction of the corresponding common backend:

cargo add lerobot-dataset-io --features aws

Then construct a source with DatasetSource::from_object_store_url, supplying the backend options/credentials accepted by object_store::parse_url_opts. The stored source identity excludes URL userinfo, query parameters, and fragments so credentials are not copied into logs or manifests. Prefer backend options or environment-based credential providers over credentials in a URL. The DatasetFileProvider trait remains independently implementable for backends outside that ecosystem.

Remote metadata is materialized eagerly. Packed data and video files are materialized on demand through the async ensure and prefetch methods before synchronous Parquet or media operations.

§Video export boundary

The crate plans video segments but deliberately does not choose or bundle a media framework. Re-encoding exports receive a caller-provided VideoSegmentWriter. Use export_subset_preserving_packed_videos when selected packed video files should be copied byte-for-byte instead.

Every synchronous export requires PreparedExportInputs. For local datasets, construct it with PreparedExportInputs::for_local_dataset; for remote datasets, await Dataset::prepare_export_inputs. The value is bound to the exact dataset, ordered episode selection, and video keys, and is verified before the destination is touched.

§Format extensions

Unknown info.json fields survive subset export. Applications that interpret their own namespaced fields or sidecars can implement DatasetExportExtension and use the extension-aware export functions. Product-specific metadata and cloud authentication belong in adapter crates rather than this crate.

§License

Apache-2.0.

Modules§

format_extension_support
Low-level readers shared with crates that implement LeRobot format extensions.

Structs§

CameraVideoWriteResult
Everything a VideoSegmentWriter reports about one written camera video.
Dataset
A LeRobot v3 dataset opened for reading.
DatasetInfo
Parsed meta/info.json of a LeRobot v3 dataset.
DatasetRelativePath
A portable, non-empty path known to stay beneath a dataset root.
DatasetRelativePathError
A path that is not safe to resolve beneath a dataset root.
Episode
Per-episode metadata read from meta/episodes/chunk-*/file-*.parquet.
EpisodeDataFileReadPlan
A materialized packed data file and every selected episode it contains.
EpisodeDataReadPlan
Complete read plan for a requested episode set.
EpisodeVectorColumn
One requested vector column’s decoded per-frame rows.
EpisodeVectorColumns
One episode’s decoded per-frame vector columns, in the order they were requested.
EpisodeVectorReadReport
Observable work performed by one packed-file reader.
EpisodeVectorReadSchema
Projected source-column availability for a streaming vector read.
EpisodeVectorRow
One full-resolution row yielded by EpisodeVectorRowReader.
EpisodeVectorRowReader
Streaming reader that decodes selected packed-file rows exactly once.
ExportEpisodes
Episode indices selected for an export. The constructor is the boundary where an external list becomes a valid export request.
ExportRangePlacement
Where an exported episode range sits inside a larger canonical dataset; the whole-dataset export is the single-range case.
ExportReport
Summary of a completed export_subset run.
FeatureSpec
One feature entry from the features object of meta/info.json.
FramesPerSecond
A finite, strictly positive dataset frame rate.
InvalidEpisodeDataReadPlan
A packed-file read plan whose row claims are internally inconsistent.
InvalidFramesPerSecond
A raw value that cannot represent a dataset frame rate.
InvalidTimestampRange
A pair of timestamps that cannot represent a video segment.
InvalidWrittenVideo
Video facts that cannot describe a written media file.
ObjectStoreDatasetFileProvider
A dataset-file provider backed by any Apache Arrow ObjectStore.
PathTemplateRenderError
An invalid or incomplete LeRobot path template.
PlannedEpisodeDataRows
One selected episode’s expected rows inside a packed data parquet.
PreparedExportInputs
Proof that the files for one exact export request are available locally.
RemoteDatasetSource
A remotely backed dataset location: its file provider plus a stable source identity.
SourceSegment
One episode’s slice of a source video file, handed to a VideoSegmentWriter for concatenation into an export video.
TimestampRange
A finite, non-empty half-open timestamp range measured in seconds.
VideoSegment
One episode’s slice of a concatenated video file, in seconds.
WrittenVideo
What a VideoSegmentWriter actually encoded into a destination file. export_subset copies these facts into the exported info.json video features, replacing the source declarations (every export re-encodes).

Enums§

DatasetCachePolicy
Local materialization policy for a remotely backed dataset.
DatasetFileMaterialization
Result of materializing one remote dataset file into a local cache.
DatasetInfoParseError
Failure to parse a packed LeRobot dataset’s meta/info.json document.
DatasetOpenProgress
A measurable edge while opening dataset metadata. Callers may use these events for liveness without turning an unconditional timer into fake progress.
DatasetSource
The parsed, validated location of a dataset’s raw files.
EpisodeVectorSelectionMode
How one packed parquet was narrowed to the selected episodes.
ExportDestinationProblem
Why the export destination guard rejected a destination directory.
FeatureDtype
Data type of a feature declared in meta/info.json.
LerobotError
Errors returned by the LeRobot v3 dataset reader and export writer.
ObjectStoreDatasetSourceError
Failure to construct an object_store-backed DatasetSource from a URL and backend options.
VideoFeatureKind
Semantic role of a feature whose LeRobot dtype is video.

Constants§

EXPORT_STAGING_OWNERSHIP_MANIFEST_FILE_NAME
Versioned ownership proof that a higher-level export coordinator may place in a staging directory before calling export_subset. The coordinator is responsible for validating its contents; this crate only reserves the file name so its destination claim does not mistake that proof for dataset data.

Traits§

DatasetExportExtension
Optional format-extension behavior applied inside a subset export.
DatasetFileProvider
The remote file capabilities required by a LeRobot dataset reader.
VideoSegmentWriter
Dependency-inversion seam for video handling: this crate identifies the source segments, while a caller-provided media implementation performs the cutting and concatenation.

Functions§

export_episode_range_preserving_packed_videos
Exports ONE episode range of a larger canonical dataset into dest_root as final-shaped artifacts: its data parquet at data/chunk-000/file-{output_file_index:03}.parquet, its episodes metadata file, and its packed video files at their source-relative (final) keys. Frame rows keep SOURCE task_index values, and every global (canonical episode index, global index, dataset_from/to) is derived from placement, so concurrent ranges of one dataset are fully independent. Deliberately absent: meta/info.json and meta/tasks.parquet — the caller finalizes those once over all ranges.
export_subset
Exports the given episodes of src (re-indexed 0..n in the given order) as a new self-contained LeRobot v3 dataset under dest_root.
export_subset_preserving_packed_videos
Exports selected episodes while copying their packed source video files byte-for-byte instead of decoding or re-encoding them.
export_subset_preserving_packed_videos_with_extension
Variant of export_subset_preserving_packed_videos that applies an optional metadata extension.
export_subset_with_extension
Variant of export_subset that applies an optional metadata extension.
read_episodes_vector_columns_from_file
Reads the requested vector columns for a SET of episodes out of a single data parquet file in ONE pass, decoding each Float32 list column (List, LargeList, or FixedSizeList) into Vec<f32> rows in frame order.
render_path_template
Renders a LeRobot v3 path template such as videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4.
write_packed_subset_info_json
Writes the assembled subset’s meta/info.json after every range landed: the SOURCE document with totals patched and non-retained video features stripped — the finalize counterpart of export_episode_range_preserving_packed_videos, which deliberately skips info.
write_packed_subset_info_json_with_extension
Variant of write_packed_subset_info_json that applies an optional metadata extension.

Type Aliases§

DatasetFileProviderError
Opaque failure returned by a DatasetFileProvider.
DatasetOpenProgressReporter
Callback invoked when opening a dataset makes measurable progress.