Expand description
§lerobot-dataset-io
Rust I/O for inspecting, accessing, and creating subsets of packed LeRobot
datasets.
The crate owns the LeRobot v3 format boundary. It parses meta/info.json,
reads packed episode metadata and per-frame Parquet columns, resolves episode
video segments, materializes remote files through a provider-neutral trait,
and writes reindexed subset datasets.
For an example of a public robotics-data application where this crate can be used, see Pareto.
§What it provides
- Local synchronous opens and async provider-backed opens.
- Ordered feature metadata with unknown fields preserved for round-trip export.
- RGB, depth-video, and unknown-video classification.
- Episode metadata and packed video-segment resolution.
- Projected and streaming reads of per-frame vector columns.
- Re-encoded or byte-preserving packed-video subset exports.
- Extension hooks for namespaced metadata and sidecars.
- Storage injection through
DatasetFileProvider; the crate does not depend on a cloud vendor SDK.
The reader accepts LeRobot v3.x datasets. It rejects older major versions and
proceeds optimistically for newer v3 minor versions.
§Local dataset example
use lerobot_dataset_io::Dataset;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let dataset = Dataset::open("/datasets/example")?;
println!(
"{} episodes at {} fps",
dataset.episodes().len(),
dataset.frames_per_second().get()
);
for episode in dataset.episodes() {
println!(
"episode {}: {} frames, {} video streams",
episode.episode_index(),
episode.length_frames(),
episode.video_segments().len()
);
}
Ok(())
}§Remote storage
Implement DatasetFileProvider to list dataset-relative files and atomically
materialize one file at a requested local path. Construct a DatasetSource
with DatasetSource::from_file_provider, then call Dataset::open_source.
This supports object stores, HTTP services, content-addressed storage, and
application-specific backends without adding them to the format crate.
The default object-store feature provides an adapter for Apache Arrow’s
object_store crate, including local filesystem and in-memory stores. Enable
aws, gcp, azure, or http for URL construction of the corresponding
common backend:
cargo add lerobot-dataset-io --features awsThen construct a source with DatasetSource::from_object_store_url, supplying
the backend options/credentials accepted by object_store::parse_url_opts.
The stored source identity excludes URL userinfo, query parameters, and
fragments so credentials are not copied into logs or manifests. Prefer backend
options or environment-based credential providers over credentials in a URL.
The DatasetFileProvider trait remains independently implementable for
backends outside that ecosystem.
Remote metadata is materialized eagerly. Packed data and video files are materialized on demand through the async ensure and prefetch methods before synchronous Parquet or media operations.
§Video export boundary
The crate plans video segments but deliberately does not choose or bundle a
media framework. Re-encoding exports receive a caller-provided
VideoSegmentWriter. Use export_subset_preserving_packed_videos when selected
packed video files should be copied byte-for-byte instead.
Every synchronous export requires PreparedExportInputs. For local datasets,
construct it with PreparedExportInputs::for_local_dataset; for remote
datasets, await Dataset::prepare_export_inputs. The value is bound to the
exact dataset, ordered episode selection, and video keys, and is verified
before the destination is touched.
§Format extensions
Unknown info.json fields survive subset export. Applications that interpret
their own namespaced fields or sidecars can implement DatasetExportExtension
and use the extension-aware export functions. Product-specific metadata and
cloud authentication belong in adapter crates rather than this crate.
§License
Apache-2.0.
Modules§
- format_
extension_ support - Low-level readers shared with crates that implement
LeRobotformat extensions.
Structs§
- Camera
Video Write Result - Everything a
VideoSegmentWriterreports about one written camera video. - Dataset
- A
LeRobotv3 dataset opened for reading. - Dataset
Info - Parsed
meta/info.jsonof aLeRobotv3 dataset. - Dataset
Relative Path - A portable, non-empty path known to stay beneath a dataset root.
- Dataset
Relative Path Error - A path that is not safe to resolve beneath a dataset root.
- Episode
- Per-episode metadata read from
meta/episodes/chunk-*/file-*.parquet. - Episode
Data File Read Plan - A materialized packed data file and every selected episode it contains.
- Episode
Data Read Plan - Complete read plan for a requested episode set.
- Episode
Vector Column - One requested vector column’s decoded per-frame rows.
- Episode
Vector Columns - One episode’s decoded per-frame vector columns, in the order they were requested.
- Episode
Vector Read Report - Observable work performed by one packed-file reader.
- Episode
Vector Read Schema - Projected source-column availability for a streaming vector read.
- Episode
Vector Row - One full-resolution row yielded by
EpisodeVectorRowReader. - Episode
Vector RowReader - Streaming reader that decodes selected packed-file rows exactly once.
- Export
Episodes - Episode indices selected for an export. The constructor is the boundary where an external list becomes a valid export request.
- Export
Range Placement - Where an exported episode range sits inside a larger canonical dataset; the whole-dataset export is the single-range case.
- Export
Report - Summary of a completed
export_subsetrun. - Feature
Spec - One feature entry from the
featuresobject ofmeta/info.json. - Frames
PerSecond - A finite, strictly positive dataset frame rate.
- Invalid
Episode Data Read Plan - A packed-file read plan whose row claims are internally inconsistent.
- Invalid
Frames PerSecond - A raw value that cannot represent a dataset frame rate.
- Invalid
Timestamp Range - A pair of timestamps that cannot represent a video segment.
- Invalid
Written Video - Video facts that cannot describe a written media file.
- Object
Store Dataset File Provider - A dataset-file provider backed by any Apache Arrow
ObjectStore. - Path
Template Render Error - An invalid or incomplete
LeRobotpath template. - Planned
Episode Data Rows - One selected episode’s expected rows inside a packed data parquet.
- Prepared
Export Inputs - Proof that the files for one exact export request are available locally.
- Remote
Dataset Source - A remotely backed dataset location: its file provider plus a stable source identity.
- Source
Segment - One episode’s slice of a source video file, handed to a
VideoSegmentWriterfor concatenation into an export video. - Timestamp
Range - A finite, non-empty half-open timestamp range measured in seconds.
- Video
Segment - One episode’s slice of a concatenated video file, in seconds.
- Written
Video - What a
VideoSegmentWriteractually encoded into a destination file.export_subsetcopies these facts into the exportedinfo.jsonvideo features, replacing the source declarations (every export re-encodes).
Enums§
- Dataset
Cache Policy - Local materialization policy for a remotely backed dataset.
- Dataset
File Materialization - Result of materializing one remote dataset file into a local cache.
- Dataset
Info Parse Error - Failure to parse a packed
LeRobotdataset’smeta/info.jsondocument. - Dataset
Open Progress - A measurable edge while opening dataset metadata. Callers may use these events for liveness without turning an unconditional timer into fake progress.
- Dataset
Source - The parsed, validated location of a dataset’s raw files.
- Episode
Vector Selection Mode - How one packed parquet was narrowed to the selected episodes.
- Export
Destination Problem - Why the export destination guard rejected a destination directory.
- Feature
Dtype - Data type of a feature declared in
meta/info.json. - Lerobot
Error - Errors returned by the
LeRobotv3 dataset reader and export writer. - Object
Store Dataset Source Error - Failure to construct an
object_store-backedDatasetSourcefrom a URL and backend options. - Video
Feature Kind - Semantic role of a feature whose
LeRobotdtype isvideo.
Constants§
- EXPORT_
STAGING_ OWNERSHIP_ MANIFEST_ FILE_ NAME - Versioned ownership proof that a higher-level export coordinator may place
in a staging directory before calling
export_subset. The coordinator is responsible for validating its contents; this crate only reserves the file name so its destination claim does not mistake that proof for dataset data.
Traits§
- Dataset
Export Extension - Optional format-extension behavior applied inside a subset export.
- Dataset
File Provider - The remote file capabilities required by a
LeRobotdataset reader. - Video
Segment Writer - Dependency-inversion seam for video handling: this crate identifies the source segments, while a caller-provided media implementation performs the cutting and concatenation.
Functions§
- export_
episode_ range_ preserving_ packed_ videos - Exports ONE episode range of a larger canonical dataset into
dest_rootas final-shaped artifacts: its data parquet atdata/chunk-000/file-{output_file_index:03}.parquet, its episodes metadata file, and its packed video files at their source-relative (final) keys. Frame rows keep SOURCEtask_indexvalues, and every global (canonical episode index, globalindex,dataset_from/to) is derived fromplacement, so concurrent ranges of one dataset are fully independent. Deliberately absent:meta/info.jsonandmeta/tasks.parquet— the caller finalizes those once over all ranges. - export_
subset - Exports the given episodes of
src(re-indexed0..nin the given order) as a new self-containedLeRobotv3 dataset underdest_root. - export_
subset_ preserving_ packed_ videos - Exports selected episodes while copying their packed source video files byte-for-byte instead of decoding or re-encoding them.
- export_
subset_ preserving_ packed_ videos_ with_ extension - Variant of
export_subset_preserving_packed_videosthat applies an optional metadata extension. - export_
subset_ with_ extension - Variant of
export_subsetthat applies an optional metadata extension. - read_
episodes_ vector_ columns_ from_ file - Reads the requested vector columns for a SET of episodes out of a single
data parquet file in ONE pass, decoding each
Float32list column (List,LargeList, orFixedSizeList) intoVec<f32>rows in frame order. - render_
path_ template - Renders a
LeRobotv3 path template such asvideos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4. - write_
packed_ subset_ info_ json - Writes the assembled subset’s
meta/info.jsonafter every range landed: the SOURCE document with totals patched and non-retained video features stripped — the finalize counterpart ofexport_episode_range_preserving_packed_videos, which deliberately skips info. - write_
packed_ subset_ info_ json_ with_ extension - Variant of
write_packed_subset_info_jsonthat applies an optional metadata extension.
Type Aliases§
- Dataset
File Provider Error - Opaque failure returned by a
DatasetFileProvider. - Dataset
Open Progress Reporter - Callback invoked when opening a dataset makes measurable progress.