lecop 0.3.1

Fast health checks for LeRobot-format robot datasets (unofficial)
Documentation
# Usage

```bash
lecop inspect  path/to/dataset   # summary from metadata only
lecop .                          # same as `lecop validate .`: scan the current directory for datasets
lecop validate                   # scan the current directory for datasets
lecop validate path/to/dataset   # one dataset, or a directory that contains many
```

`validate` is the default command: `lecop <paths> [flags]` is `lecop validate <paths> [flags]`. `validate` takes any
number of paths. A path that holds `meta/info.json` is a dataset; any other directory is
searched for datasets (`.gitignore` is respected). Datasets are checked in parallel and reported in sorted order.

```
$ lecop validate datasets
datasets/c_so100_press_red_button: ERROR   ep 7      video_frame_count        observation.images.left has 591 frames, episode has 588
datasets/c_so101_my_task_50ep: WARNING ep 18     idle_episode             64% of frames repeat the previous action; ...
Found 22 errors and 6 warnings in 3 datasets.
```

A clean run prints `All checks passed!`. Exit code: 0 clean (warnings do not fail), 1 if any error, 2 if a dataset,
the config or an argument cannot be read. A dataset whose metadata cannot be loaded (missing or invalid `meta/info.json`,
unsupported version, `fps` not positive) is reported as `cannot_open`: none of its other checks run, and `--ignore`,
`--select` and the config cannot hide it.

| Option | |
|---|---|
| `--format text\|full\|json\|github` (alias `--output-format`) | `full` adds the location and a one-line explanation per finding; `json` is one object (`errors`, `warnings`, `ignored`, `issues[]`); `github` prints `::error`/`::warning` annotations |
| `--statistics` | findings per issue code |
| `-q`, `--quiet` | summary only, nothing on success |
| `--no-color` | colour is also off when piped or when `NO_COLOR` is set |
| `--ignore a,b`, `--select a,b` | drop, or keep only, these issue codes |
| `--decode` | decode the video frames and report undecodable, frozen and black cameras; needs a build that includes the decoders (the default), see below |
| `--fix`, `--preview` | apply the safe repairs, or show them without changing anything (`--diff` is an alias), see below |
| `--config FILE`, `--isolated` | use this config, or none |
| `-v`, `-vv` | log progress, then per-file timings, to stderr (never stdout); `RUST_LOG` overrides |
| `-j N` | worker threads (default: one per core); output is identical for every `-j` |

## Repairing

`validate` alone never writes. `--fix` (also `lecop . --fix`) applies the only repairs leCop makes: the totals in
`meta/info.json` recomputed from metadata it does not touch, then validates again. `--preview` shows what `--fix` would
change and writes nothing. When a safe repair exists, a plain run ends with `[*] 1 dataset fixable with --fix` on stderr.

```bash
lecop validate --preview datasets/my_dataset   # prints what would change, on stderr
lecop validate --fix datasets/my_dataset    # applies it
```

| Finding | Repair |
|---|---|
| `frame_count` | `total_frames` becomes the sum of the episode lengths |
| `episode_count` | `total_episodes` becomes the number of episodes in the metadata |
| `tasks_count` | `total_tasks` becomes the number of tasks in the task metadata |

Nothing is repaired when the episode metadata and the data files disagree (`length_mismatch`, `episode_rows_missing`,
`data_missing`, `data_unreadable`, `timestamp_missing`, `episode_index`, `range_length`, `range_gap`, `index_mismatch`): a
total taken from that metadata could be as wrong as the one it replaces. Only the number changes, key order and
formatting stay. The original is kept as `meta/info.json.bak`, and `--fix` stops if that file already exists. Data
and video files are never edited.

## Decoding frames

By default leCop never decodes video: it reads container metadata only, which is why it takes milliseconds. Every
build includes the decoders; pass `--decode` to use them:

```bash
lecop validate --decode datasets/my_dataset
```

AV1 is decoded with rav1d's hand-written assembly, 3 to 4 times faster than without it on ARM (18 s against 67 s on
`c_so100_press_red_button`, Apple M-series) and 4.6 to 14 times on x86_64 (65 s against 296 s on the same dataset, AMD EPYC
7763 with AVX2, 4 threads). Every build has it on ARM (aarch64) and on x86_64 Linux, Windows and
macOS, with no extra tools: the PyPI wheels, the release binaries, `cargo binstall lecop`, `cargo install lecop`, a
Rust project that depends on the `lecop` crate and pip builds from the source distribution. On ARM the C compiler
assembles it; for x86_64 the decoder crate ships the assembled objects. On any other target the build leaves it out
unless you install [nasm](https://www.nasm.us) and enable the crate's `asm` feature. A `--decode` run on a build without the assembly ends with a `note:` line on
stderr saying so; stdout and `--format json` are unchanged. `cargo install lecop --no-default-features` builds without the decoders (9.1 MB
instead of 11.2 MB); `--decode` then says it needs a build with `--features decode`.

H.264 is decoded with openh264 (the `lecop-openh264` fork) and AV1 with rav1d (the `lecop-rav1d` fork). openh264's
x86_64 assembly is included on Linux, Windows and macOS the same way as the AV1 assembly, with no nasm needed; without
it H.264 decodes 1.4 to 2 times slower (AMD EPYC 7763, AVX2). Per frame leCop keeps the mean
luma and the difference to the previous frame, so it reports frames the decoder rejects (`video_decode`), a camera that
repeats the same picture (`video_frozen`) and one that shows black (`video_black`). The output ends with
`Video frames were decoded (--decode).` when it ran. A video in another codec is reported as `not decoded`, not skipped.
Measured on an Apple M2 Pro, release build: 1.5 s for `c_so101_pill_sorting_medoc`, 2.5 s for `c_careattrs_mug2v2`, 17 s for
`c_so100_press_red_button` and 2.7 s for `c_so101_my_task_50ep` (AV1); 0.26 s for `c_d3il_avoiding_vision_224` (H.264). The
feature adds about 2.1 MB to the binary (11.2 MB against 9.1 MB).

## In a pipeline

leCop is a step you put where a dataset is produced or consumed, not only in CI on a repository: right after
recording or conversion, as a gate before a training job, or on a schedule against a dataset repository. Exit code 1
means errors, so a failing run stops the job; `--format json` and `--format github` are for tools that read the result.

```bash
lecop out/my_dataset || exit 1       # after conversion: stop the pipeline on errors
lecop "$DATASET_DIR" -q || exit 1    # before training: quiet on success, summary on failure
```

The default checks read `meta/` and the parquet files, not the videos, so a dataset can be checked without downloading
them. Fetch everything but the videos, and ignore the one error that says they are absent:

```bash
hf download lerobot/pusht --repo-type dataset --exclude "videos/*" --local-dir pusht
lecop --ignore video_missing pusht
```

Without the videos, every check on the metadata and the data files runs; the checks that read an mp4 (frame count,
frame rate, resolution, `--decode`) cannot, and the output ends with `Ignored by config: video_missing (1)` so the
gap is visible. On `lerobot/pusht` (832 KB without videos) this reports the same three warnings as on the full
dataset. Download the videos and drop `--ignore` when you want the video checks.

### GitHub Action

```yaml
- uses: actions/checkout@v7
- uses: michele-xilo/leCop@v0
  with:
    path: path/to/dataset      # a dataset, or a directory to search (default: .)
    version: latest            # or a version such as 0.3.0
    args: --ignore idle_episode  # extra `lecop validate` flags
```

The step downloads the release binary for the runner (Linux, macOS and Windows; x86_64 and, on Linux and macOS,
ARM), checks its SHA-256, and runs `lecop validate --format github`, so findings appear as annotations. Errors fail
the step; warnings do not. To fail on a warning, promote its code with `error = [...]` in `lecop.toml`. leCop reads
local paths: to check a Hub dataset, download it in an earlier step (see the metadata-only fetch above).

### pre-commit

```yaml
repos:
  - repo: https://github.com/michele-xilo/leCop
    rev: v0.3.0
    hooks:
      - id: lecop
        args: [path/to/dataset]
```

The hook runs the `lecop` on your `PATH` (`uv tool install lecop`), because pre-commit would otherwise have to build
the Rust binary.

## Config file

`lecop.toml` in the scanned directory or any parent (nearest wins):

```toml
ignore = ["idle_episode"]          # drop these codes
# select = ["frame_count"]         # or keep only these
# error = ["video_fps"]            # promote a warning to an error
# warn = ["frame_count"]           # demote an error to a warning

[per-dataset."datasets/messy"]     # path relative to this file
ignore = ["video_frame_count"]

[limits."action"]                  # only `action` and `observation.*` features
min = -1.0                         # a number, or one value per dimension: [-1.0, -0.5, ...]
max = [1.0, 1.0, 0.5]
max_step = 0.2                     # largest change of any action dimension between two frames

[training]
chunk_size = 100                   # frames a policy window spans; episodes shorter than this are reported

[numeric]
clipped_fraction = 0.05            # report a dimension with this fraction of its values (or more) on its min or max
frozen_seconds = 3.0               # report an action that stays identical this long (or longer) mid-episode
```

Unknown issue codes are an error. Ignored findings are counted: `Ignored by config: idle_episode (5)`.