o41-training-data 0.1.0

Deterministic, resumable training batch streams over Parquet shards and data manifests, with a C API

Coverage
5.83%
6 out of 103 items documented0 out of 16 items with examples
Size
Source code size: 185.1 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 2.1 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 31s Average build duration of successful builds.
all releases: 24s Average build duration of successful builds in releases after 2024-10-23.
Links
Repository
crates.io
Dependencies
Versions
Owners

o41-training-data

Deterministic, resumable training batch streams. A stream walks ordered curriculum stages; each stage is a weighted mix of sources, and a source is a selection from a data manifest or a list of Parquet shard files. The crate owns storage access, shard decoding, manifest resolution, mixing, shuffling, batching, projection to fixed-shape columns, prefetch, and the resumable loop cursor.

It is the core behind the o41-training-data Python package and exposes a C API declared in include/o41_training_data.h. The crate reads nothing from the environment apart from the opt-in O41_TRAINING_DATA_PROFILE phase counters: storage settings and credentials arrive in the config.