o41-training-data

Deterministic, resumable training batch streams. A stream walks ordered curriculum stages; each stage is a weighted mix of sources, and a source is a selection from a data manifest or a list of Parquet shard files. The crate owns storage access, shard decoding, manifest resolution, mixing, shuffling, batching, projection to fixed-shape columns, prefetch, and the resumable loop cursor.

It is the core behind the o41-training-data Python package and exposes a C API declared in include/o41_training_data.h. The crate reads nothing from the environment apart from the opt-in O41_TRAINING_DATA_PROFILE phase counters: storage settings and credentials arrive in the config.

Batch assembly copies selected rows directly from cached shards. Rows held for shuffle or a partial batch stay as coordinates while their shard remains cached. Before eviction, the engine copies only the rows still needed. Returned batches own their projected buffers, so retaining a batch does not retain an evicted shard. This also applies after checkpoint restore.