o41-training-data

Deterministic, resumable training batch streams. A stream walks ordered curriculum stages; each stage is a weighted mix of sources, and a source is a selection from a data manifest or a list of Parquet shard files. The crate owns storage access, shard decoding, manifest resolution, mixing, shuffling, batching, projection to fixed-shape columns, prefetch, and the resumable loop cursor.

It is the core behind the o41-training-data Python package and exposes a C API declared in include/o41_training_data.h. The crate reads nothing from the environment apart from the opt-in O41_TRAINING_DATA_PROFILE phase counters: storage settings and credentials arrive in the config.

Batch assembly copies selected rows directly from cached shards. Rows held for shuffle or a partial batch stay as coordinates while their shard remains cached. Before eviction, the engine copies only the rows still needed. Returned batches own their projected buffers, so retaining a batch does not retain an evicted shard. This also applies after checkpoint restore.

Projected fields accept i32, i64, f32, f64, and bool. The floating types produce Arrow Float32 and Float64 buffers, including inside shaped fields. JSON and MessagePack float values and Parquet double columns reach an f64 field without an intermediate f32 conversion. Padding uses the field's declared dtype. The memory bound counts eight bytes per f64 element.