# o41-training-data
Deterministic, resumable training batch streams. A stream walks ordered
curriculum stages; each stage is a weighted mix of sources, and a source is a
selection from a data manifest or a list of Parquet shard files. The crate owns
storage access, shard decoding, manifest resolution, mixing, shuffling,
batching, projection to fixed-shape columns, prefetch, and the resumable loop
cursor.
It is the core behind the `o41-training-data` Python package and exposes a C API
declared in `include/o41_training_data.h`. The crate reads nothing from the
environment apart from the opt-in `O41_TRAINING_DATA_PROFILE` phase counters:
storage settings and credentials arrive in the config.
Batch assembly copies selected rows directly from cached shards. Rows held for
shuffle or a partial batch stay as coordinates while their shard remains cached.
Before eviction, the engine copies only the rows still needed. Returned batches
own their projected buffers, so retaining a batch does not retain an evicted
shard. This also applies after checkpoint restore.
Projected fields accept `i32`, `i64`, `f32`, `f64`, and `bool`. The floating
types produce Arrow `Float32` and `Float64` buffers, including inside shaped
fields. JSON and MessagePack float values and Parquet double columns reach an
`f64` field without an intermediate `f32` conversion. Padding uses the field's
declared dtype. The memory bound counts eight bytes per `f64` element.