# o41-training-data
Deterministic, resumable training batch streams. A stream walks ordered
curriculum stages; each stage is a weighted mix of sources, and a source is a
selection from a data manifest or a list of Parquet shard files. The crate owns
storage access, shard decoding, manifest resolution, mixing, shuffling,
batching, projection to fixed-shape columns, prefetch, and the resumable loop
cursor.
It is the core behind the `o41-training-data` Python package and exposes a C API
declared in `include/o41_training_data.h`. The crate reads nothing from the
environment apart from the opt-in `O41_TRAINING_DATA_PROFILE` phase counters:
storage settings and credentials arrive in the config.
Batch assembly copies selected rows directly from cached shards. Rows held for
shuffle or a partial batch stay as coordinates while their shard remains cached.
Before eviction, the engine copies only the rows still needed. Returned batches
own their projected buffers, so retaining a batch does not retain an evicted
shard. This also applies after checkpoint restore.
Projected fields accept `i32`, `i64`, `f32`, `f64`, and `bool`. The floating
types produce Arrow `Float32` and `Float64` buffers, including inside shaped
fields. JSON and MessagePack float values and Parquet double columns reach an
`f64` field without an intermediate `f32` conversion. Padding uses the field's
declared dtype. The memory bound counts eight bytes per `f64` element.
A Parquet row group is fetched whole, ahead of decoding, when its projected
columns hold at most `maxRowGroupBytes` uncompressed (64 MiB by default). A
larger row group is read in slices as it is decoded, its column slices together
within `maxRowGroupBytes`, so a row group of any size reads; from object storage
it can read slower, without the prefetch running ahead of decoding. Write row
groups of a few thousand rows of wide fields, for example pyarrow's
`row_group_size=4096`, to keep them under the limit. A page of a sliced row
group is held whole while decoded and three slices of a column can be alive at
once, so a page over one slice, `maxRowGroupBytes` divided by three times the
projected column count, is refused, as is a page header the reader cannot read.