# o41-training-data
Deterministic, resumable training batch streams. A stream walks ordered
curriculum stages; each stage is a weighted mix of sources, and a source is a
selection from a data manifest or a list of Parquet shard files. The crate owns
storage access, shard decoding, manifest resolution, mixing, shuffling,
batching, projection to fixed-shape columns, prefetch, and the resumable loop
cursor.
It is the core behind the `o41-training-data` Python package and exposes a C API
declared in `include/o41_training_data.h`. The crate reads nothing from the
environment apart from the opt-in `O41_TRAINING_DATA_PROFILE` phase counters:
storage settings and credentials arrive in the config.
Batch assembly copies selected rows directly from cached shards. Rows held for
shuffle or a partial batch stay as coordinates while their shard remains cached.
Before eviction, the engine copies only the rows still needed. Returned batches
own their projected buffers, so retaining a batch does not retain an evicted
shard. This also applies after checkpoint restore.
A shard read whole that its source reads next again, the single file of an
infinite source, stays cached into its next epoch. Evicted at each epoch's end,
it was read again with nothing read ahead, and the epoch's first batch waited
for the whole file.
Projected fields accept `i32`, `i64`, `f32`, `f64`, and `bool`. The floating
types produce Arrow `Float32` and `Float64` buffers, including inside shaped
fields. JSON and MessagePack float values and Parquet double columns reach an
`f64` field without an intermediate `f32` conversion. Padding uses the field's
declared dtype. The memory bound counts eight bytes per `f64` element.
A Parquet row group is fetched whole, ahead of decoding, when its projected
columns hold at most `maxRowGroupBytes` uncompressed (64 MiB by default). A
larger row group is read in slices as it is decoded, its column slices together
within `maxRowGroupBytes`, so a row group of any size reads; from object storage
it can read slower, without the prefetch running ahead of decoding. Write row
groups of a few thousand rows of wide fields, for example pyarrow's
`row_group_size=4096`, to keep them under the limit. A page of a sliced row
group is held whole while decoded and three slices of a column can be alive at
once, so a page over one slice, `maxRowGroupBytes` divided by three times the
projected column count, is refused, as is a page header the reader cannot read.