# o41-training-data
Deterministic, resumable training batch streams. A stream walks ordered
curriculum stages; each stage is a weighted mix of sources, and a source is a
selection from a data manifest or a list of Parquet shard files. The crate owns
storage access, shard decoding, manifest resolution, mixing, shuffling,
batching, projection to fixed-shape columns, prefetch, and the resumable loop
cursor.
It is the core behind the `o41-training-data` Python package and exposes a C API
declared in `include/o41_training_data.h`. The crate reads nothing from the
environment apart from the opt-in `O41_TRAINING_DATA_PROFILE` phase counters:
storage settings and credentials arrive in the config.
Batch assembly copies selected rows directly from cached shards. Rows held for
shuffle or a partial batch stay as coordinates while their shard remains cached.
Before eviction, the engine copies only the rows still needed. Returned batches
own their projected buffers, so retaining a batch does not retain an evicted
shard. This also applies after checkpoint restore.
After `next` returns `None`, Rust callers can read `Loader::dropped_rows()`: the
rows batching discarded on this rank since opening or advancing the loader. It
stays zero before the consumer reaches the end, even when prefetch has already
read the tail. Summing it across ranks counts a `dropLast` tail once, plus any
batches discarded because the last group across ranks was incomplete. This is
consumer-side information, separate from the resumable cursor.
`open` refuses a config whose memory bound exceeds `memoryLimitBytes`, by
default half the host's physical memory. The bound counts one decoded shard per
reader slot, `threads * lookahead` of them or one per file when there are fewer,
each at the largest shard's rows; a decode workspace for each of the `threads`,
or for each slot when there are fewer; and the shuffle buffer and queued
batches. A Parquet reader decodes 128 rows at a time, or as many as fit 8 MiB of
wide rows, and writes them straight into its shard.
A shard read whole that its source reads next again, the single file of an
infinite source, stays cached into its next epoch. Evicted at each epoch's end,
it was read again with nothing read ahead, and the epoch's first batch waited
for the whole file.
Projected fields accept `i32`, `i64`, `f32`, `f64`, and `bool`. The floating
types produce Arrow `Float32` and `Float64` buffers, including inside shaped
fields. JSON and MessagePack float values and Parquet double columns reach an
`f64` field without an intermediate `f32` conversion. Padding uses the field's
declared dtype. The memory bound counts eight bytes per `f64` element.
A Parquet row group is fetched whole, ahead of decoding, when its projected
columns hold at most `maxRowGroupBytes` uncompressed (64 MiB by default). A
larger row group is read in slices as it is decoded, its column slices together
within `maxRowGroupBytes`, so a row group of any size reads; from object storage
it can read slower, without the prefetch running ahead of decoding. Write row
groups of a few thousand rows of wide fields, for example pyarrow's
`row_group_size=4096`, to keep them under the limit. A page of a sliced row
group is held whole while decoded and three slices of a column can be alive at
once, so a page over one slice, `maxRowGroupBytes` divided by three times the
projected column count, is refused, as is a page header the reader cannot read.