o41-training-data 0.1.0

Deterministic, resumable training batch streams over Parquet shards and data manifests, with a C API
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
# o41-training-data

Deterministic, resumable training batch streams. A stream walks ordered
curriculum stages; each stage is a weighted mix of sources, and a source is a
selection from a data manifest or a list of Parquet shard files. The crate owns
storage access, shard decoding, manifest resolution, mixing, shuffling,
batching, projection to fixed-shape columns, prefetch, and the resumable loop
cursor.

It is the core behind the `o41-training-data` Python package and exposes a C
API declared in `include/o41_training_data.h`. The crate reads nothing from the
environment apart from the opt-in `O41_TRAINING_DATA_PROFILE` phase counters:
storage settings and credentials arrive in the config.