# Changelog
All notable changes to **thrust-rl** are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
with the pre-1.0 convention that **breaking API changes will land in MINOR
bumps** (e.g. `0.1.x -> 0.2.0`) until we cut `1.0.0`. Patch releases
(`0.1.x -> 0.1.y`) are bug fixes and additive, non-breaking improvements.
See [`docs/RELEASING.md`](docs/RELEASING.md) for the maintainer-facing
release procedure (tagging, GitHub Release creation, and `cargo publish`).
## [Unreleased]
Nothing yet.
## [0.4.0] - 2026-07-09
### Summary
The Atari release. 0.4.0 lands the full pixel-observation stack — an
opt-in `env-atari` feature wrapping the Arcade Learning Environment via a
subprocess frame protocol, the Machado et al. (2018) preprocessing
pipeline, and Nature-DQN-scale CNN policies — validated end-to-end by a
20M-frame Pong DQN training campaign with an honest, committed
learning-curve report. The large-net workload finally fired the backend
re-evaluation triggers: GPU decisively flips the small-net CPU-wins
finding (7–64× on Metal, 65–876× on an RTX 4090), and an opt-in
`training-fp16` mixed-precision path ships with dynamic loss scaling
(~1.3× end-to-end on the GPU training loop; isolated compute is
dtype-neutral on this convolution-bound net — both results measured and
reconciled in-tree). The CI-doc-tested tutorial series is complete
(tutorials 1–7), and one negative result is documented in-tree: the
Mnih-2015 RMSProp learning rate copied onto Adam does not train (flat
−20.8 through 9.5M frames) — the committed default is the
Rainbow/Dopamine 6.25e-5.
### Added
- **`env-atari` feature**: `AtariEnv` — a subprocess adapter to Farama
`ale-py` over a length-prefixed stdin/stdout frame protocol
(`envs/atari/ale_worker.py`; `ALE_ROM_PATH` resolution; no ROMs in the
repo or crate tarball, audited in CI), with seeded reset and full
`clone_state`/`restore_state` via opaque ALE state blobs (#325, #333).
Design note: `docs/ALE_BINDING_STRATEGY.md` (#324, #331).
- **`AtariPreprocess`**: Machado et al. (2018) evaluation-protocol
preprocessing on top of `AtariEnv` — sticky actions (p=0.25, seeded),
frame-skip 4 with 2-frame max-pooling, BT.601 grayscale, bilinear
84×84 downsample, 4-frame stacking, configurable
episode-end-on-life-loss; the wrapper state round-trips through
`clone_state`/`restore_state` bit-identically (#326, #336).
- **Nature-DQN CNN policies**: `NatureDqnBurnPolicy` (actor-critic) and
`NatureDqnQNetwork` (Mnih 2015 conv trunk, seeded FC init,
record-based target sync; 1.69M params at 4 actions) (#327, #332).
- **`training-fp16` feature**: opt-in f16 mixed-precision GPU training
path (`Cuda<f16>`/`Wgpu<f16>` with a compile-time guard against
CPU-only builds), dynamic loss scaling with f16-range-safe capping,
and an additive `DQNTrainerBurn::train_step_scaled` — the f32 path is
bit-identical to 0.3.0. Verified by a 500k-step CUDA acceptance run
with zero overflows (#270, #347).
- **Pong DQN training example + runbook**: `train_pong_dqn` (and
`train_pong_dqn_fp16`) with env-var knobs, learning-curve CSV output
and checkpointing; `docs/PONG_DQN_RUNBOOK.md` with an experiment log;
`docs/research/2026-07-pong-dqn-learning-curve.md` with all three
runs' committed curves (#329, #341, #343, #344).
- **Large-net throughput benchmarks**: `nature_dqn_*` criterion groups
(forward and train-step, b32/b256) registered across ndarray, wgpu,
cuda, and — behind `training-fp16` — cuda-f16/wgpu-f16 (#328, #334,
#348).
- **Memory-hard environment suite**: T-maze POMDP and burst-structured
flickering CartPole (#302, #315).
- **Tutorial series complete**: CI-doc-tested tutorials 1–7 (getting
started, PPO, DQN, SAC, recurrent PPO/POMDPs, custom environments,
WASM deployment) (#303, #314, #319–#323).
- **SignalingGame** protocol-emergence experiment and research writeup
(#308).
- **CI**: advisory `atari_integration` job running the real-`ale-py`
integration tests (bundled Pong ROM, skip-detection guard) (#338,
#339).
### Changed
- **`PsroConfig` gains `serialize_br_updates: bool` (default `true`)** —
serializes per-agent best-response backward passes to avoid a
lock-order deadlock in burn-autodiff 0.21's graph mutexes under
rayon-parallel training. Breaking for exhaustive struct literals
(pre-1.0 minor per policy). The revert-on-upstream-fix is tracked in
#337 (#307, #317).
- **`docs/BURN_BACKENDS.md` re-measured at large-net scale**: the CNN
workload flips the small-net CPU-wins finding — wgpu/Metal wins
7–64×, cuda (RTX 4090) 65–876×; timed regions now force an in-region
device sync for honest GPU numbers; f16-vs-f32 measured and the
end-to-end-vs-isolated discrepancy reconciled (#328, #335, #340,
#345, #348).
- Atari worker protocol: `Obs` responses carry the ALE `lives` count
(coordinated protocol/worker/env change within the `env-atari`
feature) (#336).
### Fixed
- `FlickeringCartPole` burst mode now rejects
`flicker_prob > burst_len / (burst_len + 1)` at construction — the
previous clamp silently pinned the stationary blank rate below the
documented value in the extreme-p regime (#316, #318).
- PSRO parallel best-response training no longer wedges on macOS arm64
(burn-autodiff graph-lock deadlock; see Changed) (#307, #317).
- GPU benchmark timing under-reported deferred kernel work (missing
device sync inside the timed region); wgpu table refreshed with
honest numbers (#335, #340).
- Doc-data corrections from post-merge audit: qnet cuda speedup ratios
and the learning-curve best-window figure recomputed from primary
logs (#345, #346).
## [0.3.0] - 2026-07-06
### Summary
The recurrent-policy and off-policy release. 0.3.0 lands the complete LSTM
policy stack — recurrent actor-critic, sequence-aware rollout buffer, and a
recurrent PPO trainer validated on a FlickeringCartPole POMDP where memory is
load-bearing (LSTM 484 peak vs memoryless MLP 176) — alongside V-trace
(IMPALA) off-policy correction and a single-host async actor-learner that
reaches the CartPole bar at 2.1× synchronous throughput. Multi-agent
communication Phases 1–2 (signaling channel + trainer hook) and the k*
coalition-improvability phase diagram (75 cells, cluster-run) round out the
research tooling. Includes one notable negative result, documented in-tree:
velocity-masked CartPole is not a POMDP (a reactive controller beats the
LSTM), which motivated the flickering variant.
All changes are additive public API (MINOR bump per the pre-1.0 policy); see
the `max_grad_norm` note under Fixed for one deliberate behavior change.
### Added
#### Recurrent policies
- `LstmBurnPolicy` recurrent actor-critic with seeded LSTM init (#291).
- `RecurrentRolloutBuffer` + full-sequence sampler (#294).
- `RecurrentPPOTrainer`, `FlickeringCartPole` / `MaskedCartPole` envs, and an
LSTM-vs-MLP memory-contrast example (#298).
#### Off-policy & throughput
- V-trace (IMPALA) off-policy advantage estimator (#283) and V-trace
correction in the actor-learner learner step (#297).
- Single-host async actor-learner PPO over crossbeam channels (#296).
#### Multi-agent communication
- `CommunicatingEnvironment` supertrait + `SignalingGame` reference env (#292).
- SignalingGame integration into the joint trainer + comms loss hook (#295).
#### Research tooling & docs
- Coalition improvability oracle measuring the k* threshold (#271); raw
(β,κ,c) k* sweep mode + full 75-cell phase-diagram artifact (#290).
- Live training dashboard page with recorded CartPole curves (#284).
- Design notes: recurrent policies (#288), distributed / multi-GPU training
(#282), multi-agent comms channels (#277), FP16 feasibility spike (#273),
burn-collective allreduce spike results (#293).
### Fixed
- **PPO trainers now honor `PPOConfig::max_grad_norm`** (#300): previously
both PPO trainers silently ignored the field (only SAC/A2C clipped).
Clipping is applied as a direction-preserving global L2-norm clip via a
shared `clip_grads_by_global_norm` helper. This is a behavior fix —
training trajectories change for existing configs, which were already
requesting clipping. Revalidated on the recurrent example (#301).
- WASM Pages build: configure the `getrandom` 0.3 `wasm_js` backend.
## [0.2.0] - 2026-06-30
### Summary
The multi-agent research release. 0.2.0 matures the population-based
multi-agent stack — PSRO (with an α-rank meta-solver) and approximate
NFSP — to the point of running the Bucket Brigade workshop paper's
no-convergence cells end to end, adds the A2C algorithm, and lands
substantial training-throughput work. It also closes out that research
question: an extensive validation effort (documented under
`docs/research/`) concluded — and root-caused via a non-PPO improvability
oracle — that neither PSRO nor NFSP beats PPO on those cells because the
single best-response has ~no team-return headroom against frozen-uniform
opponents. The negative result and its mechanism are fully documented.
All changes are additive; existing public APIs are unchanged (MINOR bump
per the pre-1.0 policy).
### Added
#### Algorithms & training
- **A2C**: `A2cTrainer` with an un-clipped advantage-actor-critic loss,
smoke tests, a CartPole example, opt-in learning-curve CSV export, and a
convergence test (#156, #158).
- **PSRO**: payoff-magnitude fixes for α-rank saturation plus best-response
critic reward scaling, making the α-rank meta-solver usable at the
no-convergence cells' payoff scale (#225).
- **NFSP**: adaptive average-policy supervised-step floor and reward scaling
to address average-policy under-training (#223).
- **Joint trainer**: tunable best-response-fit defaults and an optional
separate critic optimizer (#239).
#### Bucket Brigade research tooling
- Critic **explained-variance** diagnostic surfaced from the joint PPO
update, plus a standalone single-best-response A/B probe harness
(`train_br_probe`) for fast critic-fit investigation (#241, #247).
- Non-PPO **best-response improvability oracle** (`bucket_brigade_oracle`
module + `br_oracle` example): scores scripted/searched policies against
frozen-uniform opponents to bound achievable best-response return (#259).
- Cell-specific `gap_closed_cell` reporting in the bucket-brigade examples
(#220).
- `SEED` env override across the bucket-brigade training examples for
seeded multi-host replication, plus cluster run/dispatch scripts (#256).
#### Features
- Off-by-default `blas-threaded` feature for threaded NdArray matmul (#237).
- Opt-in `max_minibatches_per_epoch` best-response-update throughput lever
(#254).
### Changed
- Dropped the `linker = "clang"` pin in `.cargo/config.toml`, keeping the
fast `lld` linker via `-fuse-ld=lld` (works with the universal gcc/cc
driver; removes a needless clang requirement) (#257).
### Fixed
- Best-response PPO update now applies the configured grad-norm clip and
iterates all minibatches per epoch (previously the clip was dead and most
of each rollout was discarded) (#240).
- Snake env food respawn is now deterministic via an owned seeded RNG.
- Cleared `env-bucket-brigade` clippy lints and gated them in CI (#231).
### Performance
- PSRO round-robin best-response loop parallelized with rayon (#238).
- NFSP per-agent average-policy supervised training parallelized with rayon
(#236), and per-step seeded sampling batched through a single model
forward (#249).
## [0.1.0] - 2026-06-08
First publishable release of thrust-rl. Locks in the foundation laid by
the initial development sprints: two on-policy / off-policy training
algorithms (PPO and DQN), four environments with both single-agent and
multi-agent variants, the multi-agent population/matchmaking
infrastructure, WASM-deployable pure-Rust inference, browser demos, and
Bayesian-style hyperparameter exploration. Everything below is brand new
relative to "no published version".
### Added
#### Algorithms
- **PPO** (`src/train/ppo*`): clipped-surrogate Proximal Policy
Optimization with Generalized Advantage Estimation (GAE), an
auxiliary-loss hook (`PPOTrainer::set_aux_loss_fn`), and the
`MlpPolicy` / `MultiDiscreteMlpPolicy` policy heads. Solves CartPole
(~300-470 steps/episode on the curated configs in
`examples/games/cartpole/`).
- **DQN** (`src/train/dqn*`): vanilla DQN MVP plus Double-DQN target
computation and Polyak soft target updates (#58), and Prioritized
Experience Replay with proportional sampling and importance-sampling
correction (#59).
#### Environments
- **CartPole** (`src/env/games/cartpole.rs`): classic balancing task,
trained to >300 step episodes by the bundled configs.
- **Snake** (`src/env/games/snake/`): single-agent and multi-agent
variants with configurable grid size; multi-agent uses a population
through `multi_agent::PolicyLearner`.
- **Pong** (`src/env/games/pong.rs`): single-player vs. rule-based
opponent and self-play training pipeline (#47, #53).
- **SimpleBandit** (`src/env/games/simple_bandit.rs`): contextual bandit
used as a sanity test for PPO and the WASM inference stack.
- **ContinuousLqr** (`src/env/games/continuous_lqr.rs`): 1-D LQR
placeholder env exercising the `Vec<f32>` continuous-action surface
introduced alongside #61.
- **BucketBrigade** (Slepian-Wolf MARL research env, see
`src/env/games/bucket_brigade/`): adapter is in-tree but the
`env-bucket-brigade` feature is **disabled in the published v0.1.0
manifest** because the underlying `bucket-brigade-core` crate has not
been published to crates.io. Local checkouts can still enable it by
un-commenting the feature in `Cargo.toml`. See "Known limitations"
below.
#### Multi-agent infrastructure
- `multi_agent::Population` and `Agent` for population-based training.
- `GameSimulator` skeleton and `JointMultiAgentTrainer` for synchronized
joint PPO across agents (#15).
- `multi_agent::PolicyLearner` with the per-epoch / per-step bugs from
the original Slepian-Wolf port resolved (#4, #41, #34, #39).
- Four matchmaking strategies: `Random`, `RoundRobin`, `Fitness`,
`SelfPlay`.
- Multi-discrete action support in `MultiAgentEnvironment` (#14).
- `Environment::clone_state` / `restore_state` for MCTS, replay, and
rollback workflows (#62).
#### WASM and browser demos
- Pure-Rust inference module (`src/inference/`) compiles to WASM with no
PyTorch dependency.
- Universal inference system with JSON model format
(`docs/UNIVERSAL_INFERENCE.md`).
- React + Vite web app under `web/` (excluded from the crate tarball)
with live CartPole, Snake, and SimpleBandit demos; SimpleBandit
agent visualizer; Pong web demo with self-play model and rule-based
fallback (#76).
#### Hyperparameter optimization
- Bayesian / Pareto-style hyperparameter search machinery
(`src/optim/`) used by the `optimize_cartpole` and `optimize_snake`
examples to sweep PPO clip ranges, value-function coefficients, and
network widths.
#### Documentation
- `README.md` with project vision, status, and demo links.
- `ROADMAP.md` and `MULTI_AGENT_DESIGN.md` for forward planning.
- `docs/PPO_BEST_PRACTICES.md`, `docs/RESEARCH_PAPERS.md`,
`docs/RL_TOYBOX_SURVEY.md` (#49).
- `docs/GPU_SETUP.md`, `docs/LIBTORCH_SETUP.md`,
`docs/REMOTE_TRAINING.md`, `docs/TRAINING_*` per-environment guides.
- Public-API rustdoc audit completed (#36, #38).
#### Tooling
- `build.rs` defeats GNU ld's `--as-needed` for `libtorch_cuda` on
Linux so CUDA-equipped boxes don't silently fall back to CPU (#13).
- GPU training scripts (`scripts/gpu-*.sh`, excluded from the crate
tarball) for an NVIDIA L4 sandbox.
- Loom orchestration framework installed under `.loom/` for AI-assisted
development (excluded from the crate tarball).
### Known limitations
- **DQN does not reliably solve CartPole.** PPO does. Both Double-DQN
and PER landed in v0.1.0, but the canonical CartPole DQN config still
shows high variance run-to-run. Tracked by the open follow-up issues
surfaced during #58/#59 review.
- **`env-bucket-brigade` is source-only in the published crate.** The
underlying `bucket-brigade-core` Cargo crate lives in a git submodule
and is not on crates.io, so `cargo publish` cannot include it as an
optional dependency. Users who need the env should depend on
`thrust-rl` from a git checkout with submodules initialized and
un-comment the relevant lines in `Cargo.toml`. Re-publishing
`bucket-brigade-core` is tracked as a v0.2.x follow-up.
- **`multi_agent::PolicyLearner` API is experimental** and is expected
to change as multi-agent training matures. Pin to a specific 0.1.x
patch if you depend on its current shape.
- **`tch-rs` version pinned to 0.22** (PyTorch 2.9), which requires
Rust nightly + `edition = "2024"`. See `rust-toolchain.toml`.
- **No binary `release` artifacts.** Consumers build from source via
`cargo install thrust-rl` (library) or by cloning the repo for
examples. A binary release workflow is intentionally out of scope
for v0.1.0.
### Removed (relative to never-published state)
- Empty `BayesianOptimizer` / `ParetoFrontier` / `TrialScheduler`
placeholder modules were dropped before publish.
- `bincode` dependency was removed to resolve RUSTSEC-2025-0141 (#37).
- Misleading `target_synced` field on `DQNStepStats` (#66).
[Unreleased]: https://github.com/rjwalters/thrust/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/rjwalters/thrust/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/rjwalters/thrust/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/rjwalters/thrust/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/rjwalters/thrust/releases/tag/v0.1.0