Struct RecurrentRolloutBuffer 

pub struct RecurrentRolloutBuffer { /* private fields */ }

Rollout buffer that preserves temporal order for a recurrent policy.

Stores the same nine [num_steps, num_envs] transition arrays as RolloutBuffer (via composition) plus per-step recurrent state hidden / cell, each shaped [num_steps, num_envs, hidden_dim].

Implementations

impl RecurrentRolloutBuffer

pub fn new( num_steps: usize, num_envs: usize, obs_dim: usize, hidden_dim: usize, ) -> Self

Create a new recurrent rollout buffer.

Arguments

• num_steps - Number of timesteps per rollout
• num_envs - Number of parallel environments
• obs_dim - Dimensionality of observations
• hidden_dim - Width of the LSTM (h, c) state

pub fn add( &mut self, step: usize, env_id: usize, observation: &[f32], action: i64, reward: f32, value: f32, log_prob: f32, terminated: bool, truncated: bool, )

Add a transition to the buffer.

Delegates verbatim to RolloutBuffer::add; see that method for argument semantics. The recurrent (h, c) for this step is recorded separately via Self::add_recurrent_state.

pub fn add_recurrent_state( &mut self, step: usize, env_id: usize, h: &[f32], c: &[f32], )

Record the recurrent state (h, c) entering step step for env env_id.

The rollout loop calls this before the env step, so the stored state is the one that entered the step — used for warm-start verification (via Self::seed_warm_start) and reserved as the hook a future Strategy B trainer would reuse. The training forward itself does not read these back (it recomputes states from a zeroed initial_state).

Panics

Panics (debug builds) if step/env_id are out of range or if h/c do not have length hidden_dim.

pub fn seed_warm_start( &mut self, last_step: usize, final_hidden: &[Vec<f32>], final_cell: &[Vec<f32>], )

Seed the step-0 recurrent state for the next rollout iteration, in place.

For each env, if it did not end its episode (terminated || truncated) on step last_step, its final recurrent state final_hidden[env] / final_cell[env] (the state exiting the last collected step) is carried into hidden[0][env] / cell[0][env]. If it did end its episode, step 0 is seeded with zeros — the episode already ended, so no memory should survive into the fresh one. This is the warm-start half of the GAE/state-reset asymmetry: the carry decision follows terminated || truncated, not terminated alone.

final_hidden / final_cell are indexed [env][hidden_dim] and must have num_envs rows.

This method also records the per-env cross-iteration carry-in flag that becomes episode_starts[0] of the next materialized batch: an env that ended its episode gets flag 1.0 (its step-0 state resets), a live env gets 0.0 (its state continues). The flag is set in lockstep with the (h, c) carry so the reset mask and the seeded state always agree.

Strategy A note (the (h, c) carry is a Strategy-B hook)

The (h, c) seeded into hidden[0] / cell[0] here is not consumed by Strategy A’s Self::to_sequence_batch, which always passes initial_state: None (zeros) to the forward — an intentional BPTT truncation at the iteration boundary. Cross-iteration continuity is instead approximated only through the carry-in flag above (which controls the reset, not the value, of step 0’s state). The seeded (h, c) storage is a forward-looking hook a future Strategy B trainer (fixed-length subsequences with stored boundary states) would feed in; under Strategy A it is deliberately left unread.

Panics

Panics if last_step >= num_steps, if final_hidden / final_cell do not have num_envs rows, or (debug builds) if any row is not hidden_dim wide.

pub fn reset(&mut self)

Reset the buffer’s advantages/returns for a new rollout.

Delegates to RolloutBuffer::reset. The recurrent (h, c) arrays and the episode_start_carry flag are intentionally left in place — (h, c) is overwritten by Self::add_recurrent_state during the next collection, and both the step-0 state and the carry flag are seeded by Self::seed_warm_start so they must survive the reset to bridge the iteration boundary.

pub fn compute_advantages( &mut self, last_values: &[f32], gamma: f32, gae_lambda: f32, )

Compute GAE over the full [num_steps, num_envs] capacity.

Delegates unchanged to compute_advantages — the [step][env] advantage/return grid is byte-identical to the feedforward path (GAE bootstraps on terminated only).

pub fn compute_advantages_partial( &mut self, valid_steps: usize, last_values: &[f32], gamma: f32, gae_lambda: f32, )

Compute GAE over the first valid_steps rows.

Delegates unchanged to compute_advantages_partial.

pub fn shape(&self) -> (usize, usize, usize)

Buffer shape (num_steps, num_envs, obs_dim).

pub fn hidden_dim(&self) -> usize

Width of the recurrent (h, c) state.

pub fn observations(&self) -> &[Vec<Vec<f32>>]

Per-step observations, indexed [step][env] then by obs dimension.

pub fn actions(&self) -> &[Vec<i64>]

Per-step discrete actions, indexed [step][env].

pub fn values(&self) -> &[Vec<f32>]

Per-step value estimates, indexed [step][env].

pub fn log_probs(&self) -> &[Vec<f32>]

Per-step behavior-policy log-probs, indexed [step][env].

pub fn terminated(&self) -> &[Vec<bool>]

Per-step terminal flags, indexed [step][env].

pub fn truncated(&self) -> &[Vec<bool>]

Per-step truncation flags, indexed [step][env].

pub fn advantages(&self) -> &[Vec<f32>]

Per-step GAE advantages, indexed [step][env].

pub fn returns(&self) -> &[Vec<f32>]

Per-step value-function targets, indexed [step][env].

pub fn hidden(&self) -> &[Vec<Vec<f32>>]

Recurrent hidden state entering each step, indexed [step][env] then by hidden dimension.

pub fn cell(&self) -> &[Vec<Vec<f32>>]

Recurrent cell state entering each step, indexed [step][env] then by hidden dimension.

pub fn episode_start_carry(&self) -> &[f32]

Per-env cross-iteration carry-in flag (episode_starts[0] source), indexed [env]. 1.0 marks that the env’s episode ended at the end of the previous rollout iteration (step 0 resets); 0.0 marks a carried-over episode.

pub fn to_sequence_batch<B: Backend>( &self, device: &B::Device, ) -> RecurrentRolloutBatch<B>

Materialize the full buffer as a rank-3 sequence batch (T = num_steps).

Convenience wrapper over Self::to_sequence_batch_partial with valid_steps == num_steps.

pub fn to_sequence_batch_partial<B: Backend>( &self, valid_steps: usize, device: &B::Device, ) -> RecurrentRolloutBatch<B>

Materialize the first valid_steps rows as a rank-3 sequence batch (T = valid_steps).

Selects all envs. See RecurrentRolloutBatch for the exact field shapes. Env-major layout: each env-trajectory is one contiguous row of length T.

Panics

Panics if valid_steps > num_steps.

pub fn to_minibatches<'a, B: Backend>( &'a self, envs_per_minibatch: usize, shuffle: bool, device: &B::Device, ) -> RecurrentMinibatchIterator<'a, B>

Build an env-major minibatch iterator over full trajectories (T = num_steps).

Shuffles environment indices (not global timesteps) and chunks them by envs_per_minibatch — the recurrent analogue of the feedforward batch_size, counting whole env-trajectories rather than loose timesteps. Over one epoch every env index appears in exactly one minibatch.

When shuffle is false the env order is 0..num_envs (useful for deterministic tests / reproducible evaluation).

Trait Implementations

impl Clone for RecurrentRolloutBuffer

fn clone(&self) -> RecurrentRolloutBuffer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RecurrentRolloutBuffer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.