Struct TrajectoryBatch 

pub struct TrajectoryBatch<T: Float + Debug + Send + Sync + 'static> {
    pub observations: Array2<T>,
    pub actions: Array2<T>,
    pub log_probs: Array1<T>,
    pub rewards: Array1<T>,
    pub values: Array1<T>,
    pub dones: Array1<bool>,
    pub advantages: Array1<T>,
    pub returns: Array1<T>,
    pub final_observation: Option<Array1<T>>,
}

Trajectory data for RL optimization

Fields

observations: Array2<T>

Observations

actions: Array2<T>

Actions taken

log_probs: Array1<T>

Log probabilities of actions

rewards: Array1<T>

Rewards received

values: Array1<T>

Value function estimates

dones: Array1<bool>

Done flags (episode termination)

advantages: Array1<T>

Advantage estimates

returns: Array1<T>

Target returns

final_observation: Option<Array1<T>>

Observation reached after the final transition of the batch (s_T).

GAE and V-trace both need the value of the state that follows the last stored transition in order to bootstrap. That state is not part of the batch — observations.row(len - 1) is s_{T-1}, the state the last action was taken from. Bootstrapping on s_{T-1} is an off-by-one error that biases every advantage in the batch, so the successor state is carried explicitly here.

None means “no successor available” (e.g. the batch ends on a terminal transition, or the caller did not record it); consumers then bootstrap with zero. When the final transition is terminal the bootstrap is masked out by dones regardless of this field.

Implementations

impl<T: Float + Debug + Send + Sync + 'static + FromPrimitive> TrajectoryBatch<T>

pub fn new( observations: Array2<T>, actions: Array2<T>, log_probs: Array1<T>, rewards: Array1<T>, values: Array1<T>, dones: Array1<bool>, ) -> Result<Self>

Create a new trajectory batch

pub fn with_final_observation( self, final_observation: Array1<T>, ) -> Result<Self>

Attach the successor observation s_T used to bootstrap the final step.

See TrajectoryBatch::final_observation. Returns an error if the dimensionality does not match the batch’s observation dimension.

pub fn compute_gae(&mut self, gamma: T, lambda: T, nextvalue: T) -> Result<()>

Compute Generalized Advantage Estimation (GAE) without normalizing.

δ_t = r_t + γ·(1 − done_t)·V(s_{t+1}) − V(s_t)
A_t = δ_t + γ·λ·(1 − done_t)·A_{t+1}
R_t = A_t + V(s_t)

done_t is read from self.dones[t] for every t, including the last one: a batch whose final transition terminates the episode must not bootstrap. nextvalue is V(s_T), the value of the successor of the final transition (see TrajectoryBatch::final_observation); it is ignored when the final transition is terminal.

pub fn compute_advantages( &mut self, gamma: T, lambda: T, nextvalue: T, ) -> Result<()>

Compute GAE advantages/returns and normalize the advantages to zero mean and unit variance (the usual policy-gradient variance reduction).

Normalization is skipped for batches of fewer than two samples (where the sample standard deviation is zero and normalizing would annihilate the signal) and whenever the spread is numerically negligible.

pub fn compute_discounted_returns( &mut self, gamma: T, nextvalue: T, ) -> Result<()>

Fill returns with plain discounted Monte-Carlo returns G_t = r_t + γ·(1 − done_t)·G_{t+1}, bootstrapping the final step with nextvalue when the final transition is non-terminal.

Used by baseline-free REINFORCE, where the advantage is the return. advantages is set to G_t − V(s_t) so downstream code that reads advantages stays meaningful when a value baseline happens to be present.

pub fn get_mini_batches(&self, mini_batchsize: usize) -> Vec<TrajectoryBatch<T>>

Get mini-batches for optimization

Trait Implementations

impl<T: Clone + Float + Debug + Send + Sync + 'static> Clone for TrajectoryBatch<T>

fn clone(&self) -> TrajectoryBatch<T>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug + Float + Debug + Send + Sync + 'static> Debug for TrajectoryBatch<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.