Struct AsyncActorLearnerConfig 

pub struct AsyncActorLearnerConfig {
    pub num_actors: usize,
    pub num_steps: usize,
    pub total_env_steps: usize,
    pub broadcast_every: usize,
    pub max_lead_steps: usize,
    pub gamma: f32,
    pub gae_lambda: f32,
    pub use_vtrace: bool,
    pub vtrace_rho_bar: f32,
    pub vtrace_c_bar: f32,
    pub seed: u64,
}

Configuration for the single-host asynchronous actor-learner runner.

See the module docs for the architecture and the staleness caveat.

Fields

num_actors: usize

Number of actor threads (each owns one environment instance).

num_steps: usize

Per-actor env steps consumed per PPO update. Each update trains on a [num_steps, num_actors] rollout buffer, i.e. num_steps * num_actors transitions.

total_env_steps: usize

Total env-step budget. The learner runs total_env_steps / (num_steps * num_actors) PPO updates.

broadcast_every: usize

Broadcast updated policy weights to actors after every broadcast_every learner updates. 1 (the default) keeps actors as fresh as possible, which is what makes the uncorrected (no V-trace) PPO update acceptable on simple envs.

max_lead_steps: usize

Soft staleness valve: the maximum number of env steps an actor may run ahead of the learner’s consumption. Policy version v proves the learner has consumed v * broadcast_every * num_steps transitions from each actor, so an actor pauses (and waits for the next crate::multi_agent::PolicyBroadcast) once its total sent steps reach that mark plus max_lead_steps. This bounds the learner’s queue depth and the worst-case policy lag at max_lead_steps — a cumulative budget, unlike a per-version allowance, which would let production permanently outpace consumption and grow the backlog (and therefore the staleness) linearly over the run.

0 selects the default of 2 * broadcast_every * num_steps: one broadcast cycle of lead for pipeline overlap (actors collect while the learner trains) plus one cycle in flight. Must be at least broadcast_every * num_steps or the learner starves. See AsyncActorLearnerConfig::effective_max_lead_steps.

gamma: f32

Discount factor for GAE.

gae_lambda: f32

GAE lambda. Ignored when Self::use_vtrace is true (V-trace has no lambda parameter), but still validated to lie in (0, 1].

use_vtrace: bool

If true, replace GAE with V-trace (Espeholt et al. 2018) for off-policy correction of stale actor trajectories. The learner re-evaluates each stored (obs, action) under the current policy to obtain target log-probs log π(a_t|s_t), and the importance ratio against the actor’s stored behavior log-probs log μ(a_t|s_t) corrects for policy lag before PPO sees the advantages. Defaults to false, preserving the GAE path and all existing behavior.

vtrace_rho_bar: f32

V-trace rho clipping threshold (Espeholt 2018, eq. 1). Ignored when Self::use_vtrace is false. Typical value: 1.0 (the IMPALA paper default).

vtrace_c_bar: f32

V-trace c clipping threshold (Espeholt 2018, eq. 2). Ignored when Self::use_vtrace is false. Typical value: 1.0.

seed: u64

Base seed. Actor i samples actions with StdRng::seed_from_u64(seed + 1 + i) when wired through spawn_actor by the caller.

Implementations

impl AsyncActorLearnerConfig

pub fn validate(&self) -> Result<()>

Validate the configuration.

Errors

Returns an error when any count is zero, when the step budget is smaller than a single update’s worth of transitions, when gamma / gae_lambda fall outside (0, 1], or when the staleness valve is too tight for the broadcast cadence (which would deadlock the learner’s fill loop).

pub fn num_updates(&self) -> usize

Number of PPO updates the learner will run for this budget.

pub fn effective_max_lead_steps(&self) -> usize

Resolve the staleness valve: max_lead_steps when nonzero, otherwise 2 * broadcast_every * num_steps.

pub fn actor_throttle(&self) -> ActorThrottle

Build the per-actor ActorThrottle for this configuration.

pub fn use_vtrace(self, enabled: bool) -> Self

Enable or disable V-trace off-policy correction (builder style).

pub fn vtrace_rho_bar(self, rho_bar: f32) -> Self

Set the V-trace rho clip threshold (builder style).

pub fn vtrace_c_bar(self, c_bar: f32) -> Self

Set the V-trace c clip threshold (builder style).

Trait Implementations

impl Clone for AsyncActorLearnerConfig

fn clone(&self) -> AsyncActorLearnerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AsyncActorLearnerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AsyncActorLearnerConfig

fn default() -> Self

Returns the “default value” for a type.