Struct AsyncTrainer

pub struct AsyncTrainer<A, E, R>
where
    A: Agent<E, R> + Configurable + SyncModel,
    E: Env,
    R: ExperienceBufferBase + ReplayBufferBase,
    R::Item: Send + 'static,
{ /* private fields */ }

Manages asynchronous training loop in a single machine.

It interacts with ActorManager as shown below:

flowchart LR subgraph ActorManager E[Actor]-->|ReplayBufferBase::PushedItem|H[ReplayBufferProxy] F[Actor]-->H G[Actor]-->H end K-->|SyncModel::ModelInfo|E K-->|SyncModel::ModelInfo|F K-->|SyncModel::ModelInfo|G subgraph I[AsyncTrainer] H-->|PushedItemMessage|J[ReplayBuffer] J-->|ReplayBufferBase::Batch|K[Agent] end

• The Agent in AsyncTrainer (left) is trained with batches of type ReplayBufferBase::Batch, which are taken from the replay buffer.
• The model parameters of the Agent in AsyncTrainer are wrapped in SyncModel::ModelInfo and periodically sent to the Agents in Actors. Agent must implement SyncModel to synchronize the model parameters.
• In ActorManager (right), Actors sample transitions, which have type ReplayBufferBase::Item, and push the transitions into ReplayBufferProxy.
• ReplayBufferProxy has a type parameter of ReplayBufferBase and the proxy accepts ReplayBufferBase::Item.
• The proxy sends the transitions into the replay buffer in the AsyncTrainer.

Implementations

impl<A, E, R> AsyncTrainer<A, E, R>

where A: Agent<E, R> + Configurable + SyncModel + 'static, E: Env, R: ExperienceBufferBase + ReplayBufferBase, R::Item: Send + 'static,

pub fn build( config: &AsyncTrainerConfig, agent_config: &A::Config, env_config: &E::Config, replay_buffer_config: &R::Config, r_bulk_pushed_item: Receiver<PushedItemMessage<R::Item>>, model_info_sender: Sender<(usize, A::ModelInfo)>, stop: Arc<Mutex<bool>>, ) -> Self

Creates AsyncTrainer.

pub fn train<D>( &mut self, recorder: &mut Box<dyn Recorder<E, R>>, evaluator: &mut D, guard_init_env: Arc<Mutex<bool>>, ) -> AsyncTrainStat

where D: Evaluator<E>,

Runs training loop.

In the training loop, the following values will be pushed into the given recorder:

• samples_total - Total number of samples pushed into the replay buffer. Here, a “sample” is an item in ExperienceBufferBase::Item.
• opt_steps_per_sec - The number of optimization steps per second.
• samples_per_sec - The number of samples per second.
• samples_per_opt_steps - The number of samples per optimization step.

These values will typically be monitored with tensorboard.