Struct border_async_trainer::AsyncTrainer

pub struct AsyncTrainer<A, E, R>where
    A: Agent<E, R> + SyncModel,
    E: Env,
    R: ReplayBufferBase,
    R::PushedItem: Send + 'static,{ /* private fields */ }

Manages asynchronous training loop in a single machine.

It interacts with ActorManager as shown below:

flowchart LR subgraph ActorManager E[Actor]-->|ReplayBufferBase::PushedItem|H[ReplayBufferProxy] F[Actor]-->H G[Actor]-->H end K-->|SyncModel::ModelInfo|E K-->|SyncModel::ModelInfo|F K-->|SyncModel::ModelInfo|G subgraph I[AsyncTrainer] H-->|PushedItemMessage|J[ReplayBuffer] J-->|ReplayBufferBase::Batch|K[Agent] end

• In ActorManager (right), Actors sample transitions, which have type ReplayBufferBase::PushedItem, in parallel and push the transitions into ReplayBufferProxy. It should be noted that ReplayBufferProxy has a type parameter of ReplayBufferBase and the proxy accepts ReplayBufferBase::PushedItem.
• The proxy sends the transitions into the replay buffer, implementing ReplayBufferBase, in the AsyncTrainer.
• The Agent in AsyncTrainer trains its model parameters by using batches of type ReplayBufferBase::Batch, which are taken from the replay buffer.
• The model parameters of the Agent in AsyncTrainer are wrapped in SyncModel::ModelInfo and periodically sent to the Agents in Actors. Agent must implement SyncModel to synchronize its model.

Implementations

impl<A, E, R> AsyncTrainer<A, E, R>where A: Agent<E, R> + SyncModel, E: Env, R: ReplayBufferBase, R::PushedItem: Send + 'static,

pub fn build( config: &AsyncTrainerConfig, agent_config: &A::Config, env_config: &E::Config, replay_buffer_config: &R::Config, r_bulk_pushed_item: Receiver<PushedItemMessage<R::PushedItem>>, model_info_sender: Sender<(usize, A::ModelInfo)>, stop: Arc<Mutex<bool>> ) -> Self

Creates AsyncTrainer.

pub fn train<D>( &mut self, recorder: &mut impl Recorder, evaluator: &mut D, guard_init_env: Arc<Mutex<bool>> ) -> AsyncTrainStatwhere D: Evaluator<E, A>,

Runs training loop.

In the training loop, the following values will be pushed into the given recorder:

• samples_total - Total number of samples pushed into the replay buffer. Here, a “sample” is an item in ExperienceBufferBase::PushedItem.
• opt_steps_per_sec - The number of optimization steps per second.
• samples_per_sec - The number of samples per second.
• samples_per_opt_steps - The number of samples per optimization step.

These values will typically be monitored with tensorboard.