Struct border_core::core::trainer::Trainer

pub struct Trainer<E: Env, A: Agent<E>> { /* fields omitted */ }

Manages training process.

Training loop

For training an agent with standard RL algorithms in the library, the agent and environment interact as illustrated in the following diagram:

flowchart TB Trainer -. 0. Env::reset .-> Env Env --> Obs ObsPrev -- 3. Policy::sample --> Policy Policy --> Act Act -- 4. Env::step --> Env Obs --> Step Obs -- 1. RefCell::replace --> ObsPrev Act --> Step ObsPrev -- 2. Agent::push_obs --> ObsPrev' Step -- 5. Agent::observe --> Transition subgraph Agent ObsPrev' --> Transition ReplayBuffer -- 6. update policy parameters --- Policy Transition --> ReplayBuffer end

0. Call Env::reset for resetting the enbironment and getting an observation. An episode starts.
1. Call std::cell::RefCell::replace for placing the observation in PrevObs.
2. Call Agent::push_obs for placing the observation in PrevObs'.
3. Call Policy::sample for sampling an action from Policy.
4. Call Env::step for taking an action, getting a new observation, and creating Step object.
5. Call Agent::observe for updating the replay buffer with the new and previous observations.
6. Call some methods in the agent for updating policy parameters.
7. Back to 1.

Actually, Trainer is not responsible for the step 6. The Agent does it.

Model evaluation and saving

Trainer::train() evaluates the agent being trained with the interval of optimization steps specified by TrainerBuilder::eval_interval(). If the evaluation reward is greater than the maximum in the history of training, the agent will be saved in the directory specified by TrainerBuilder::model_dir().

A trained agent often consists of a number of neural networks like an action-value network, its target network, a policy network. Typically, Agent saves all of these neural networks in a directory.

Implementations

impl<E: Env, A: Agent<E>> Trainer<E, A>

pub fn get_agent(&self) -> &impl Agent<E>

Get the reference to the agent.

pub fn get_env(&self) -> &E

Get the reference to the environment.

pub fn get_env_eval(&self) -> &E

Get the reference to the environment for evaluation.

pub fn train<T: Recorder>(&mut self, recorder: &mut T)

Train the agent.

In the training loop, the following values are recorded in the recorder:

• n_steps - The nunber of steps interacting with the environment.
• n_opts - The number of optimization steps.
• datetime - Date and time.
• mean_cum_eval_reward - Cumulative rewards in evaluation runs.