Struct ReinforcementLearner 

pub struct ReinforcementLearner {
    pub algorithm: RlAlgorithm,
    pub q_table: HashMap<StateId, Vec<f64>>,
    pub q_table2: HashMap<StateId, Vec<f64>>,
    pub n_actions: usize,
    pub total_steps: u64,
    pub total_episodes: u64,
    pub episode_returns: VecDeque<f64>,
    pub rng_state: u64,
}

Tabular reinforcement-learning agent supporting Q-learning, SARSA, and Double Q-learning.

Q-values are stored in a HashMap and initialised lazily to 0.0 on first access, so memory consumption grows only with the number of visited states.

Fields

algorithm: RlAlgorithm

The learning algorithm and its hyper-parameters.

q_table: HashMap<StateId, Vec<f64>>

Primary Q-table: q_table[s][a] = Q(s, a).

q_table2: HashMap<StateId, Vec<f64>>

Secondary Q-table used only by Double Q-learning.

n_actions: usize

Number of discrete actions.

total_steps: u64

Total environment steps executed via update.

total_episodes: u64

Total episodes recorded via start_episode / end_episode.

episode_returns: VecDeque<f64>

Sliding window of episode returns (capped at 1 000 entries).

rng_state: u64

Current xorshift64 PRNG state (always non-zero).

Implementations

impl ReinforcementLearner

pub fn new(algorithm: RlAlgorithm, n_actions: usize, seed: u64) -> Self

Create a new learner with the given algorithm, action-space size, and random seed.

A seed of 0 is promoted to 1 to satisfy the xorshift64 requirement.

pub fn select_action(&mut self, state: StateId, policy: &Policy) -> ActionId

Select an action for state according to policy.

• Policy::EpsilonGreedy — random with prob ε, else argmax Q(s, ·).
• Policy::Greedy — always argmax Q(s, ·).
• Policy::Random — uniformly random over the action space.

pub fn update(&mut self, experience: &Experience) -> f64

Apply one temporal-difference update from experience; return the TD error δ.

Also increments total_steps.

pub fn batch_update(&mut self, experiences: &[Experience]) -> Vec<f64>

Apply TD updates for a slice of experiences in order; return the corresponding TD errors.

pub fn best_action(&self, state: StateId) -> ActionId

Return the action with the highest Q-value in the primary table. Falls back to ActionId(0) when the state has never been visited.

pub fn q_value(&self, state: StateId, action: ActionId) -> f64

Return Q(s, a) from the primary table; 0.0 if the pair is unseen.

pub fn value(&self, state: StateId) -> f64

Return V(s) = max_a Q(s, a) from the primary table; 0.0 if unseen.

pub fn explored_states(&self) -> usize

Number of distinct states whose Q-values have been initialised.

pub fn start_episode(&mut self)

Signal the start of a new episode.

Pushes a placeholder return of 0.0 and increments total_episodes.

pub fn end_episode(&mut self, total_return: f64)

Signal the end of the current episode, recording its total return.

Updates the last entry in episode_returns and caps the deque at 1 000 entries (oldest entries are dropped).

pub fn avg_return(&self, last_n: usize) -> f64

Return the mean episode return over the last last_n episodes. Returns 0.0 if no episodes have been recorded.

pub fn stats(&self) -> RlStats

Return a snapshot of the current learning statistics.

Trait Implementations

impl Clone for ReinforcementLearner

fn clone(&self) -> ReinforcementLearner

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ReinforcementLearner

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.