Struct MDP 

pub struct MDP {
    pub num_states: usize,
    pub num_actions: usize,
    pub transitions: Vec<Vec<Vec<f64>>>,
    pub rewards: Vec<Vec<f64>>,
    pub discount: f64,
}

A finite Markov Decision Process (S, A, P, R, γ).

• num_states: |S|
• num_actions: |A|
• transitions[s][a][s']: P(s’ | s, a)
• rewards[s][a]: R(s, a)
• discount: γ ∈ [0,1)

Fields

num_states: usize

Number of states.

num_actions: usize

Number of actions.

transitions: Vec<Vec<Vec<f64>>>

Transition probabilities: transitions[s][a] is a probability vector over next states.

rewards: Vec<Vec<f64>>

Expected reward: rewards[s][a].

discount: f64

Discount factor γ ∈ [0,1).

Implementations

impl MDP

pub fn new( num_states: usize, num_actions: usize, transitions: Vec<Vec<Vec<f64>>>, rewards: Vec<Vec<f64>>, discount: f64, ) -> Self

Construct a new MDP.

pub fn bellman_operator(&self, v: &[f64]) -> Vec<f64>

Apply the Bellman operator: (TV)(s) = max_a [R(s,a) + γ Σ P(s’|s,a) V(s’)].

pub fn value_iteration(&self, tol: f64, max_iter: usize) -> Vec<f64>

Value iteration: iterate the Bellman operator until convergence.

pub fn policy_improvement(&self, v: &[f64]) -> Vec<usize>

Extract the greedy policy from a value function.

pub fn policy_evaluation( &self, policy: &[usize], tol: f64, max_iter: usize, ) -> Vec<f64>

Policy evaluation: compute V^π for a deterministic policy using iterative updates.

Trait Implementations

impl Clone for MDP

fn clone(&self) -> MDP

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MDP

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.