Struct MdpSolver 

pub struct MdpSolver {
    pub num_states: usize,
    pub num_actions: usize,
    pub discount: f64,
    pub rewards: Vec<Vec<f64>>,
    pub transitions: Vec<Vec<Vec<f64>>>,
}

Discrete-state, discrete-action MDP solver.

Implements value iteration and policy iteration for discounted MDPs.

Fields

num_states: usize

Number of states.

num_actions: usize

Number of actions.

discount: f64

Discount factor γ ∈ [0, 1).

rewards: Vec<Vec<f64>>

Reward matrix R[s][a].

transitions: Vec<Vec<Vec<f64>>>

Transition probabilities P[s][a][s’].

Implementations

impl MdpSolver

pub fn new( num_states: usize, num_actions: usize, discount: f64, rewards: Vec<Vec<f64>>, transitions: Vec<Vec<Vec<f64>>>, ) -> Self

Create a new MDP solver.

pub fn bellman_update(&self, v: &[f64]) -> Vec<f64>

Apply the Bellman operator once: T V(s) = max_a [R(s,a) + γ Σ P(s’|s,a) V(s’)]

pub fn value_iteration(&self, tol: f64, max_iter: usize) -> (Vec<f64>, usize)

Run value iteration until convergence.

Returns (V*, iterations) where V*[s] is the optimal value of state s.

pub fn extract_policy(&self, v: &[f64]) -> Vec<usize>

Extract greedy policy from value function V.

Returns policy[s] = argmax_a [R(s,a) + γ Σ P(s’|s,a) V(s’)].

pub fn policy_evaluation( &self, policy: &[usize], tol: f64, max_iter: usize, ) -> Vec<f64>

Evaluate a fixed policy: V^π(s) = R(s,π(s)) + γ Σ P(s’|s,π(s)) V^π(s’)

Solved by iteration (similar to value iteration for fixed policy).

pub fn policy_iteration( &self, tol: f64, max_iter: usize, ) -> (Vec<usize>, Vec<f64>, usize)

Policy iteration: alternate between evaluation and improvement.

Returns (optimal_policy, optimal_value, iterations).