Trait PolicyNetwork 

pub trait PolicyNetwork<T: Float + Debug + Send + Sync + 'static> {
    // Required methods
    fn evaluate_actions(
        &self,
        observations: &Array2<T>,
        actions: &Array2<T>,
    ) -> Result<PolicyEvaluation<T>>;
    fn get_action_distribution(
        &self,
        observations: &Array2<T>,
    ) -> Result<ActionDistribution<T>>;
    fn update_parameters(
        &mut self,
        deltas: &HashMap<String, Array1<T>>,
    ) -> Result<()>;
    fn get_parameters(&self) -> HashMap<String, Array1<T>>;

    // Provided methods
    fn log_prob_gradient(
        &self,
        observations: &Array2<T>,
        actions: &Array2<T>,
        coefficients: &Array1<T>,
    ) -> Result<HashMap<String, Array1<T>>> { ... }
    fn entropy_gradient(
        &self,
        observations: &Array2<T>,
    ) -> Result<HashMap<String, Array1<T>>> { ... }
    fn mean_action_gradient(
        &self,
        observations: &Array2<T>,
        weights: &Array2<T>,
    ) -> Result<HashMap<String, Array1<T>>> { ... }
    fn score_matrix(
        &self,
        observations: &Array2<T>,
        actions: &Array2<T>,
    ) -> Result<Array2<T>> { ... }
    fn kronecker_factors(
        &self,
        observations: &Array2<T>,
        actions: &Array2<T>,
    ) -> Result<Vec<KroneckerBlock<T>>> { ... }
}

Policy network interface for RL optimizers.

Parameter update contract

PolicyNetwork::update_parameters receives a parameter delta, not a raw gradient: the optimizer has already applied the learning rate, the gradient clipping and the sign (descent on the loss). Implementations must therefore add the supplied arrays to their parameters. Every optimizer in this module (policy gradient, trust region, natural gradient, target-network soft updates) relies on this additive semantics.

Gradient oracle

The *_gradient methods form the differentiable path used by every learning rule here. They have no meaningful default, so the default bodies return OptimError::UnsupportedOperation — a policy that cannot differentiate itself must fail loudly rather than be “trained” with a fabricated gradient. linear_models provides ready-made analytic implementations.

Required Methods

fn evaluate_actions( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<PolicyEvaluation<T>>

Evaluate actions for given observations

fn get_action_distribution( &self, observations: &Array2<T>, ) -> Result<ActionDistribution<T>>

Get action distribution for given observations

fn update_parameters( &mut self, deltas: &HashMap<String, Array1<T>>, ) -> Result<()>

Add a parameter delta to the policy parameters (see the trait docs).

fn get_parameters(&self) -> HashMap<String, Array1<T>>

Get current policy parameters

Provided Methods

fn log_prob_gradient( &self, observations: &Array2<T>, actions: &Array2<T>, coefficients: &Array1<T>, ) -> Result<HashMap<String, Array1<T>>>

Gradient of a coefficient-weighted sum of log-probabilities: ∂/∂θ Σᵢ cᵢ · log π(aᵢ | sᵢ).

Every surrogate loss implemented in this module — REINFORCE, A2C/A3C, PPO-clip, PPO adaptive-KL, V-trace/IMPALA — has a policy gradient of exactly this shape with cᵢ = ∂L/∂ log π(aᵢ|sᵢ), so this single oracle is enough to train all of them end to end.

The returned map must have the same keys and lengths as Self::get_parameters.

fn entropy_gradient( &self, observations: &Array2<T>, ) -> Result<HashMap<String, Array1<T>>>

Gradient of the batch-mean entropy ∂/∂θ (1/N) Σᵢ H[π(·|sᵢ)].

Only consulted when the entropy coefficient is non-zero.

fn mean_action_gradient( &self, observations: &Array2<T>, weights: &Array2<T>, ) -> Result<HashMap<String, Array1<T>>>

Gradient of a weighted sum of the distribution mean: ∂/∂θ Σᵢ Σⱼ w[i,j] · μⱼ(sᵢ).

This is the chain-rule hook required by the deterministic policy gradient (DDPG/TD3) and by the reparameterized SAC actor update, where the loss depends on the parameters through the sampled action rather than through the log-probability.

fn score_matrix( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<Array2<T>>

Per-sample score vectors g_i = ∇_θ log π(aᵢ|sᵢ), one per row, flattened with flatten_named’s canonical layout.

Used to build empirical / block-diagonal Fisher estimates. The default implementation derives them from Self::log_prob_gradient one sample at a time, which is correct but costs N oracle calls; policies that can produce them in one pass should override it.

fn kronecker_factors( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<Vec<KroneckerBlock<T>>>

Per-sample Kronecker factors of the Fisher information matrix.

See KroneckerBlock for the exact contract. Returning OptimError::UnsupportedOperation (the default) makes K-FAC estimation fail loudly instead of silently degrading to an identity Fisher.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl<T: Float + Debug + Send + Sync + 'static, P: PolicyNetwork<T> + ?Sized> PolicyNetwork<T> for &mut P

Blanket forwarding so a &mut P can stand in for an owned policy.

This lets an optimizer that already owns a policy hand a borrow of it to another optimizer (e.g. policy_gradient::PolicyGradientOptimizer routing its TRPO update through trust_region::TrustRegionOptimizer) without transferring ownership. Every method — including the gradient oracle — is forwarded, so the borrow behaves exactly like the underlying policy.

fn evaluate_actions( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<PolicyEvaluation<T>>

fn get_action_distribution( &self, observations: &Array2<T>, ) -> Result<ActionDistribution<T>>

fn update_parameters( &mut self, deltas: &HashMap<String, Array1<T>>, ) -> Result<()>

fn get_parameters(&self) -> HashMap<String, Array1<T>>

fn log_prob_gradient( &self, observations: &Array2<T>, actions: &Array2<T>, coefficients: &Array1<T>, ) -> Result<HashMap<String, Array1<T>>>

fn entropy_gradient( &self, observations: &Array2<T>, ) -> Result<HashMap<String, Array1<T>>>

fn mean_action_gradient( &self, observations: &Array2<T>, weights: &Array2<T>, ) -> Result<HashMap<String, Array1<T>>>

fn score_matrix( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<Array2<T>>

fn kronecker_factors( &self, observations: &Array2<T>, actions: &Array2<T>, ) -> Result<Vec<KroneckerBlock<T>>>

Implementors

impl<T: Float + Debug + Send + Sync + 'static> PolicyNetwork<T> for LinearGaussianPolicy<T>

impl<T: Float + Debug + Send + Sync + 'static> PolicyNetwork<T> for LinearSoftmaxPolicy<T>