Struct ADMM 

pub struct ADMM { /* private fields */ }

ADMM (Alternating Direction Method of Multipliers) for distributed and constrained optimization.

Solves problems of the form:

minimize  f(x) + g(z)
subject to Ax + Bz = c

Applications

• Distributed Lasso: Split data across workers for large-scale regression
• Consensus optimization: Average models from different sites (federated learning)
• Constrained problems: Equality-constrained optimization via consensus
• Model parallelism: Parallelize training across devices

Algorithm

ADMM alternates between three updates:

1. x-update: x^{k+1} = argmin_x { f(x) + (ρ/2)‖Ax + Bz^k - c + u^k‖² }
2. z-update: z^{k+1} = argmin_z { g(z) + (ρ/2)‖Ax^{k+1} + Bz - c + u^k‖² }
3. u-update: u^{k+1} = u^k + (Ax^{k+1} + Bz^{k+1} - c)

where u is the scaled dual variable and ρ is the penalty parameter.

Convergence

• Rate: O(1/k) for convex f and g
• Stopping criteria: Both primal and dual residuals below tolerance
• Adaptive ρ: Automatically adjusts penalty parameter for faster convergence

Example: Consensus Form (Lasso)

For Lasso regression with consensus constraint x = z:

use aprender::optim::ADMM;
use aprender::primitives::{Vector, Matrix};

let n = 5;
let m = 10;

// Create problem data
let A = Matrix::eye(n); // Identity for consensus
let B = Matrix::eye(n);
let c = Vector::zeros(n);

// x-minimizer: least squares update
let data_matrix = Matrix::eye(m); // Your data matrix
let observations = Vector::ones(m); // Your observations
let lambda = 0.1;

let x_minimizer = |z: &Vector<f32>, u: &Vector<f32>, _c: &Vector<f32>, rho: f32| {
    // Minimize ½‖Dx - b‖² + (ρ/2)‖x - z + u‖²
    // Closed form: x = (DᵀD + ρI)⁻¹(Dᵀb + ρ(z - u))
    let mut rhs = Vector::zeros(n);
    for i in 0..n {
        rhs[i] = rho * (z[i] - u[i]);
    }
    rhs // Simplified for example
};

// z-minimizer: soft-thresholding (proximal operator for L1)
let z_minimizer = |ax: &Vector<f32>, u: &Vector<f32>, _c: &Vector<f32>, rho: f32| {
    let mut z = Vector::zeros(n);
    for i in 0..n {
        let v = ax[i] + u[i];
        let threshold = lambda / rho;
        z[i] = if v > threshold {
            v - threshold
        } else if v < -threshold {
            v + threshold
        } else {
            0.0
        };
    }
    z
};

let mut admm = ADMM::new(100, 1.0, 1e-4).with_adaptive_rho(true);
let x0 = Vector::zeros(n);
let z0 = Vector::zeros(n);

let result = admm.minimize_consensus(
    x_minimizer,
    z_minimizer,
    &A,
    &B,
    &c,
    x0,
    z0,
);

Reference

Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011). “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers” Foundations and Trends in Machine Learning, 3(1), 1-122.

Implementations

impl ADMM

pub fn new(max_iter: usize, rho: f32, tol: f32) -> Self

Creates a new ADMM optimizer.

Parameters

• max_iter: Maximum number of iterations (typical: 100-1000)
• rho: Penalty parameter (typical: 0.1-10.0, problem-dependent)
• tol: Convergence tolerance for residuals (typical: 1e-4 to 1e-6)

Returns

A new ADMM optimizer with default settings:

• Adaptive rho: disabled (use with_adaptive_rho(true) to enable)
• Rho increase factor: 2.0
• Rho decrease factor: 2.0

Example

use aprender::optim::ADMM;

let admm = ADMM::new(100, 1.0, 1e-4);

pub fn with_adaptive_rho(self, adaptive: bool) -> Self

Enables or disables adaptive penalty parameter adjustment.

When enabled, rho is automatically adjusted based on the ratio of primal to dual residuals:

• If primal residual >> dual residual: increase rho (enforce constraints more)
• If dual residual >> primal residual: decrease rho (improve objective)

Example

use aprender::optim::ADMM;

let admm = ADMM::new(100, 1.0, 1e-4).with_adaptive_rho(true);

pub fn with_rho_factors(self, increase: f32, decrease: f32) -> Self

Sets the factors for adaptive rho adjustment.

Parameters

• increase: Factor to multiply rho when primal residual is large (default: 2.0)
• decrease: Factor to divide rho when dual residual is large (default: 2.0)

pub fn minimize_consensus<F, G>( &mut self, x_minimizer: F, z_minimizer: G, a: &Matrix<f32>, b_mat: &Matrix<f32>, c: &Vector<f32>, x0: Vector<f32>, z0: Vector<f32>, ) -> OptimizationResult

where F: Fn(&Vector<f32>, &Vector<f32>, &Vector<f32>, f32) -> Vector<f32>, G: Fn(&Vector<f32>, &Vector<f32>, &Vector<f32>, f32) -> Vector<f32>,

Minimizes a consensus-form ADMM problem.

Solves: minimize f(x) + g(z) subject to Ax + Bz = c

Parameters

• x_minimizer: Function that solves x-subproblem given (z, u, c, rho)
• z_minimizer: Function that solves z-subproblem given (Ax, u, c, rho)
• A, B, c: Constraint matrices and vector (Ax + Bz = c)
• x0, z0: Initial values for x and z

Returns

OptimizationResult containing the optimal x value and convergence information.

Minimizer Functions

The x_minimizer should solve:

argmin_x { f(x) + (ρ/2)‖Ax + Bz - c + u‖² }

The z_minimizer should solve:

argmin_z { g(z) + (ρ/2)‖Ax + Bz - c + u‖² }

These often have closed-form solutions or can use proximal operators.

Trait Implementations

impl Clone for ADMM

fn clone(&self) -> ADMM

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ADMM

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Optimizer for ADMM

fn step(&mut self, _params: &mut Vector<f32>, _gradients: &Vector<f32>)

Stochastic update (mini-batch mode) - for SGD, Adam, RMSprop.

fn reset(&mut self)

Resets the optimizer state (momentum, history, etc.).

fn minimize<F, G>( &mut self, _objective: F, _gradient: G, _x0: Vector<f32>, ) -> OptimizationResult

where F: Fn(&Vector<f32>) -> f32, G: Fn(&Vector<f32>) -> Vector<f32>,

Batch optimization (deterministic mode) - for L-BFGS, CG, Damped Newton.