Struct LBFGS 

pub struct LBFGS<A: Float + ScalarOperand + Debug> { /* private fields */ }

L-BFGS optimizer

Implements the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm. This is a quasi-Newton method that approximates the Hessian inverse using a limited amount of memory by storing only a few vectors from previous iterations.

Curvature pairs

The optimizer stores the previous parameters and the previous gradient, so the curvature pair is the true s = x_k - x_{k-1}, y = g_k - g_{k-1} even when the caller post-processes the returned parameters (projection, clipping, weight decay, a different step size, …). Pairs with y·s <= 0 are skipped so the implicit inverse-Hessian stays positive definite.

Step size and line search

Optimizer::step applies a fixed step size (the learning rate) along the two-loop direction: it has no access to the objective, so it cannot run a line search. Use LBFGS::step_with_loss to get a backtracking Armijo line search that uses the configured c1 and max_ls parameters.

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::{LBFGS, Optimizer};

// Initialize parameters and gradients
let params = Array1::zeros(5);
let gradients = Array1::from_vec(vec![0.1, 0.2, -0.3, 0.0, 0.5]);

// Create an L-BFGS optimizer
let mut optimizer = LBFGS::new(1.0);

// Update parameters
let new_params = optimizer.step(&params, &gradients).expect("optimizer.step succeeds");

Implementations

impl<A: Float + ScalarOperand + Debug + Send + Sync> LBFGS<A>

pub fn new(learning_rate: A) -> Self

Creates a new L-BFGS optimizer with the given learning rate

Arguments

• learning_rate - The learning rate for parameter updates

pub fn new_with_config( learning_rate: A, history_size: usize, tolerance_grad: A, c1: A, c2: A, max_ls: usize, ) -> Self

Creates a new L-BFGS optimizer with full configuration

Arguments

• learning_rate - The learning rate for parameter updates
• history_size - Number of past gradients/steps to store (default: 100)
• tolerance_grad - Gradient norm tolerance for convergence (default: 1e-7)
• c1 - Wolfe line search parameter for Armijo condition (default: 1e-4)
• c2 - Wolfe line search parameter for curvature condition (default: 0.9)
• max_ls - Maximum line search iterations (default: 25)

pub fn learning_rate(&self) -> A

Gets the current learning rate

pub fn set_lr(&mut self, lr: A)

Sets the learning rate

pub fn c1(&self) -> A

Armijo (sufficient-decrease) line search parameter c1

pub fn c2(&self) -> A

Wolfe curvature line search parameter c2

pub fn max_ls(&self) -> usize

Maximum number of line search iterations

pub fn history_len(&self) -> usize

Number of curvature pairs currently stored

pub fn last_curvature_pair(&self) -> Option<(&Array1<A>, &Array1<A>)>

The most recently stored curvature pair (s, y), if any.

s is the true parameter difference x_k - x_{k-1} (as observed across two consecutive calls) and y is the corresponding gradient difference g_k - g_{k-1}.

pub fn initial_hessian_scale(&self) -> A

The current initial inverse-Hessian scaling gamma_k = (s·y) / (y·y).

pub fn set_line_search_contraction(&mut self, rho: A) -> Result<()>

Sets the backtracking contraction factor used by LBFGS::step_with_loss.

Must lie strictly between 0 and 1; other values are rejected.

pub fn reset(&mut self)

Resets the internal state of the optimizer

pub fn step_with_loss<D, F>( &mut self, params: &Array<A, D>, gradients: &Array<A, D>, loss_fn: F, ) -> Result<Array<A, D>>

where D: Dimension, F: FnMut(&Array<A, D>) -> A,

Performs an L-BFGS step with a backtracking Armijo line search.

Unlike Optimizer::step, which has no access to the objective and therefore applies a fixed step size, this method evaluates loss_fn at trial points and accepts the first step size satisfying the Armijo sufficient-decrease condition

f(x + alpha * d) <= f(x) + c1 * alpha * g^T d

starting from alpha = learning_rate and contracting by the line search contraction factor (default 0.5) for at most max_ls iterations. If no trial step satisfies the condition, the trial with the lowest objective value is used when it improves on f(x); otherwise the parameters are returned unchanged.

If the two-loop direction is not a descent direction (which can only happen through numerical error, since non-positive curvature pairs are never stored), the search falls back to steepest descent for this step.

Errors

Returns OptimError::DimensionMismatch if params and gradients have different shapes, and OptimError::InvalidConfig if the objective is not finite at the current parameters.

Examples

use scirs2_core::ndarray::Array1;
use optirs_core::optimizers::LBFGS;

let mut optimizer = LBFGS::new(1.0);
let mut params = Array1::from_vec(vec![2.0_f64, -3.0]);
let loss = |x: &Array1<f64>| x.iter().map(|v| v * v).sum::<f64>();

for _ in 0..30 {
    let grads = params.mapv(|v| 2.0 * v);
    params = optimizer
        .step_with_loss(&params, &grads, loss)
        .expect("step succeeds");
}
assert!(params.iter().all(|v| v.abs() < 1e-6));

Trait Implementations

impl<A: Clone + Float + ScalarOperand + Debug> Clone for LBFGS<A>

fn clone(&self) -> LBFGS<A>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<A: Debug + Float + ScalarOperand + Debug> Debug for LBFGS<A>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A, D> Optimizer<A, D> for LBFGS<A>

where A: Float + ScalarOperand + Debug + Send + Sync, D: Dimension,

fn step( &mut self, params: &Array<A, D>, gradients: &Array<A, D>, ) -> Result<Array<A, D>>

Performs an L-BFGS step with a fixed step size.

This trait method has no access to the objective, so no line search is possible; the two-loop direction is scaled by the learning rate (reduced by 1 / (1 + ||g||) on the very first step, before any curvature information exists). Use LBFGS::step_with_loss for the backtracking Armijo line search that uses the configured c1 and max_ls.

fn get_learning_rate(&self) -> A

Gets the current learning rate

fn set_learning_rate(&mut self, learning_rate: A)

Sets a new learning rate

fn step_list( &mut self, params_list: &[&Array<A, D>], gradients_list: &[&Array<A, D>], ) -> Result<Vec<Array<A, D>>>

Updates multiple parameter arrays at once