Crate candle_optimisers 

Optimisers for use with the candle framework for lightweight machine learning. Apart from LBFGS, these all implement the candle_nn::optim::Optimizer trait from candle-nn

Example

Training an MNIST model using the Adam optimiser

    // create the Adam optimiser

    // set the learning rate to 0.004 and use the default parameters for everything else
    let params = ParamsAdam {
            lr: 0.004,
            ..Default::default()
        };
    // create the optimiser by passing in the variable to be optimised and the parameters
    let mut optimiser = Adam::new(varmap.all_vars(),  params)?;

    // loop for model optimisation
    for epoch in 0..100 {
        // run the model forwards
        // get log probabilities of results
        let logits = model.forward(&train_images)?;
        // softmax the log probabilities
        let log_sm = ops::log_softmax(&logits, D::Minus1)?;
        // get the loss
        let loss = loss::nll(&log_sm, &train_labels)?;
        // step the tensors by backpropagating the loss
        optimiser.backward_step(&loss)?;

    }
    Ok(())

Modules

adadelta

Adadelta optimiser

adagrad

Adagrad optimiser

adam

Adam optimiser (inlcuding AdamW)

adamax

Adamax optimiser

esgd

Stochastic Gradient Descent

lbfgs

Limited memory Broyden–Fletcher–Goldfarb–Shanno algorithm

nadam

NAdam optimiser: Adam with Nesterov momentum

radam

RAdam optimiser

rmsprop

RMS prop algorithm

Enums

Decay

Method of weight decay to use

ModelOutcome

Outcomes of an optimiser step for methods such as LBFGS

Momentum

Type of momentum to use

Traits

LossOptimizer

trait for optimisers like LBFGS that need the ability to calculate the loss and its gradient

Model

Trait for Models: this is needed for optimisers that require the ability to calculate the loss such as LBFGS

OptimParams

Trait for optimisers to expose their parameters