Crate numdiff 

  

Automatic and numerical differentiation.

Overview

This crate implements two different methods for evaluating derivatives in Rust:

1. Automatic differentiation (forward-mode using first-order dual numbers).
2. Numerical differentiation (using forward difference and central difference approximations).

This crate provides generic functions (for numerical differentiation) and macros (for automatic differentiation) to evaluate various types of derivatives of the following types of functions:

• Univariate, scalar-valued functions ($f:\mathbb{R}\to\mathbb{R}$)
• Univariate, vector-valued functions ($\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$)
• Multivariate, scalar-valued functions ($f:\mathbb{R}^{n}\to\mathbb{R}$)
• Multivariate, vector-valued functions ($\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$)

These functions and macros are made generic over the choice of vector representation, as long as the vector type implements the linalg_traits::Vector trait. See the linalg_traits documentation for more information.

Automatic Differentiation (Forward-Mode)

This crate provides forward-mode automatic differentiation of functions that are generic over their vector and scalar types.

Typing R: linalg_traits::RealField allows the automatic differentiation macros to substitute this crate’s Dual and HyperDual types, which allow propagating 1st and 2nd-order derivative information through the computation. The resulting way functions have to be written differs mainly from a more “standard” numerical computing by requiring concrete types such as f64 and SVector<f64> to be written in terms of generic types such as R: linalg_traits::RealField and V: linalg_traits::Vector<R>. It is, however, a close approximation to the standard Rust numeric style: the linalg_traits::RealField requires a superset of the functionality provided by common traits such as num_traits::Float and nalgebra::RealField, while additionally requiring a high degree of interoperability with f64. In that sense, this crate aims to stay near the ergonomics of normal Rust code without forcing a custom AD type at every call site.

A major benefit of this approach is that the same function can be evaluated with an f64, allowing it to be used both in ordinary numerical computations and in automatic differentiation contexts. As an example, consider a function operating with conrete f64’s:

fn f_f64(x: f64) -> f64 {
    x * x.sin() + 1.0
}

To write this function in a generic way suitable for automatic differentiation:

use linalg_traits::RealField;

fn f_generic<R: RealField>(x: R) -> R {
    x * x.sin() + 1.0
}

f_generic can be called with an f64 just like f_f64, but it can also be called with a Dual or HyperDual type, which is the special sauce that allows this function to be automatically differentiated with one of the derivative macros provided by this crate.

1st-Order Derivatives

Derivative Type	Function Type	Macro to Generate Derivative Function
derivative	$f:\mathbb{R}\to\mathbb{R}$	get_sderivative!
derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	get_vderivative!
partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_spartial_derivative!
partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	get_vpartial_derivative!
gradient	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_gradient!
directional derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_directional_derivative!
Jacobian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	get_jacobian!

2nd-Order Derivatives

Derivative Type	Function Type	Macro to Generate Derivative Function
2nd derivative	$f:\mathbb{R}\to\mathbb{R}$	get_sderivative2!
2nd derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	get_vderivative2!
2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_spartial_derivative2!
2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	get_vpartial_derivative2!
mixed 2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_mixed_spartial_derivative2!
mixed 2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	get_mixed_vpartial_derivative2!
Hessian	$f:\mathbb{R}^{n}\to\mathbb{R}$	get_shessian!
Hessian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	get_vhessian!

Passing Runtime Parameters

Many times, we want to automatically differentiate functions that can also depend on parameters defined at runtime. However, automatic differentiation is performed at compile time, so we cannot simply “capture” these parameters using closures. To solve this problem, all automatic differentiation macros expect the function being differentiated to not only accept the point at which it is differentiated, but also a runtime parameter of an arbitrary type (could be a &[f64], could be a &T where T is some custom struct, etc.).

Examples are included for each macro (for example, here is the example for the get_jacobian! macro): get_jacobian! - Example Passing Custom Parameter Types

Limitations

• These macros only work on functions that are generic both over the type of scalar and the type of vector.
  • Consequently, these macros do not work on closures.
• Occasionally, constants (e.g. 5.0_f64) need to be defined using R::from (e.g. if a function has the generic parameter R: RealField, then instead of defining a constant number such as 5.0_f64, we need to do 5.0).
  • However, this is not always required, for example if you are doing something like 5.0 * x where x: R and R: RealField, then you don’t have to do 5.0 * x because RealField requires that a type is interoperable with f64.
• We cannot do assignment operations on f64 where right hand side is an x: R where R: RealField (e.g. let y = 1.0; y += x;).
  • In cases like this you should do let y = 1.0; y += x;.

Alternatives

The Rust ecosystem has many different crates and approaches for automatic differentiation. Here we summarize some of the alternatives.

Standard Library

The most prominent alternative nowadays is the standard library’s std::autodiff module which is in nightly Rust.

This module provides low-level AD capabilities, but is not yet available in stable Rust. If it were, this crate perhaps would have been written as a convenience wrapper around that lower-level API.

Forward-mode implementations

These crates are broadly similar in computational model to this one, but they tend to require a custom numeric type or library-specific expression style rather than a more generic, backend-agnostic (i.e. not tied to a specific linear algebra library such as nalgebra).

• num-dual — similar to this approach, but uses a DualNum trait instead of a more generic-sounding RealField, and is only compatible with nalgebra types.
• autodiff — a solid forward-mode AD crate, but centered on custom types such as FT<T> instead of a generic scalar interface.
• autodj — works through custom AD types, but writing autodifferentiable mathematical functions is more cumbersome, especially for multivariate cases.
• fwd_ad — another crate using forward-mode AD, but requires significant changes to make functions autodifferentiable.
• kophy/autodiff — only provides a very minimal Dual number implementation, and is unpublished.

Reverse-mode implementations

All three of these libraries implement reverse-mode AD, but require users to use a significantly different syntax when writing autodifferentiable functions:

• gad
• reverse
• revad
• rustydiff
• rust-autograd

Note that the first four crates listed here also have not been maintained for 3+ years (some much longer), with at least one of them (gad) having already been archived.

Finite Difference Methods

Central Difference Approximations

First-Order Derivatives

Derivative Type	Function Type	Function to Approximate Derivative
derivative	$f:\mathbb{R}\to\mathbb{R}$	central_difference::sderivative()
derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	central_difference::vderivative()
partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::spartial_derivative()
partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	central_difference::vpartial_derivative()
gradient	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::gradient()
directional derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::directional_derivative()
Jacobian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	central_difference::jacobian()

Second-Order Derivatives

Derivative Type	Function Type	Function to Approximate Derivative
Hessian	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::shessian()
Hessian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	central_difference::vhessian()
2nd derivative	$f:\mathbb{R}\to\mathbb{R}$	central_difference::sderivative2()
2nd derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	central_difference::vderivative2()
2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::spartial_derivative2()
2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	central_difference::vpartial_derivative2()
mixed 2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	central_difference::mixed_spartial_derivative2()
mixed 2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	central_difference::mixed_vpartial_derivative2()

Forward Difference Approximations

First-Order Derivatives

Derivative Type	Function Type	Function to Approximate Derivative
derivative	$f:\mathbb{R}\to\mathbb{R}$	forward_difference::sderivative()
derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	forward_difference::vderivative()
partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::spartial_derivative()
partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	forward_difference::vpartial_derivative()
gradient	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::gradient()
directional derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::directional_derivative()
Jacobian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	forward_difference::jacobian()

Second-Order Derivatives

Derivative Type	Function Type	Function to Approximate Derivative
Hessian	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::shessian()
Hessian	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	forward_difference::vhessian()
2nd derivative	$f:\mathbb{R}\to\mathbb{R}$	forward_difference::sderivative2()
2nd derivative	$\mathbf{f}:\mathbb{R}\to\mathbb{R}^{m}$	forward_difference::vderivative2()
2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::spartial_derivative2()
2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	forward_difference::vpartial_derivative2()
mixed 2nd partial derivative	$f:\mathbb{R}^{n}\to\mathbb{R}$	forward_difference::mixed_spartial_derivative2()
mixed 2nd partial derivative	$\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{m}$	forward_difference::mixed_vpartial_derivative2()

Passing Runtime Parameters

For the finite difference methods, we can simply capture any runtime parameters using closures.

Examples are included for each function (for example, here is the example for the central_difference::jacobian() function): central_difference::jacobian() - Example Passing Runtime Parameters

Modules

central_difference

Central difference approximations.

constants

Constants used for numerical differentiation.

forward_difference

Forward difference approximations.

Macros

get_directional_derivative

Get a function that returns the directional derivative of the provided multivariate, scalar-valued function.

get_gradient

Get a function that returns the gradient of the provided multivariate, scalar-valued function.

get_jacobian

Get a function that returns the Jacobian of the provided multivariate, vector-valued function.

get_mixed_spartial_derivative2

Get a function that returns the mixed second-order partial derivative of the provided multivariate, scalar-valued function.

get_mixed_vpartial_derivative2

Get a function that returns the mixed second-order partial derivative of the provided multivariate, vector-valued function.

get_sderivative

Get a function that returns the derivative of the provided univariate, scalar-valued function.

get_sderivative2

Get a function that returns the second derivative of the provided univariate, scalar-valued function.

get_shessian

Get a function that returns the Hessian of the provided multivariate, scalar-valued function.

get_spartial_derivative

Get a function that returns the partial derivative of the provided multivariate, scalar-valued function.

get_spartial_derivative2

Get a function that returns the second-order partial derivative of the provided multivariate, scalar-valued function.

get_vderivative

Get a function that returns the derivative of the provided univariate, vector-valued function.

get_vderivative2

Get a function that returns the second derivative of the provided univariate, vector-valued function.

get_vhessian

Get a function that returns the Hessian of the provided multivariate, vector-valued function.

get_vpartial_derivative

Get a function that returns the partial derivative of the provided multivariate, vector-valued function.

get_vpartial_derivative2

Get a function that returns the second-order partial derivative of the provided multivariate, vector-valued function.

Structs

Dual

First-order dual number.

HyperDual

Second-order hyper-dual number.

Traits

DualVector

Trait to create a vector of dual numbers.

HyperDualVector

Trait to create a vector of hyper-dual numbers.