Struct easy_ml::differentiation::Trace

pub struct Trace<T: Primitive> {
    pub number: T,
    pub derivative: T,
}

A dual number which traces a real number and keeps track of its derivative. This is used to perform Forward Automatic Differentiation

Trace implements only first order differentiation. For example, given a function 3x², you can use calculus to work out that its derivative with respect to x is 6x. You can also take the derivative of 6x with respect to x and work out that the second derivative is 6. By instead writing the function 3x² in code using Trace types as your numbers you can compute the first order derivative for a given value of x by passing your function Trace { number: x, derivative: 1.0 }.

use easy_ml::differentiation::Trace;
let x = Trace { number: 3.2, derivative: 1.0 };
let dx = Trace::constant(3.0) * x * x;
assert_eq!(dx.derivative, 3.2 * 6.0);

Why the one for the starting derivative? Because δx/δx = 1, as with symbolic differentiation.

Acknowledgments

The wikipedia page on Automatic Differentiation provided a very useful overview and explanation for understanding Forward Mode Automatic Differentiation as well as the implementation rules.

Fields

number: T

The real number

derivative: T

The first order derivative of this number.

Implementations

impl<T: Numeric + Primitive> Trace<T>

The main set of methods for using Trace types for Forward Differentiation.

The general steps are

1. create one variable
2. create as many constants as needed
3. do operations on the variable and constants
4. the outputs will have derivatives computed which can be accessed from the .derivative field, with each derivative being the output with respect to the input variable.
5. if you need derivatives for a different input then do everything all over again or do them all in parallel

pub fn constant(c: T) -> Trace<T>

Constants are lifted to Traces with a derivative of 0

Why zero for the starting derivative? Because for any constant C δC/δx = 0, as with symbolic differentiation.

pub fn variable(x: T) -> Trace<T>

To lift a variable that you want to find the derivative of a function to, the Trace starts with a derivative of 1

Why the one for the starting derivative? Because δx/δx = 1, as with symbolic differentiation.

pub fn derivative(function: impl FnOnce(Trace<T>) -> Trace<T>, x: T) -> T

Computes the derivative of a function with respect to its input x.

This is a shorthand for (function(Trace::variable(x))).derivative

In the more general case, if you provide a function with an input x and it returns N outputs y₁ to y_N then you have computed all the derivatives δy_i/δx for i = 1 to N.

impl<T: Numeric + Primitive> Trace<T>

where for<'a> &'a T: NumericRef<T>,

pub fn unary(&self, fx: impl Fn(T) -> T, dfx_dx: impl Fn(T) -> T) -> Trace<T>

Creates a new Trace from a reference to an existing Trace by applying some unary function to it which operates on the type the Trace wraps.

To compute the new trace, the unary function of some input x to some output y is needed along with its derivative with respect to its input x.

For example, tanh is a commonly used activation function, but the Real trait does not include this operation and Trace has no operations for it specifically. However, you can use this function to compute the tanh of a Trace like so:

use easy_ml::differentiation::Trace;
let x = Trace::variable(0.7f32);
// the derivative of tanh(x) is sech(x) * sech(x) which is equivalent to
// 1 / (cosh(x) * cosh(x))
let y = x.unary(|x| x.tanh(), |x| 1.0 / (x.cosh() * x.cosh()));
assert_eq!(y.derivative, 1.0f32 / (0.7f32.cosh() * 0.7f32.cosh()));

pub fn binary( &self, rhs: &Trace<T>, fxy: impl Fn(T, T) -> T, dfxy_dx: impl Fn(T, T) -> T, dfxy_dy: impl Fn(T, T) -> T ) -> Trace<T>

Creates a new Trace from a reference to two existing Traces by applying some binary function to them which operates on two arguments of the type the Traces wrap.

To compute the new trace, the binary function of some inputs x and y to some output z is needed along with its derivative with respect to its first input x and its derivative with respect to its second input y.

For example, atan2 takes two arguments, but the Real trait does not include this operation and Trace has no operations for it specifically. However, you can use this function to compute the atan2 of two Traces like so:

use easy_ml::differentiation::Trace;
let x = Trace::variable(3.0f32);
let y = Trace::variable(3.0f32);
// the derivative of atan2 with respect to x is y/(x*x + y*y)
// https://www.wolframalpha.com/input/?i=d%28atan2%28x%2Cy%29%29%2Fdx
// the derivative of atan2 with respect to y is -x/(x*x + y*y)
// https://www.wolframalpha.com/input/?i=d%28atan2%28x%2Cy%29%29%2Fdy
let z = x.binary(&y,
    |x, y| x.atan2(y),
    |x, y| y/((x*x) + (y*y)),
    |x, y| -x/((x*x) + (y*y))
);

Trait Implementations

impl<T: Numeric + Primitive> Add<&T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Addition for a trace and a constant of the same type with both referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: &T) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add<&T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: &T) -> Self::Output

Performs the + operation.

impl<'l, 'r, T: Numeric + Primitive> Add<&'r Trace<T>> for &'l Trace<T>

where for<'a> &'a T: NumericRef<T>,

Addition for two traces of the same type with both referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: &Trace<T>) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add<&Trace<T>> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: &Trace<T>) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add<T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: T) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add<T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: T) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add<Trace<T>> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: Trace<T>) -> Self::Output

Performs the + operation.

impl<T: Numeric + Primitive> Add for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type.

type Output = Trace<T>

The resulting type after applying the + operator.

fn add(self, rhs: Trace<T>) -> Self::Output

Performs the + operation.

impl<T: Clone + Primitive> Clone for Trace<T>

Any trace of a Cloneable type implements clone

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Numeric + Real + Primitive> Cos for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Cosine of a Trace by reference.

type Output = Trace<T>

fn cos(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Cos for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace by value.

type Output = Trace<T>

fn cos(self) -> Self::Output

impl<T: Debug + Primitive> Debug for Trace<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Display + Primitive> Display for Trace<T>

A trace is displayed by showing its number component.

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Numeric + Primitive> Div<&T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Dvision for a trace and a constant of the same type with both referenced.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: &T) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div<&T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: &T) -> Self::Output

Performs the / operation.

impl<'l, 'r, T: Numeric + Primitive> Div<&'r Trace<T>> for &'l Trace<T>

where for<'a> &'a T: NumericRef<T>,

Division for two referenced traces of the same type.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: &Trace<T>) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div<&Trace<T>> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: &Trace<T>) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div<T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: T) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div<T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: T) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div<Trace<T>> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: Trace<T>) -> Self::Output

Performs the / operation.

impl<T: Numeric + Primitive> Div for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type.

type Output = Trace<T>

The resulting type after applying the / operator.

fn div(self, rhs: Trace<T>) -> Self::Output

Performs the / operation.

impl<T: Numeric + Real + Primitive> Exp for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Exponential, ie e^x of a Trace by reference.

type Output = Trace<T>

fn exp(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Exp for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace by value.

type Output = Trace<T>

fn exp(self) -> Self::Output

impl<T: Numeric + Primitive> FromUsize for Trace<T>

fn from_usize(n: usize) -> Option<Trace<T>>

impl<T: Numeric + Real + Primitive> Ln for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Natural logarithm, ie ln(x) of a Trace by reference.

type Output = Trace<T>

fn ln(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Ln for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace by value.

type Output = Trace<T>

fn ln(self) -> Self::Output

impl<T: Numeric + Primitive> Mul<&T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Multiplication for a trace and a constant of the same type with both referenced.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: &T) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul<&T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: &T) -> Self::Output

Performs the * operation.

impl<'l, 'r, T: Numeric + Primitive> Mul<&'r Trace<T>> for &'l Trace<T>

where for<'a> &'a T: NumericRef<T>,

Multiplication for two referenced traces of the same type.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: &Trace<T>) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul<&Trace<T>> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: &Trace<T>) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul<T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: T) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul<T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: T) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul<Trace<T>> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: Trace<T>) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Mul for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type.

type Output = Trace<T>

The resulting type after applying the * operator.

fn mul(self, rhs: Trace<T>) -> Self::Output

Performs the * operation.

impl<T: Numeric + Primitive> Neg for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Negation for a referenced Trace of some type.

type Output = Trace<T>

The resulting type after applying the - operator.

fn neg(self) -> Self::Output

Performs the unary - operation.

impl<T: Numeric + Primitive> Neg for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Negation for a Trace by value of some type.

type Output = Trace<T>

The resulting type after applying the - operator.

fn neg(self) -> Self::Output

Performs the unary - operation.

impl<T: PartialEq + Primitive> PartialEq for Trace<T>

Any trace of a PartialEq type implements PartialEq

Note that as a Trace is intended to be substitutable with its type T only the number parts of the trace are compared. Hence the following is true

use easy_ml::differentiation::Trace;
assert_eq!(Trace { number: 0, derivative: 1 }, Trace { number: 0, derivative: 2 })

fn eq(&self, other: &Self) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: PartialOrd + Primitive> PartialOrd for Trace<T>

Any trace of a PartialOrd type implements PartialOrd

Note that as a Trace is intended to be substitutable with its type T only the number parts of the trace are compared. Hence the following is true

use easy_ml::differentiation::Trace;
assert!(Trace { number: 1, derivative: 1 } > Trace { number: 0, derivative: 2 })

fn partial_cmp(&self, other: &Self) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl<T: Numeric + Real + Primitive> Pi for Trace<T>

fn pi() -> Trace<T>

impl<T: Numeric + Real + Primitive> Pow<&T> for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Power of a trace to a constant of the same type with both referenced.

type Output = Trace<T>

fn pow(self, rhs: &T) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<&T> for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

fn pow(self, rhs: &T) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<&Trace<T>> for &T

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Power of a constant to a trace of the same type with both referenced.

type Output = Trace<T>

fn pow(self, rhs: &Trace<T>) -> Self::Output

impl<'l, 'r, T: Numeric + Real + Primitive> Pow<&'r Trace<T>> for &'l Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Power of one Trace to another, ie self^rhs for two traces of the same type with both referenced.

type Output = Trace<T>

fn pow(self, rhs: &Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<&Trace<T>> for T

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

fn pow(self, rhs: &Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<&Trace<T>> for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for two traces of the same type with the right referenced.

type Output = Trace<T>

fn pow(self, rhs: &Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<T> for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

fn pow(self, rhs: T) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<T> for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

fn pow(self, rhs: T) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<Trace<T>> for &T

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

fn pow(self, rhs: Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<Trace<T>> for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for two traces of the same type with the left referenced.

type Output = Trace<T>

fn pow(self, rhs: Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow<Trace<T>> for T

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

fn pow(self, rhs: Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Pow for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for two traces of the same type.

type Output = Trace<T>

fn pow(self, rhs: Trace<T>) -> Self::Output

impl<T: Numeric + Real + Primitive> Sin for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Sine of a Trace by reference.

type Output = Trace<T>

fn sin(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Sin for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace by value.

type Output = Trace<T>

fn sin(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Sqrt for &Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Square root of a Trace by reference.

type Output = Trace<T>

fn sqrt(self) -> Self::Output

impl<T: Numeric + Real + Primitive> Sqrt for Trace<T>

where for<'a> &'a T: NumericRef<T> + RealRef<T>,

Operation for a trace by value.

type Output = Trace<T>

fn sqrt(self) -> Self::Output

impl<T: Numeric + Primitive> Sub<&T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Subtraction for a trace and a constant of the same type with both referenced.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: &T) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub<&T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: &T) -> Self::Output

Performs the - operation.

impl<'l, 'r, T: Numeric + Primitive> Sub<&'r Trace<T>> for &'l Trace<T>

where for<'a> &'a T: NumericRef<T>,

Subtraction for two referenced traces of the same type.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: &Trace<T>) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub<&Trace<T>> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the right referenced.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: &Trace<T>) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub<T> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: T) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub<T> for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for a trace and a constant of the same type.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: T) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub<Trace<T>> for &Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type with the left referenced.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: Trace<T>) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sub for Trace<T>

where for<'a> &'a T: NumericRef<T>,

Operation for two traces of the same type.

type Output = Trace<T>

The resulting type after applying the - operator.

fn sub(self, rhs: Trace<T>) -> Self::Output

Performs the - operation.

impl<T: Numeric + Primitive> Sum for Trace<T>

Any trace of a Numeric type implements Sum, which is the same as adding a bunch of Trace types together.

fn sum<I>(iter: I) -> Trace<T>

where I: Iterator<Item = Trace<T>>,

Method which takes an iterator and generates Self from the elements by “summing up” the items.

impl<T: Numeric + Primitive> ZeroOne for Trace<T>

Trace implements ZeroOne by returning constants.

fn zero() -> Trace<T>

fn one() -> Trace<T>

impl<T: Copy + Primitive> Copy for Trace<T>

Any trace of a Copy type implements Copy