variational-regression 0.1.0

Regression models trained using variational inference
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

# variational-regression

Regression models trained using variational inference

## Motivation

This library provides implementations of Bayesian regression models that learn parameter values by optimizing a variational lower bound on the log likelihood. The benefit of using variational inference is that training is efficient, and we have a well defined objective function to optimize. The Bayesian approach also naturally incorporates regularization into the models.

## Linear Regression

The model is specified as follows:

p(<b>y</b> | <b>&theta;</b>) = &prod; N(y<sub>i</sub> | <b>&theta;</b><sup>T</sup><b>x</b><sub>i</sub>, &beta;<sup>-1</sup>)

p(<b>&theta;</b> | <b>&alpha;</b>) = &prod; N(&theta;<sub>j</sub> | 0, &alpha;<sub>j</sub><sup>-1</sup>)

p(<b>&alpha;</b>) = &prod; Gam(&alpha;<sub>j</sub> | a<sub>0</sub>, b<sub>0</sub>)

p(&beta;) = Gam(&beta; | c<sub>0</sub>, d<sub>0</sub>)

with the given notation:

<b>y</b> = labels

<b>x</b> = features

<b>&theta;</b> = model weights

<b>&alpha;</b> = weight precision (inverse variance)

&beta; = noise precision (inverse variance)

## Logistic Regression

The model is specified as follows:

p(<b>y</b> | <b>&theta;</b>) = &prod; &sigma;(<b>&theta;</b><sup>T</sup><b>x</b><sub>i</sub>)<sup>y<sub>i</sub></sup> {1 - &sigma;(<b>&theta;</b><sup>T</sup><b>x</b><sub>i</sub>)}<sup>1-y<sub>i</sub></sup>

p(<b>&theta;</b> | <b>&alpha;</b>) = &prod; N(&theta;<sub>j</sub> | 0, &alpha;<sub>j</sub><sup>-1</sup>)

p(<b>&alpha;</b>) = &prod; Gam(&alpha;<sub>j</sub> | a<sub>0</sub>, b<sub>0</sub>)

with the given notation:

<b>y</b> = labels

<b>x</b> = features

<b>&theta;</b> = model weights

<b>&alpha;</b> = weight precision (inverse variance)

&sigma; = logistic sigmoid function

## Reference

The models implemented here are heavily based on those presented in Chapter 10 of "Pattern Recognition and Machine Learning" (Bishop, 2006). However a key difference is that here, each model weight has its own distribution for precision, as does the noise term for the linear regression model.

## Examples

```rust
use variational_regression::*;

fn main() -> Result<(), RegressionError> {

    // construct features
    let features: &[&[f64]] = &[
        &[-0.2, -0.9, -0.5, 0.3],
        &[0.6, 0.3, 0.3, -0.4],
        &[0.9, -0.4, -0.5, -0.6],
        &[-0.7, 0.8, 0.3, -0.3],
        &[-0.5, -0.7, -0.1, 0.8],
        &[0.5, 0.5, 0.0, 0.1],
        &[0.1, -0.0, 0.0, -0.2],
        &[0.4, 0.0, 0.2, 0.0],
        &[-0.2, 0.9, -0.1, -0.9],
        &[0.1, 0.4, -0.5, 0.9],
    ];
    
    // construct labels
    let labels: &[f64] = &[-0.4, 0.1, -0.8, 0.5, 0.6, -0.2, 0.0, 0.7, -0.3, 0.2];
    
    // configure and train model
    let config = LinearTrainConfig::default();
    let model = VariationalLinearRegression::train(features, labels, &config)?;
    
    // inspect model bias
    if let Some(bias) = model.bias() {
        println!("Bias: {}", bias);
    }

    // inspect model weights
    for (ind, weight) in model.weights().iter().enumerate() {
        println!("Weight {}: {}", ind + 1, weight);
    }

    // inspect noise variance
    println!("Noise Variance: {}", 1.0 / model.noise_precision.mean());
    
    // get predictive distribution
    let prediction = model.predict(&[0.1, -0.5, 0.3, 0.9])?;
    println!("Predictive mean: {}", prediction.mean());
    
    Ok(())
}
```

```rust
use variational_regression::*;

fn main() -> Result<(), RegressionError> {

    // construct features
    let features: &[&[f64]] = &[
        &[-0.2, -0.9, -0.5, 0.3],
        &[0.6, 0.3, 0.3, -0.4],
        &[0.9, -0.4, -0.5, -0.6],
        &[-0.7, 0.8, 0.3, -0.3],
        &[-0.5, -0.7, -0.1, 0.8],
        &[0.5, 0.5, 0.0, 0.1],
        &[0.1, -0.0, 0.0, -0.2],
        &[0.4, 0.0, 0.2, 0.0],
        &[-0.2, 0.9, -0.1, -0.9],
        &[0.1, 0.4, -0.5, 0.9],
    ];
    
    // construct labels
    let labels: &[bool] = &[true, false, true, false, true, false, true, false, true, false];
    
    // configure and train model
    let config = LogisticTrainConfig::default();
    let model = VariationalLogisticRegression::train(features, labels, &config)?;
    
    // inspect model bias
    if let Some(bias) = model.bias() {
        println!("Bias: {}", bias);
    }

    // inspect model weights
    for (ind, weight) in model.weights().iter().enumerate() {
        println!("Weight {}: {}", ind + 1, weight);
    }

    // get predictive distribution
    let prediction = model.predict(&[0.1, -0.5, 0.3, 0.9])?;
    println!("Predictive mean: {}", prediction.mean());

    Ok(())
}
```