<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="UTF-8">
<title></title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<base href="">
<link rel="stylesheet" href="book.css">
<link href='http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,600italic,700italic,800italic,400,300,600,700,800' rel='stylesheet' type='text/css'>
<link rel="shortcut icon" href="favicon.png">
<!-- Font Awesome -->
<link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<!-- MathJax -->
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<!-- Fetch JQuery from CDN but have a local fallback -->
<script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
<script>
if (typeof jQuery == 'undefined') {
document.write(unescape("%3Cscript src='jquery.js'%3E%3C/script%3E"));
}
</script>
</head>
<body>
<!-- Set the theme before any content is loaded, prevents flash -->
<script type="text/javascript">
var theme = localStorage.getItem('theme');
if (theme == null) { theme = 'light'; }
$('body').removeClass().addClass(theme);
</script>
<!-- Hide / unhide sidebar before it is displayed -->
<script type="text/javascript">
var sidebar = localStorage.getItem('sidebar');
if (sidebar === "hidden") { $("html").addClass("sidebar-hidden") }
else if (sidebar === "visible") { $("html").addClass("sidebar-visible") }
</script>
<div id="sidebar" class="sidebar">
<ul class="chapter"><li><a href="./juice.html"><strong>1.</strong> Juice</a></li><li><a href="./layers.html"><strong>2.</strong> Layers</a></li><li><ul class="section"><li><a href="./layer-lifecycle.html"><strong>2.1.</strong> Layer Lifecycle</a></li><li><a href="./building-networks.html"><strong>2.2.</strong> Create a Network</a></li><li><a href="./create-new-layer.html"><strong>2.3.</strong> Create a new Layer</a></li></ul></li><li><a href="./solvers.html"><strong>3.</strong> Solvers</a></li><li><ul class="section"><li><a href="./optimize-layers.html"><strong>3.1.</strong> Optimize Layers</a></li><li><a href="./multi-device-optimization.html"><strong>3.2.</strong> Multi-Device Optimization</a></li><li><a href="./distributed-optimization.html"><strong>3.3.</strong> Distributed Optimization</a></li></ul></li><li><a href="./backend.html"><strong>4.</strong> Backend</a></li><li><a href="./deep-learning-glossary.html"><strong>5.</strong> Glossary</a></li><li class="spacer"></li><li><a href="http://spearow.github.io/juice/juice/index.html"><strong>6.</strong> Rust API Documentation</a></li></ul>
</div>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar" class="menu-bar">
<div class="left-buttons">
<i id="sidebar-toggle" class="fa fa-bars"></i>
<i id="theme-toggle" class="fa fa-paint-brush"></i>
</div>
<h1 class="menu-title"></h1>
<div class="right-buttons">
<i id="print-button" class="fa fa-print" title="Print this book"></i>
</div>
</div>
<div id="content" class="content">
<h1>Juice - Machine Learning for Hackers</h1>
<blockquote>
<p>Our life is frittered away by detail. Simplify, simplify. -
<em>Henry David Thoreau</em></p>
</blockquote>
<p>This short book teaches you how you can build machine learning applications (with
<a href="https://github.com/spearow/juice">Juice</a>).</p>
<p>Juice is a Machine Intelligence Framework engineered by hackers, not scientists.
It has a very simple API consisting of <a href="./layers.html">Layers</a> and <a href="./solvers.html">Solvers</a>, with which
you can build classical machine as well as deep learning and other fancy machine
intelligence applications. Although Juice is just a few months old,
thanks to Rust and Coaster it is already one of the fastest machine intelligence
frameworks available.</p>
<p>Juice was inspired by the brilliant people behind TensorFlow, Torch, Caffe,
Rust and numerous research papers and brings modularity, performance and
portability to deep learning.</p>
<p><br/></p>
<div align="center">
<iframe src="https://ghbtns.com/github-btn.html?user=spearow&repo=juice&type=star&count=true" frameborder="0" scrolling="0" width="120px" height="20px"></iframe>
<a href="https://twitter.com/autumn_eng" class="twitter-follow-button" data-show-count="false">Follow @autumn_eng</a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
</div>
<p><br/></p>
<blockquote>
<p>To make the most of the book, a basic understanding of the fundamental concepts
of machine and deep learning is recommended. Good resources to get you from
zero to almost-ready-to-build-machine-learning-applications:</p>
<ul>
<li><a href="http://neuralnetworksanddeeplearning.com/">Neural Networks and Deep Learning</a> or</li>
<li><a href="http://cs231n.github.io/">Stanford Course on (Convolutional) Neural Networks</a></li>
</ul>
<p>And if you already have some experience, <a href="http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/">A 'brief' history of Deep Learning</a> or <a href="./deep-learning-glossary.html">The Glossary</a>
might prove informative.</p>
</blockquote>
<p>Both machine and deep learning are really easy with Juice.</p>
<p>Construct a <a href="./deep-learning-glossary.html#Network">Network</a> by chaining <a href="./deep-learning-glossary.html#Layer">Layers</a>.
Then optimize the network by feeding it examples.
This is why Juice's entire API consists of only two concepts: <a href="./layers.html">Layers</a>
and <a href="./solvers.html">Solvers</a>. Use layers to construct almost any kind of model: deep,
classical, stochastic or hybrids, and solvers for executing and optimizing the
model.</p>
<p>This is already the entire API for machine learning with Juice. To learn how
this is possible and how to build machine learning applications, refer to chapters
<a href="./layers.html">2. Layers</a> and <a href="./solvers.html">3. Solvers</a>. Enjoy!</p>
<h2>Benefits+</h2>
<p>Juice was built with three concepts in mind: accessibility/simplicity,
performance and portability. We want developers and companies to be able to
run their machine learning applications anywhere: on servers, desktops,
smartphones and embedded devices. Any combination of platform and
computation language (OpenCL, CUDA, etc.) is a first class citizen in Juice.</p>
<p>We coupled portability with simplicity, meaning you can deploy your machine
learning applications to almost any machine and device with no code changes.
Learn more at chapter <a href="./backend.html">4. Backend</a> or at the
<a href="https://github.com/spearow/coaster">Coaster Github repository</a>.</p>
<h2>Contributing</h2>
<p>Want to contribute? Awesome!
<a href="https://github.com/spearow/juice/blob/master/CONTRIBUTING.md">We have instructions to help you get started</a>.</p>
<p>Juice has a near real-time collaboration culture, which happens at the <a href="https://github.com/spearow/juice">Github
repository</a> and on the
<a href="https://gitter.im/spearow/juice">Juice Gitter Channel</a>.</p>
<h2>API Documentation</h2>
<p>Alongside this book you can also read the Rust API documentation if
you would like to use Juice as a crate, write a library on top of it or
just want a more low-level overview.</p>
<p><a href="http://spearow.github.io/juice/">> Rust API documentation</a></p>
<h2>License</h2>
<p>Juice is free for anyone for whatever purpose.
Juice is licensed under either
<a href="https://github.com/spearow/juice/blob/master/LICENSE-APACHE">Apache License v2.0</a> or,
<a href="https://github.com/spearow/juice/blob/master/LICENSE-MIT">MIT license</a>.
Whatever strikes your fancy.</p>
<h1>Layers</h1>
<h3>What is a Layer?</h3>
<p><a href="./deep-learning-glossary.html#Layer">Layers</a> are the only building
blocks in Juice. As we will see later on, everything is a layer. Even when
we construct <a href="./deep-learning-glossary.html#Network">networks</a>, we are still just
working with layers composed of smalle layers. This makes the API clean and expressive.</p>
<p>A layer is like a function: given an input it computes an output.
It could be some mathematical expression, like Sigmoid, ReLU, or a non-mathematical instruction,
like querying data from a database, logging data, or anything in between.
In Juice, layers describe not only the interior 'hidden layers' but also the input and
output layer.</p>
<p>Layers in Juice are only slightly opinionated, they need to take
an input and produce an output. This is required in order to successfully stack
layers on top of each other to build a network. Other than that, a
layer in Juice can implement any behaviour.</p>
<p>Layers are constructed via the <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>LayerConfig</code>
(/src/layer.rs)</a>, which makes creating even complex networks easy
and manageable.</p>
<pre><code class="language-rust">// construct the config for a fully connected layer with 500 notes
let linear_1: LayerConfig = LayerConfig::new("linear1", LinearConfig { output_size: 500 })
</code></pre>
<p>A <code>LayerConfig</code> can be turned into an initialized, fully operable <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>Layer</code>
(/src/layer.rs)</a> with its <code>from_config</code> method.</p>
<pre><code class="language-rust">// construct the config for a fully connected layer with 500 notes
let linear_1: LayerConfig = LayerConfig::new("linear1", LinearConfig { output_size: 500 })
let linear_network_with_one_layer: Layer = Layer::from_config(backend, &linear_1);
</code></pre>
<p>Hurray! We just constructed a <a href="./deep-learning-glossary.html#Network">network</a>
with one layer. (In the following chapter we will learn how to create more
powerful networks).</p>
<p>The <code>from_config</code> method initializes a <code>Layer</code>, which wraps the specific implementation (a struct that has <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>ILayer</code>(/src/layer.rs)</a> implemented) in a worker field.
In the tiny example above, the worker field of the <code>linear_network_with_one_layer</code>
is a <a href="https://github.com/spearow/juice/blob/master/src/layers/common/linear.rs"><code>Linear</code> (/src/layers/common/linear.rs)</a> because we constructed
the <code>linear_network_with_one_layer</code> from a <code>LinearConfig</code>. The worker field
introduces the specific behaviour of the layer.</p>
<p>In the following chapters we explore more about how we can construct
real-world networks, the layer lifecycle and how we can add new layers to the Juice framework.</p>
<h3>What can Layers do?</h3>
<p>A layer can implement basically any behaviour: deep learning related like
convolutions or LSTM, classical machine learning related like nearest neighbors
or random forest, or utility related like logging or normalization. To make the
behaviour of a layer more explicit, Juice groups layers into one of five
categories based on their (machine learning) functionality:</p>
<ol>
<li><a href="#Activation%20Layers">Activation</a></li>
<li><a href="#Common%20Layers">Common</a></li>
<li><a href="#Loss%20Layers">Loss</a></li>
<li><a href="#Utility%20Layers">Utility</a></li>
<li><a href="#Container%20Layers">Container.</a></li>
</ol>
<p>In practice, the groups are not really relevant, it helps make the file
structure cleaner. And it simplifies the explanation of what a layer is
doing.</p>
<h4>Activation Layers</h4>
<p>Activation layers provide element-wise operations and return an output of
the same size as the input. Activation layers can be seen as equivalent to
nonlinear <a href="https://en.wikipedia.org/wiki/Activation_function">Activation Functions</a>
and are a fundamental piece in neural networks.</p>
<p>Examples of activation layers are <code>Sigmoid</code>, <code>TanH</code> or <code>ReLU</code>. All available
activation layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/activation">src/layers/activation</a>.</p>
<h4>Loss Layers</h4>
<p>Loss layers compare an output to a target value and assign a cost to minimize.
Loss layers are often the last layer in a network.</p>
<p>Examples of loss layers are <code>Hinge Loss</code>, <code>Softmax Loss</code> or <code>Negative Log Likelihood</code>. All available loss layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/loss">src/layers/loss</a>.</p>
<h4>Common Layers</h4>
<p>Common layers can differ in their connectivity and behavior. They are typically
anything that is not an activation or loss layer.</p>
<p>Examples of common layers are <code>fully-connected</code>, <code>convolutional</code>, <code>pooling</code>, <code>LSTM</code>,
etc. All available common layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/common">src/layers/common</a>.</p>
<h4>Utility Layers</h4>
<p>Utility layers introduce all kind of helpful functionality, which might not be
directly related to machine learning and neural nets. These could be operations
for normalizing, restructuring or transforming information, log and debug
behavior or data access. Utility Layers follow the general behavior of a layer
like the other types.</p>
<p>Examples of Utility layers are <code>Reshape</code>, <code>Flatten</code> or <code>Normalization</code>. All
available utility layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/utility">src/layers/utility</a>.</p>
<h4>Container Layers</h4>
<p>Container layers take <code>LayerConfig</code>s and connect them on initialization, which
creates a "network". But as container layers are layers themselves, one can stack multiple
container layers on top of another and compose even bigger container layers.
Container layers differ in how they connect the layers that it receives.</p>
<p>Examples of container layers are <code>Sequential</code>. All available container layers
can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/container">src/layers/container</a>.</p>
<h3>Why Layers?</h3>
<p>The benefit of using a layer-based design approach is that it allows for a very expressive
setup that can represent, as far as we know, any machine learning algorithm.
That makes Juice a framework, that can be used to construct practical machine
learning applications that combine different paradigms.</p>
<p>Other machine learning frameworks take a symbolic instead of a layered approach.
For Juice we decided against it, as we found it easier for developers to work with
layers than mathematical expressions. More complex algorithms like LSTMs are
also harder to replicate in a symbolic framework. We
believe that Juices layer approach strikes a great balance between
expressiveness, usability and performance.</p>
<h1>Layer Lifecycle</h1>
<p>In chapter <a href="./layers.html">2. Layers</a> we saw how to
construct a simple <code>Layer</code> from a <code>LayerConfig</code>. In this chapter, we take
a closer look at what happens inside Juice when initializing a <code>Layer</code> and when running its
<code>.forward</code> and <code>.backward</code> methods. In the next chapter <a href="./building-networks.html">2.2 Create a Network</a> we
apply our knowledge to construct deep networks with the container layer.</p>
<p>The most important methods of a <code>Layer</code> are initialization (<code>::from_config</code>), <code>.forward</code> and <code>.backward</code>.
They basically describe the entire API, so let's take a closer look at what happens inside Juice when these methods are called.</p>
<h3>Initialization</h3>
<p>A layer is constructed from a <code>LayerConfig</code> with the <code>Layer::from_config</code>
method, which returns a fully initialized <code>Layer</code>.</p>
<pre><code class="language-rust">let mut sigmoid: Layer = Layer::from_config(backend.clone(), &LayerConfig::new("sigmoid", LayerType::Sigmoid))
let mut alexnet: Layer = Layer::from_config(backend.clone(), &LayerConfig::new("alexnet", LayerType::Sequential(cfg)))
</code></pre>
<p>In the example above, the first layer has a Sigmoid worker
(<code>LayerType::Sigmoid</code>) and the second layer has a Sequential worker.
Although both <code>::from_config</code> methods return a <code>Layer</code>, the behavior of
that <code>Layer</code> depends on the <code>LayerConfig</code> it was constructed with. The
<code>Layer::from_config</code> internally calls the <code>worker_from_config</code> method, which
constructs the specific worker defined by the <code>LayerConfig</code>.</p>
<pre><code class="language-rust">fn worker_from_config(backend: Rc<B>, config: &LayerConfig) -> Box<ILayer<B>> {
match config.layer_type.clone() {
// more matches
LayerType::Pooling(layer_config) => Box::new(Pooling::from_config(&layer_config)),
LayerType::Sequential(layer_config) => Box::new(Sequential::from_config(backend, &layer_config)),
LayerType::Softmax => Box::new(Softmax::default()),
// more matches
}
}
</code></pre>
<p>The layer-specific <code>::from_config</code> (if available or needed) then takes care of
initializing the worker struct, allocating memory for weights and so on.</p>
<p>If the worker is a container layer, its <code>::from_config</code> takes
care of initializing all the <code>LayerConfig</code>s it contains (which were added via its
<code>.add_layer</code> method) and connecting them in the order they were provided.</p>
<p>Every <code>.forward</code> or <code>.backward</code> call that is made on the returned <code>Layer</code> is
run by the internal worker.</p>
<h3>Forward</h3>
<p>The <code>forward</code> method of a <code>Layer</code> threads the input through the constructed
network and returns the output of the network's final layer.</p>
<p>The <code>.forward</code> method does three things:</p>
<ol>
<li>Reshape the input data if necessary</li>
<li>Sync the input/weights to the device where the computation happens. This step
removes the need for the worker layer to care about memory synchronization.</li>
<li>Call the <code>forward</code> method of the internal worker layer.</li>
</ol>
<p>If the worker layer is a container layer, the <code>.forward</code> method
takes care of calling the <code>.forward</code> methods of its managed
layers in the right order.</p>
<h3>Backward</h3>
<p>The <code>.backward</code> method of a <code>Layer</code> works similarly to <code>.forward</code>, apart from
needing to reshape the input. The <code>.backward</code> method computes
the gradient with respect to the input as well as the gradient w.r.t. the parameters. However,
the method only returns the input gradient because that is all that is needed to compute the
gradient of the entire network via the chain rule.</p>
<p>If the worker layer is a container layer, the <code>.backward</code> method
takes care of calling the <code>.backward_input</code> and
<code>.backward_parameter</code> methods of its managed layers in the right order.</p>
<h1>Create a Network</h1>
<p>In the previous chapters, we learned that in Juice everything is build by
layers and that the constructed thing is again a layer, which means it can
function as a new building block for something bigger. This is possible, because
a <code>Layer</code> can implement any behavior as long as it takes an input and produces
an output.</p>
<p>In <a href="./layer-lifecycle.html">2.1 Layer Lifecycle</a>
we have seen, that only one <code>LayerConfig</code> can be used to turn it via
<code>Layer::from_config</code> into an actual <code>Layer</code>. But as Deep Learning relies on
chaining multiple layers together, we need a <code>Layer</code>, who implements this
behavior for us.</p>
<p>Enter the container layers.</p>
<h3>Networks via the <code>Sequential</code> layer</h3>
<p>A <code>Sequential</code> Layer is a layer of type container layer. The config of a
container layer has a special method called,
<code>.add_layer</code> which takes one <code>LayerConfig</code> and adds it to an ordered list in the
<code>SequentialConfig</code>.</p>
<p>When turning a <code>SequentialConfig</code> into a <code>Layer</code> by passing the config to
<code>Layer::from_config</code>, the behavior of the <code>Sequential</code> is to initialize all the
layers which were added via <code>.add_layer</code> and connect the layers with each other.
This means, the output of one layer becomes the input of the next layer in the
list.</p>
<p>The input of a sequential <code>Layer</code> becomes the input of the
first layer in the sequential worker, the sequential worker then takes care
of passing the input through all the layers and the output of the last layer
then becomes the output of the <code>Layer</code> with the sequential worker. Therefore
a sequential <code>Layer</code> fulfills the requirements of a <code>Layer</code> - take an input,
return an output.</p>
<pre><code class="language-rust">// short form for: &LayerConfig::new("net", LayerType::Sequential(cfg))
let mut net_cfg = SequentialConfig::default();
net_cfg.add_input("data", &vec![batch_size, 28, 28]);
net_cfg.add_layer(LayerConfig::new("reshape", ReshapeConfig::of_shape(&vec![batch_size, 1, 28, 28])));
net_cfg.add_layer(LayerConfig::new("conv", ConvolutionConfig { num_output: 20, filter_shape: vec![5], stride: vec![1], padding: vec![0] }));
net_cfg.add_layer(LayerConfig::new("pooling", PoolingConfig { mode: PoolingMode::Max, filter_shape: vec![2], stride: vec![2], padding: vec![0] }));
net_cfg.add_layer(LayerConfig::new("linear1", LinearConfig { output_size: 500 }));
net_cfg.add_layer(LayerConfig::new("sigmoid", LayerType::Sigmoid));
net_cfg.add_layer(LayerConfig::new("linear2", LinearConfig { output_size: 10 }));
net_cfg.add_layer(LayerConfig::new("log_softmax", LayerType::LogSoftmax));
// set up the sequential layer aka. a deep, convolutional network
let mut net = Layer::from_config(backend.clone(), &net_cfg);
</code></pre>
<p>As a sequential layer is like any other layer, we can use sequential layers as
building blocks for larger networks. Important building blocks of a network can
be grouped into a sequential layer and published as a crate for others to use.</p>
<pre><code class="language-rust">// short form for: &LayerConfig::new("net", LayerType::Sequential(cfg))
let mut conv_net = SequentialConfig::default();
conv_net.add_input("data", &vec![batch_size, 28, 28]);
conv_net.add_layer(LayerConfig::new("reshape", ReshapeConfig::of_shape(&vec![batch_size, 1, 28, 28])));
conv_net.add_layer(LayerConfig::new("conv", ConvolutionConfig { num_output: 20, filter_shape: vec![5], stride: vec![1], padding: vec![0] }));
conv_net.add_layer(LayerConfig::new("pooling", PoolingConfig { mode: PoolingMode::Max, filter_shape: vec![2], stride: vec![2], padding: vec![0] }));
conv_net.add_layer(LayerConfig::new("linear1", LinearConfig { output_size: 500 }));
conv_net.add_layer(LayerConfig::new("sigmoid", LayerType::Sigmoid));
conv_net.add_layer(LayerConfig::new("linear2", LinearConfig { output_size: 10 }));
let mut net_cfg = SequentialConfig::default();
net_cfg.add_layer(conv_net);
net_cfg.add_layer(LayerConfig::new("linear", LinearConfig { output_size: 500 }));
net_cfg.add_layer(LayerConfig::new("log_softmax", LayerType::LogSoftmax));
// set up the 'big' network
let mut net = Layer::from_config(backend.clone(), &net_cfg);
</code></pre>
<h3>Networks via other container layers</h3>
<p>So far, there is only the sequential layer, but other container layers, with
slightly different behaviors are conceivable. For example a parallel or
concat layer in addition to the sequential layer.</p>
<p>How to 'train' or optimize the constructed network is topic of chapter <a href="./solvers.html">3.
Solvers</a></p>
<h1>Create a new Layer</h1>
<p>A layer in Juice can implement any behavior as long as it takes an input and
produces an output. As Juice is new, there are still many valuable layers that
are not yet implemented. This is why this chapter shows how you can add new
layers to Juice.</p>
<p>A not exclusive list of steps to take in order to implement a new layer:</p>
<blockquote>
<p>The Rust compiler is also very helpful with pointing out the necessary steps
for implementing a new layer struct. It might be beneficial to start the
implementation of a new layer from a copied file of an already existing layer.</p>
</blockquote>
<ol>
<li>
<p>Decide to which of the <a href="./layers.html#What%20can%20Layers%20do?">five types</a>
the new layer belongs. This decides under which directory to put the layer
implementation in the Juice project.</p>
</li>
<li>
<p>Create the <code>Layer</code> worker struct.</p>
</li>
<li>
<p>Expose the <code>Layer</code> worker struct in the <code>mod.rs</code> of the layer type directory.</p>
</li>
<li>
<p>Expose the <code>Layer</code> worker struct in the <code>mod.rs</code> of the <code>/layers</code> directory.</p>
</li>
<li>
<p>Implement <code>ILayer</code> and its trait boundaries for the new <code>Layer</code> worker struct.</p>
</li>
<li>
<p>Add the new layer to the <code>LayerType</code> in <code>layer.rs</code> and add the matching
for <code>.support_in_place</code> and <code>.worker_from_config</code>.</p>
</li>
<li>
<p>If the new layer relies on a coaster operation, also add the coaster
trait boundary.</p>
</li>
<li>
<p>Add documentation and serialization to the new layer.</p>
</li>
<li>
<p>(optional) Depending on how complex the layer is, you might also add tests and more
advanced implementations for its <code>.from_config</code>, <code>.reshape</code> or other helper
methods.</p>
</li>
</ol>
<h1>Solvers</h1>
<p>Solvers optimize the layer with a given objective. This might happen
by updating the weights of the layer, which is the usual practice for
Neural Networks but is not limited to this kind of learning.</p>
<p>A solver can have different learning (solving) policies. With Neural Networks, it
is common to use a Stochastic Gradient Descent based approach
like Adagrad, whereas for a classical regression the solving might be
done via a maximum likelihood estimation.</p>
<p>Similar to <code>Layer</code>s, we can construct a <a href="https://github.com/spearow/juice/blob/master/src/solver/mod.rs"><code>Solver</code> (<em>/src/solver/mod.rs</em>)</a>
from a <a href="https://github.com/spearow/juice/blob/master/src/solver/mod.rs"><code>SolverConfig</code> (<em>/src/solver/mod.rs</em>)</a>.
When passing this <code>SolverConfig</code> (e.g. an Adagrad <code>SolverConfig</code>) to the
<code>Solver::from_config</code> method, a <code>Solver</code> with the behavior
of the config is returned.</p>
<p>The most characteristic feature of the <code>SolverConfig</code> is its <code>network</code>
and <code>objective</code> fields. These two fields expect one <code>LayerConfig</code> each. When
passing the <code>SolverConfig</code> to the <code>Solver::from_config</code> method, the
<code>LayerConfig</code> of the <code>network</code> and <code>objective</code> fields are turned into
an initialized <code>Layer</code> and provided to the returned, <code>Solver</code>.</p>
<pre><code class="language-rust">// set up a Solver
let mut solver_cfg = SolverConfig { minibatch_size: batch_size, base_lr: learning_rate, momentum: momentum, .. SolverConfig::default() };
solver_cfg.network = LayerConfig::new("network", net_cfg);
solver_cfg.objective = LayerConfig::new("classifier", classifier_cfg);
let mut solver = Solver::from_config(backend.clone(), backend.clone(), &solver_cfg);
</code></pre>
<p>The now initialized <code>Solver</code> can be feed with data to optimize the <code>network</code>.</p>
<h1>Optimize Layers</h1>
<p>In the previous chapter <a href="./solvers.html">3. Solver</a>, we learned what a solver
is and what it does. In this chapter, we take a look on how to optimize a
network via a <code>Solver</code>.</p>
<p>A <code>Solver</code> after its initialization has two <code>Layer</code>s, one for the <code>network</code>
which will be optimized and one for the <code>objective</code>. The output of the <code>network</code>
layer is used by the <code>objective</code> to compute the loss. The loss is then used by
the <code>Solver</code> to optimize the <code>network</code>.</p>
<p>The <code>Solver</code> has a very simple API - <code>.train_minibatch</code> and <code>.network</code>. The
optimization of the <code>network</code> is kicked off by the <code>.train_minibatch</code>
method, which takes two input parameters - some data that is feed to the network
and the expected target value for the network.</p>
<p>A SGD (Stochastic Gradient Descent) <code>Solver</code> would now compute the output of
the <code>network</code> using as input the data, put the output together with the expected
target value into the <code>objective</code> layer and use it, together with the gradient
of the <code>network</code> to optimize the weights of the <code>network</code>.</p>
<pre><code class="language-rust">/// Train the network with one minibatch
pub fn train_minibatch(&mut self, mb_data: ArcLock<SharedTensor<f32>>, mb_target: ArcLock<SharedTensor<f32>>) -> ArcLock<SharedTensor<f32>> {
// forward through network and classifier
let network_out = self.net.forward(&[mb_data])[0].clone();
let _ = self.objective.forward(&[network_out.clone(), mb_target]);
// forward through network and classifier
let classifier_gradient = self.objective.backward(&[]);
self.net.backward(&classifier_gradient[0 .. 1]);
self.worker.compute_update(&self.config, &mut self.net, self.iter);
self.net.update_weights(self.worker.backend());
self.iter += 1;
network_out
}
</code></pre>
<p>Using the <code>.train_minibatch</code> is straight forward. We pass the data as well as the
expected result of the <code>network</code> to the <code>.train_minibatch</code> method of the
initialized <code>Solver</code> struct. A more detailed example can be found at the
<a href="https://github.com/spearow/juice-examples">spearow/juice-examples</a> repository.</p>
<pre><code class="language-rust">let inp_lock = Arc::new(RwLock::new(inp));
let label_lock = Arc::new(RwLock::new(label));
// train the network!
let inferred_out = solver.train_minibatch(inp_lock.clone(), label_lock.clone());
</code></pre>
<p>If we don't want the <code>network</code> to be trained, we can use the <code>.network</code> method
of the <code>Solver</code> to receive access to the network. The <code>Solver</code> has actually
two network methods - <code>.network</code> and <code>mut_network</code>.</p>
<p>To run just the forward of the <code>network</code> without any optimization we can run</p>
<pre><code class="language-rust">let inferred_out = solver.network().forward(inp_lock.clone());
</code></pre>
<p>Juice ships with a <a href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrix</a>,
which is a convenient way to visualize the performance of the optimized
<code>network</code>.</p>
<pre><code class="language-rust">let inferred_out = solver.train_minibatch(inp_lock.clone(), label_lock.clone());
let mut inferred = inferred_out.write().unwrap();
let predictions = confusion.get_predictions(&mut inferred);
confusion.add_samples(&predictions, &targets);
println!("Last sample: {} | Accuracy {}", confusion.samples().iter().last().unwrap(), confusion.accuracy());
</code></pre>
<p>A more detailed example can be found at the
<a href="https://github.com/spearow/juice-examples">spearow/juice-examples</a> repository.</p>
<h1>Multi-Device Optimization</h1>
<p>Optimization of a Layer over multiple devices is planned for the Juice 0.3
release. Thanks to the decoupling of computation and representation through
Coaster, multi-device optimization is fairly straight forward to implement.</p>
<p>Pull Requests are welcome :)</p>
<h1>Distributed Optimization</h1>
<p>The distributed optimization of networks will (very likely) be managed by a
standalone crate on top of Juice. Although distributed optimization will not be a
core part of Juice itself, we will cover the topic of distributed optimization
with Juice here in this chapter of the book.</p>
<h1>Backend</h1>
<p>Via the concept of a backend we can abstract over the platform we will execute
or optimize a network on. The construction of a backend is trivial. The backend
is passed to the <code>Solver</code>, (one backend for <code>network</code> and one for the
<code>objectve</code>). The Solver than executes all operations on the provided backend.</p>
<pre><code class="language-rust">let backend = ::std::rc::Rc::new(Backend::<Cuda>::default().unwrap());
// set up solver
let mut solver_cfg = SolverConfig { minibatch_size: batch_size, base_lr: learning_rate, momentum: momentum, .. SolverConfig::default() };
solver_cfg.network = LayerConfig::new("network", net_cfg);
solver_cfg.objective = LayerConfig::new("classifier", classifier_cfg);
let mut solver = Solver::from_config(backend.clone(), backend.clone(), &solver_cfg);
</code></pre>
<p>The backend is a concept of
<a href="https://github.com/spearow/coaster">Coaster</a>, to which you can refer
for now, until this chapter becomes more fleshed out.</p>
<h1>Glossary</h1>
<h3>Layer</h3>
<p><strong>In General</strong></p>
<p>A layer is the highest-level building block in a (Deep) Neural Network. A layer
is a container that usually receives weighted input, transforms it and returns
the result as output to the next layer. A layer usually contains one type of
function like ReLU, pooling, convolution etc. so that it can be easily compared
to other parts of the network. The first and last layers in a network are called
input and output layers, respectively, and all layers in between are called
hidden layers.</p>
<p><strong>In Juice</strong></p>
<p>In Juice, a layer is very similar to the general understanding of a layer. A layer
in Juice, like a layer in a (Deep) Neural Network,</p>
<ul>
<li>is the highest-level building block</li>
<li>needs to receive input, might transform it and needs to return the result</li>
<li>should be uniform (it does one type of function)</li>
</ul>
<p>Additionally to a Neural Network layer, a Juice layer can implement any
functionality, not only those related to Neural Networks like ReLU, pooling,
LSTM, etc. For example, the <code>Sequential</code> layer in Juice, allows it to connect
multiple layers, creating a network.</p>
<h3>Network</h3>
<p><strong>In General</strong></p>
<p>A network, also often called Neural Network (NN) or Artificial Neural Network
(ANN) is a subset of Machine Learning methods.</p>
<p>A not exhaustive list of other Machine Learning methods:<br />
<em>Linear Regression, SVM, Genetic/Evolution Algorithms, dynamic programming,
deterministic algorithmic optimization methods.</em></p>
<p><strong>In Juice</strong></p>
<p>In Juice, a network means a graph (a connected set) of one or more
<a href="./layers.html">layers</a>. This network can consist of Artificial Neural Network
methods, other Machine Learning methods or any other (not Machine Learning
related) methods. As described in <a href="./layers.html">2. Layers</a> a network in Juice
is actually a layer which connects other layers.</p>
<p>An initialized network is a network, which is ready to be executed, meaning it
is fully constructed e.g. all necessary memory is allocated on the host or device.</p>
<h1>Rust API Documentation</h1>
</div>
<!-- Mobile navigation buttons -->
</div>
</div>
<!-- Local fallback for Font Awesome -->
<script>
if ($(".fa").css("font-family") !== "FontAwesome") {
$('<link rel="stylesheet" type="text/css" href="_FontAwesome/css/font-awesome.css">').prependTo('head');
}
</script>
<script src="highlight.js"></script>
<script src="book.js"></script>
</body>
</html>
