<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="UTF-8">
<title></title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<base href="">
<link rel="stylesheet" href="book.css">
<link href='http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,600italic,700italic,800italic,400,300,600,700,800' rel='stylesheet' type='text/css'>
<link rel="shortcut icon" href="favicon.png">
<!-- Font Awesome -->
<link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<!-- MathJax -->
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<!-- Fetch JQuery from CDN but have a local fallback -->
<script src="http://code.jquery.com/jquery-2.1.4.min.js"></script>
<script>
if (typeof jQuery == 'undefined') {
document.write(unescape("%3Cscript src='jquery.js'%3E%3C/script%3E"));
}
</script>
</head>
<body>
<!-- Set the theme before any content is loaded, prevents flash -->
<script type="text/javascript">
var theme = localStorage.getItem('theme');
if (theme == null) { theme = 'light'; }
$('body').removeClass().addClass(theme);
</script>
<!-- Hide / unhide sidebar before it is displayed -->
<script type="text/javascript">
var sidebar = localStorage.getItem('sidebar');
if (sidebar === "hidden") { $("html").addClass("sidebar-hidden") }
else if (sidebar === "visible") { $("html").addClass("sidebar-visible") }
</script>
<div id="sidebar" class="sidebar">
<ul class="chapter"><li><a href="./juice.html"><strong>1.</strong> Juice</a></li><li><a href="./layers.html" class="active"><strong>2.</strong> Layers</a></li><li><ul class="section"><li><a href="./layer-lifecycle.html"><strong>2.1.</strong> Layer Lifecycle</a></li><li><a href="./building-networks.html"><strong>2.2.</strong> Create a Network</a></li><li><a href="./create-new-layer.html"><strong>2.3.</strong> Create a new Layer</a></li></ul></li><li><a href="./solvers.html"><strong>3.</strong> Solvers</a></li><li><ul class="section"><li><a href="./optimize-layers.html"><strong>3.1.</strong> Optimize Layers</a></li><li><a href="./multi-device-optimization.html"><strong>3.2.</strong> Multi-Device Optimization</a></li><li><a href="./distributed-optimization.html"><strong>3.3.</strong> Distributed Optimization</a></li></ul></li><li><a href="./backend.html"><strong>4.</strong> Backend</a></li><li><a href="./deep-learning-glossary.html"><strong>5.</strong> Glossary</a></li><li class="spacer"></li><li><a href="http://spearow.github.io/juice/juice/index.html"><strong>6.</strong> Rust API Documentation</a></li></ul>
</div>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar" class="menu-bar">
<div class="left-buttons">
<i id="sidebar-toggle" class="fa fa-bars"></i>
<i id="theme-toggle" class="fa fa-paint-brush"></i>
</div>
<h1 class="menu-title"></h1>
<div class="right-buttons">
<i id="print-button" class="fa fa-print" title="Print this book"></i>
</div>
</div>
<div id="content" class="content">
<h1>Layers</h1>
<h3>What is a Layer?</h3>
<p><a href="./deep-learning-glossary.html#Layer">Layers</a> are the only building
blocks in Juice. As we will see later on, everything is a layer. Even when
we construct <a href="./deep-learning-glossary.html#Network">networks</a>, we are still just
working with layers composed of smalle layers. This makes the API clean and expressive.</p>
<p>A layer is like a function: given an input it computes an output.
It could be some mathematical expression, like Sigmoid, ReLU, or a non-mathematical instruction,
like querying data from a database, logging data, or anything in between.
In Juice, layers describe not only the interior 'hidden layers' but also the input and
output layer.</p>
<p>Layers in Juice are only slightly opinionated, they need to take
an input and produce an output. This is required in order to successfully stack
layers on top of each other to build a network. Other than that, a
layer in Juice can implement any behaviour.</p>
<p>Layers are constructed via the <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>LayerConfig</code>
(/src/layer.rs)</a>, which makes creating even complex networks easy
and manageable.</p>
<pre><code class="language-rust">// construct the config for a fully connected layer with 500 notes
let linear_1: LayerConfig = LayerConfig::new("linear1", LinearConfig { output_size: 500 })
</code></pre>
<p>A <code>LayerConfig</code> can be turned into an initialized, fully operable <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>Layer</code>
(/src/layer.rs)</a> with its <code>from_config</code> method.</p>
<pre><code class="language-rust">// construct the config for a fully connected layer with 500 notes
let linear_1: LayerConfig = LayerConfig::new("linear1", LinearConfig { output_size: 500 })
let linear_network_with_one_layer: Layer = Layer::from_config(backend, &linear_1);
</code></pre>
<p>Hurray! We just constructed a <a href="./deep-learning-glossary.html#Network">network</a>
with one layer. (In the following chapter we will learn how to create more
powerful networks).</p>
<p>The <code>from_config</code> method initializes a <code>Layer</code>, which wraps the specific implementation (a struct that has <a href="https://github.com/spearow/juice/blob/master/src/layer.rs"><code>ILayer</code>(/src/layer.rs)</a> implemented) in a worker field.
In the tiny example above, the worker field of the <code>linear_network_with_one_layer</code>
is a <a href="https://github.com/spearow/juice/blob/master/src/layers/common/linear.rs"><code>Linear</code> (/src/layers/common/linear.rs)</a> because we constructed
the <code>linear_network_with_one_layer</code> from a <code>LinearConfig</code>. The worker field
introduces the specific behaviour of the layer.</p>
<p>In the following chapters we explore more about how we can construct
real-world networks, the layer lifecycle and how we can add new layers to the Juice framework.</p>
<h3>What can Layers do?</h3>
<p>A layer can implement basically any behaviour: deep learning related like
convolutions or LSTM, classical machine learning related like nearest neighbors
or random forest, or utility related like logging or normalization. To make the
behaviour of a layer more explicit, Juice groups layers into one of five
categories based on their (machine learning) functionality:</p>
<ol>
<li><a href="#Activation%20Layers">Activation</a></li>
<li><a href="#Common%20Layers">Common</a></li>
<li><a href="#Loss%20Layers">Loss</a></li>
<li><a href="#Utility%20Layers">Utility</a></li>
<li><a href="#Container%20Layers">Container.</a></li>
</ol>
<p>In practice, the groups are not really relevant, it helps make the file
structure cleaner. And it simplifies the explanation of what a layer is
doing.</p>
<h4>Activation Layers</h4>
<p>Activation layers provide element-wise operations and return an output of
the same size as the input. Activation layers can be seen as equivalent to
nonlinear <a href="https://en.wikipedia.org/wiki/Activation_function">Activation Functions</a>
and are a fundamental piece in neural networks.</p>
<p>Examples of activation layers are <code>Sigmoid</code>, <code>TanH</code> or <code>ReLU</code>. All available
activation layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/activation">src/layers/activation</a>.</p>
<h4>Loss Layers</h4>
<p>Loss layers compare an output to a target value and assign a cost to minimize.
Loss layers are often the last layer in a network.</p>
<p>Examples of loss layers are <code>Hinge Loss</code>, <code>Softmax Loss</code> or <code>Negative Log Likelihood</code>. All available loss layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/loss">src/layers/loss</a>.</p>
<h4>Common Layers</h4>
<p>Common layers can differ in their connectivity and behavior. They are typically
anything that is not an activation or loss layer.</p>
<p>Examples of common layers are <code>fully-connected</code>, <code>convolutional</code>, <code>pooling</code>, <code>LSTM</code>,
etc. All available common layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/common">src/layers/common</a>.</p>
<h4>Utility Layers</h4>
<p>Utility layers introduce all kind of helpful functionality, which might not be
directly related to machine learning and neural nets. These could be operations
for normalizing, restructuring or transforming information, log and debug
behavior or data access. Utility Layers follow the general behavior of a layer
like the other types.</p>
<p>Examples of Utility layers are <code>Reshape</code>, <code>Flatten</code> or <code>Normalization</code>. All
available utility layers can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/utility">src/layers/utility</a>.</p>
<h4>Container Layers</h4>
<p>Container layers take <code>LayerConfig</code>s and connect them on initialization, which
creates a "network". But as container layers are layers themselves, one can stack multiple
container layers on top of another and compose even bigger container layers.
Container layers differ in how they connect the layers that it receives.</p>
<p>Examples of container layers are <code>Sequential</code>. All available container layers
can be found at
<a href="https://github.com/spearow/juice/tree/master/src/layers/container">src/layers/container</a>.</p>
<h3>Why Layers?</h3>
<p>The benefit of using a layer-based design approach is that it allows for a very expressive
setup that can represent, as far as we know, any machine learning algorithm.
That makes Juice a framework, that can be used to construct practical machine
learning applications that combine different paradigms.</p>
<p>Other machine learning frameworks take a symbolic instead of a layered approach.
For Juice we decided against it, as we found it easier for developers to work with
layers than mathematical expressions. More complex algorithms like LSTMs are
also harder to replicate in a symbolic framework. We
believe that Juices layer approach strikes a great balance between
expressiveness, usability and performance.</p>
</div>
<!-- Mobile navigation buttons -->
<a href="./juice.html" class="mobile-nav-chapters previous">
<i class="fa fa-angle-left"></i>
</a>
<a href="./layer-lifecycle.html" class="mobile-nav-chapters next">
<i class="fa fa-angle-right"></i>
</a>
</div>
<a href="./juice.html" class="nav-chapters previous" title="You can navigate through the chapters using the arrow keys">
<i class="fa fa-angle-left"></i>
</a>
<a href="./layer-lifecycle.html" class="nav-chapters next" title="You can navigate through the chapters using the arrow keys">
<i class="fa fa-angle-right"></i>
</a>
</div>
<!-- Local fallback for Font Awesome -->
<script>
if ($(".fa").css("font-family") !== "FontAwesome") {
$('<link rel="stylesheet" type="text/css" href="_FontAwesome/css/font-awesome.css">').prependTo('head');
}
</script>
<script src="highlight.js"></script>
<script src="book.js"></script>
</body>
</html>
