\subsection{Learning Rates}

It seems that when learning rates are updated, they are updated at a per-batch 
schedule.

The standard thing is to have some learning rate $\alpha$, and at each step 
we scale the gradient $\vect g$ 
\[
    \vect{g} \leftarrow \frac{\vect g}{\|\vect g\|} \cdot \alpha
\]

However sometimes we want $\alpha$ to change at every iteration. 
We also sometimes want to be more adaptive. So, our implementation
has something called a ``Gradient Update Schedule,'' which is a trait that 
allows a user to determine what rule is used to rescale or transform the 
gradient at every step before the neural network is updated. For example,
if we are just doing a static learning rate of $\alpha$, then the 
gradient update rule is just what is written above. Several 
default implementations are already included in this library, including 
AdaGrad (covered in CS 4780).