## Custom Optimizer - Sparse Adam optimizer
This example implements an Adam optimizer for sparse and dense gradients. This is useful for large embedding matrices, where the gradient is very sparse. Instead of updating the whole embedding, just a small portion of the matrix is updated and therefore a significant speed-up in training time is gained.
For the dense update step it uses the `addcdiv_` (fraction and then add assign in-place) and `addcmul_` (multiply and then add assign in-place) functions and for the sparse part `index_select` and `index_add` to reduce the necessary work. The sparse gradient update subroutine is only faster for sparse matrices. There is a `force_sparse` parameter, which enforces that the sparse subroutine is used, even for dense gradients. This is only for testing purposes, because the problem is actual of dense nature.
As a toy example, a MNIST classification problem is solved with help of the Adam implementation. For further information take a look at the [python implementation](https://github.com/pytorch/pytorch/blob/master/torch/optim/sparse_adam.py) and at [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980). The Adam optimizer should reach 97% in about 170 epochs.