If you are working on deep learning model like convolutional neural network (CNN), you must have come across this line in your code.

[Yourmodel] in the above line is presenting your model.  

Now most deep learning enthusiasts or students have questions on loss, optimizer, and metrics.

In this post, we are going to address optimizer.

But before we start with optimizers in deep learning, you need to understand that training a deep learning model is an iterative process. You run your model on data set multiple times, this is called epoch.

Your model will product an output of continuous numbers or classes/labels for regression and classification respectively.

To put simply, each iteration is intended to reduce or minimize the loss function.

What is optimizer

Needed to improve learning of a deep learning model.

Update weight and bias in back propagation based on error.

May slow down the model if dealing with large dataset.

So, let us understand what is happening here.

As you probably know, we pass bias and weight along with input through neural network and model produces an output.

Step 1- model performs forward propagation, produces an output and loss is calculated.

Step 2- model performs back propagation and based on loss (residual error), bias and weights are adjusted to reduce the error.

But do not go overboard with it, it is usually considered as bad idea to have a very high momentum. Something under 1 is fine. You should play with it and tweak it multiple times to validate the performance of your model.

Learning rate

Learning rate in deep learning is another important hyperparameter which is usually between 0 to 1 and it is used for scheduling the learning of your model.

You can think of learning rate schedule as a step. With this hyperparameter, you are essentially controlling the size of the step which model should take while minimizing the loss function or optimizing the weights.

So, it is the value by which model should update the weights in back propagation to minimize the loss.

You need to randomly set a value between 0 to 1 while compiling your model. If you do not specify, most packages will default it to 0.

For example, if you, as a person, takes larger step, you could probably jump over the center but if your step size is too small, you will need more steps to reach that point.

It works similarly.

If you’re learning rate is too small, it will take lots of steps to arrive at the convergence whereas if your step is too large, you could miss it entirely.

So, if you refer to MAE table shown earlier in this article, you will see that the difference in the first row is 30 (absolute error). Now if your model is trying to reduce this error and your learning rate is too small, your model will need too many steps to reach that point which means gradient descent will be too slow or worst-case scenario, gradient descent may get stuck at local minima and never reach that point.

And if learning rate is too large, gradient descent could just overstep it and cross it.

In the above-mentioned code, we are applying exponential decay and specifying decay rate.  Similarly, you can specify other variations of learning rates also.

This is popular and useful while compiling your deep learning models like CNN etc.

Please note all the algorithms mentioned in this post are gradient descent-based optimizers which means they use gradient descent to minimize the loss.

There are few other techniques also, but they are not covered here.

Stochastic gradient descent

Stochastic gradient descent is a variation of gradient descent. By default, instead of using all the data points from a dataset, you are randomly selecting 1 data element and updating its weight.

So, you can imagine that it will be so much faster.

Therefore, it is a much popular optimizer in deep learning. It is also known as SGD.

There is another version of SGD where instead of 1, you are taking a mini batch out of entire dataset and using that.

But by and large, its core benefit is that it helps in speeding up the training. But sometimes it causes huge fluctuations in loss so watch for that.

Adagrad

Adagrad or Adaptive Gradient descent Optimizer is all about dynamically updating the learning rate. Gradient descent by itself, will apply same learning rates to all the weight across all the layers and across all the epochs.

With Adagrad, you do not need to manually control the learning rate.

 It will adapt to features. for example, while working on parameters associated with frequent features, it will apply smaller learning rate whereas while working on less frequent features, it could apply larger learning rate.

In fact, because of this feature it has gained lot of popularity in recent past. But by design, learning rate will drastically fall which leads to very little to No learning after few epochs. This happens because in adagrad’s formula, sum of squared gradient descent will continue to accumulate in denominator which results in a big denominator over time which essentially means that learning rate will become very small.

RMSprop

RMSprop counters adagrad’s problem of increasing denominator by using the decaying average of squared gradient which essentially means it will account for recent gradient descent more than the distant ones. This helps in controlling the denominator.

By and large it is like adagrad except this one difference due to which there is no sharp fall in learning rate. It helps in ensuring that your gradient does not overshoot near minima.

It is also an adaptive learning rate which means it uses different learning rate for different parameters.

Adam

Adam (Adaptive Moment Estimation) is one more method that computes adaptive learning rates for each parameter.

Adam is essentially a hybrid of adagrad and RMSprop or you can say it takes best of both. So Adam starts with adagrap followed by RMSprop and finally weight optimization.

This way, it stores an exponentially 5 decaying average of past squared gradients, but it also keeps an exponentially decaying average of past gradients. What it means is that it gives more weightage to recent momentum than the earlier ones.

If you remember, the challenge with adagrad was that it kept accumulating squared gradients which resulted in higher denominator which led to low learning rates. With this ADAM, this issue is resolved.

It is one of the best optimizers as it is easy to implement and work on most problems.

Don't forget to share it with your friends & families to help them too. 

Leave your comment if you have any feedback or question.