Implementing optimization algorithms in ANN

So far, we have built several neural networks and obtained satisfactory overall performances. We have evaluated the model's performance using the loss function, which is a mathematical way to measure how wrong our predictions are. To improve the performance of a model based on neural networks, during the training process, weights are modified to try to minimize the loss function and make our predictions as correct as possible. To do this, optimizers are used: they are algorithms that regulate the parameters of the model, updating it in relation to what is returned by the loss function. In practice, optimizers shape the model in its most accurate form possible by overcoming weights: The loss function tells the optimizer when it is moving in the right or wrong direction.

...