Pain point #2 – The exploding gradient problem
Another drawback of recurrent neural networks is the problem of exploding gradients. This is similar to the vanishing gradient problem but the exact opposite. Sometimes, during backpropagation, the gradients explode to extraordinarily large values. As with the vanishing gradient problem, the problem of exploding gradients occurs when network architectures get deeper.
Getting ready
The name exploding gradient problem stems from the fact that, during the backpropagation step, some of the gradients vanish or become zero. Technically, this means that there is no error term being propagated backward during the backward pass of the network. This becomes a problem when the network gets deeper and more complex.
How to do it...
This section will describe the exploding gradient problem in recurrent neural networks:
- The exploding gradient problem is very similar to the vanishing gradient problem, but just the opposite.
- When long-term dependencies arise in recurrent...
