How GANs can fail
Training a GAN is a tricky thing, to say the least. There are an amazing number of ways one fail at training a GAN. In fact, in writing this chapter, I found myself expanding the vocabulary of my profanity vector significantly while also spending a small fortune on cloud GPU time! Before I show you two working GANs later in the chapter, let's consider what could break and how we might be able to fix those things.
Stability
Training a GAN requires a careful balancing act between the discriminator and generator. The discriminator and generator are both fighting against each other for deep network supremacy. On the other hand, they also need each other to learn and grow. In order for this to work, neither can overpower the other one.
In an unstable GAN, the discriminator might overpower the generator, and become absolutely certain that the generator is fake. The loss goes to zero, and there is no gradient available to be sent to the generator, so it can no longer improve. Game...
