Generative Adversarial Networks (GANs)

Last updated Mar 30, 2023 Edit Source

A GAN consists of two networks; a generator (G) and a discriminator (D), given a set of training examples, G will generate outputs and D will classify them as either being from the same distribution as the training examples or not. In doing so D is optimized so as to be able to discriminate between examples from the training example and from the generator network which in turn is optimized to fool D into classifying its output as being drawn from the training examples. After such training G can now generate samples with properties very similar to those of the training examples. GANs tend to be devilishly hard to train

See AI/Deep learning/Generative modelling

# Resources

List of papers and other on Generative Adversarial Networks
Generative Adversarial Networks
Generative adversarial networks
Introduction to deep generative modeling: Generative Adversarial Networks (GANs)
Generative adversarial networks for beginners
Intuitive explanation of GANs. Subtypes
https://adeshpande3.github.io/adeshpande3.github.io/Deep-Learning-Research-Review-Week-1-Generative-Adversarial-Nets
http://www.openias.org/hybrid-generative-discriminative
http://edwardlib.org/tutorials/gan
Play with GANs in your browser
Do GANs actually do distribution learning?
The GAN Zoo - A list of all named GANs!
Advances in Generative Adversarial Networks
- Drawbacks of using GANs: Mode collapse, Convergence, Quality evaluation, Metrics
- Techniques for Improving Performance:
  - Alternative Loss Functions: One of the most popular fixes to the shortcomings of GANs is the Wasserstein GAN. It essentially replaces the Jensen Shannon divergence of conventional GANs with the Earth Mover distance (Wasserstein-1 distance or EM distance)
  - Two Timescale Update Rule (TTUR): In this method, we use a different learning rate for the discriminator and the generator. Typically, a slower update rule is used for the generator and a faster update rule is used for the discriminator
  - Gradient Penalty: In the paper Wasserstein GAN GP, a simple gradient penalty was introduced which is added to the loss function to avoid exploding vanishing gradients and optimization issues (caused by weight clipping)
  - Spectral Normalization: weight normalization technique that is typically used on the Discriminator to enhance the training process
  - Unrolling and Packing
  - Stacking GANs: use multiple GANs placed consecutively, where each GAN solves an easier version of the problem. For instance, FashionGAN used two GANs to perform localized image translation. Progressive GANs (ProGANs) can generate high quality images of excellent resolution.
  - Relativistic GANs: Conventional GANs measure the probability of the generated data being real. Relativistic GANs measure the probability of the generated data being “more realistic” than the real data. We can measure this “relative realism” using an appropriate distance measure, as mentioned in the RGAN paper
  - Self Attention Mechanism (SAGAN): The authors of Self Attention GANs claim that convolutions used for generating images look at information that are spread locally. That is, they miss out on relationships that span globally due to their restrictive receptive field. Self-Attention Generative Adversarial Network allows attention-driven, long-range dependency modeling for image generation tasks.
  - Miscellaneous Techniques: Feature Matching, Mini Batch Discrimination, Historical Averaging, One-sided Label Smoothing, Virtual Batch Normalization.