Probabilistic deep learning

Last updated May 18, 2023 Edit Source

See:
AI/Bayesian modelling
AI/Deep learning/GFlowNets

# Resources

A Comprehensive Introduction to Bayesian Deep Learning
Bayesian Neural Network tutorial
Bayesian Deep Learning - NeurIPS Workshop
Deep Learning Is Not Good Enough, We Need Bayesian Deep Learning for Safe AI
Making Your Neural Network Say “I Don’t Know” — Bayesian NNs using Pyro and PyTorch
Building a Bayesian deep learning classifier
Physics - a Gateway to Bayesian Deep Learning
Bayesian deep learning with Fastai : how not to be uncertain about your uncertainty!
- BNNs are a way to add uncertainty handling in our models. The idea is simple, instead of having deterministic weights that we learn, we instead learn the parameters of a random variable which we will use to sample our weights during forward propagation. Then, to learn the parameters, we will use backpropagation, sometimes with a little trick to make our parameters differentiable.
- Dropout is a way to make your Neural Networks Bayesian almost for free, and to use it during inference you just have to keep the Dropout, and sample several models, this is called MC Dropout.

Monte Carlo Dropout
What is MC Dropout
normal dropout (only at training time) serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly (e.g. multiplied by the keep ratio, which is 1 - dropout_ratio). Such a model during test time can be understood as a average of an ensemble of neural networks.
Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.
For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same datapoint, your model could predict different values each time.
The primary goal of MC dropout is to generate random predictions and interpret them as samples from a probabilistic distribution.
#TALK Estimacion de la Incertidumbre en Redes Neuronales (Valdenegro)