Hossein Azizpour
During the first modules, core deep learning techniques and optimization methods are taught. This module contains some of the advanced topics of deep learning research, knowledge of which will be useful in various PhD projects within WASP. The module for 2021, contains three main topics as follows.
1.1 Uncertainty Estimation with (Probabilistic) Deep Networks
Deep networks have demonstrated fascinating generalization when encountered with unseen data from the same distribution as that of the training data. However, the accuracy is still far from perfect. This necessitates equipping the deep-learning-based systems with an understanding of the confidence in their predictions. Uncertainty estimation/quantification is a sub-field that tries to estimate such confidence through probabilistic techniques. This includes 1) works that propose a probabilistic variant of standard networks, 2) those that cast a deep network’s property as an approximate Bayesian model, as well as 3) other approaches of obtaining uncertainty without modification of the training procedure such as bagging and bootstrapping.
1.2 Deep Generative Modeling
Uncertainty estimation techniques in the previous part includes probabilistic deep networks for supervised discriminative learning. This part, on the other hand, explores the probabilistic deep networks that tackle (unsupervised) generative modeling. This mainly includes Variational AutoEncoders, Autoregressive, and Normalizing-Flow-based deep generative models.
1.3 Generative Adversarial Networks (GAN)
An important family of deep generative models, covered in the previous part, are Generative Adversarial Networks (GANs). The approach of GANs to generative modelling are fundamentally different from the previously-mentioned families in two main aspects: 1) non-maximum-likelihood estimation of model parameters, 2) game-theoretic adversarial loss function. Both of these are important concepts that will be covered in this part. The latter aspect of adversarial learning extends beyond the topic of generative modeling. These interesting extensions are also briefly covered.
From the theoretical point of view, the course assumes some general knowledge about machine learning including core deep learning, optimization as well as basic understanding of probability theory/modeling and calculus. Furthermore, for the course practicals, basic-to-intermediate knowledge of python is required. Lack of prior experience with deep networks packages such as TensorFlow and PyTorch will require additional effort from the students.
After successful completion of the course, the student should be able to:
- explain and justify the research topics of uncertainty estimation and generative modeling.
- account for the theoretical background of those topics
- recognize the directions (under those topics) in which further research can be done to advance the field of deep learning
- implement recently-published advanced deep learning works, under those topics
- follow the research in the field and critically evaluate the methods’ weaknesses and strengths
The following materials will be provided in this module:
- handouts and lecture notes per part.
- hands-on practicals
- suggested online resources for more in-depth exploration such as video-lectures, seminar talks, and conference tutorials that are relevant to each topic.
- a list of important recent papers of each topic for exploration of the recent developments.
The course will contain the following activities:
- before: pre-studies
- during: lecture, discussion, and practical sessions
- after: paper-reading assignment and final group project
The examination includes the following items:
- hands-on practicals
- paper-reading assignment (reading, analyzing, and presenting the analysis of recent publications on the covered topics)
- final group project implementing recently-published papers
This year’s course page is found here: