This is the webpage for the tutorial “Optimization Methods for Deep Learning - Theory and Practice” to be organized in the Winter Conf. on Applications of Computer Vision (WACV) 18. Here is the schedule for the tutorial session. All the material including the slides, and code based on what we covered in the tutorial are posted. References will be posted shortly (this is actually hard to do!). If you have any comments, suggestions please leave a comment or email me!

Schedule: (All slides), (Code), and (Notebook)

• Session 1 – 8:30 am to 9:30 am

  Basics of optimization algorithms: In this session we will cover basics of numerical techniques that can be deployed right off the bat for numerous problems in Machine Learning, in particular, Deep Learning. We will show simple techniques to analyze these algorithms for both unconstrained and constrained problems. The goal of this session is to provide a high level idea of the algorithmic landscape which can then be used in training deep networks.

• Break – 9:30 am to 9:45 am

• Session 2 – 9:45 am to 10:45 am

  Going beyond computations: Now that we are equipped with the knowledge of computational aspects of popular training algorithms, we turn to an equally (if not more) important statistical/learning theoretic properties of these algorithms. To that end, we will discuss the generalization theory associated with these algorithms, thereby completing the full picture in theory. The goal of this session is to understand why a model works or try to fix a model if it underperforms from an algorithmic perspective.

• Break – 10:45 am to 11 am

• Session 3 – 11 am to 11:45 am

  Getting our hands dirty: In our final session, we will see three popular applications and the performance of the discussed training algorithms in practice. The goal of this session is to show how to use ready-made framework available, and train models with just 50 lines of code.

• Conclusions and Questions – 15 minutes

  ---