Statistical learning theory 2026
General Information
First lecture on Monday Sept 7 as described below. Afterwards, lectures will probably on Mondays 9h30, see this sheet.
Lectures: on Monday 13h -- 14h20 in Pokrovkaya Room TBA and in zoom by Bruno Bauwens
Seminars: on Monday 14h40 -- 16h online in Zoom by Nikita Lukianenko.
For questions about materials and last minute practical issues, join the telegram group. The course is similar to last year.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| 16 Sep | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| 23 Sep | The standard optimal algorithm. The perceptron algorithm. | ch02 ch03 | prob02 | sol02 | |
| 30 Sep | Prediction with expert advice. Recap probability theory (seminar). | ch04 ch05 | prob03 Upd 7 Oct | sol03 | |
| Part 2. Distribution independent risk bounds | |||||
| 07 Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | ch06 | prob04 | sol04 | |
| 14 Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ch07 ch08 | prob05 | sol05 | |
| 21 Oct | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | ch09 | prob06 | sol06 | |
| 23 Oct | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | sol07 |
| Part 3. Margin risk bounds with applications | |||||
| 06 Nov | Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to old recording for SVM stuff). | sl08 | ch12 ch13 | prob08 | sol08 |
| 11 Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | sol09 |
| 18 Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | sol10 |
| Part 4. Neural nets | |||||
| 25 Nov | Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ch16 | See next | ||
| 02 Dec | Lazy training and the neural tangent kernel in overparameterized nets. | ch17 | prob11 | sol11 | |
| 09 Dec | Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets. | ch18 | Consult 15.12 | ||
| 16 Dec | Colloquium Rules and questions. Select a timeslot. |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.25 * ([score colloquium 1] + [score of exam 1] + [score on colloquium 2] + [score on exam 2]) + bonus from quizzes.
There is a colloquium and exam during the sessions at the end of October and December.
At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.
Problems exam
Date: Saturday 20.12, 13h-17h, room D203
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
-- Example of an exam (a bit easier, during COVID).
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default.
Office hours
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible