Statistical learning theory 2026: различия между версиями
Перейти к навигации
Перейти к поиску
Bauwens (обсуждение | вклад) Нет описания правки |
Bauwens (обсуждение | вклад) update slides |
||
| Строка 22: | Строка 22: | ||
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep] | | [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep] | ||
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | || Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | ||
|| [https://www.dropbox.com/scl/fi/ | || [https://www.dropbox.com/scl/fi/tfwv5w7064yxl05ssggnl/01slides_all.pdf?rlkey=g1ymron26rxw4kmjjht9sdjd0&st=5wm2lbl5&dl=0 sl01] | ||
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] | || [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] | ||
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01] | || [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01] | ||
| Строка 29: | Строка 29: | ||
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep] | | [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep] | ||
|| The standard optimal algorithm. The perceptron algorithm. | || The standard optimal algorithm. The perceptron algorithm. | ||
|| | || | ||
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03] | || [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03] | ||
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02] | || [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02] | ||
| Строка 36: | Строка 36: | ||
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep] | | [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep] | ||
|| Prediction with expert advice. Recap probability theory (seminar). | || Prediction with expert advice. Recap probability theory (seminar). | ||
|| | || | ||
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05] | || [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05] | ||
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct | || [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct | ||
| Строка 46: | Строка 46: | ||
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct] | | [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct] | ||
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | || Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | ||
|| | || | ||
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] | || [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] | ||
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] | || [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] | ||
| Строка 53: | Строка 53: | ||
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct] | | [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct] | ||
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | || Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ||
|| | || | ||
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08] | || [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08] | ||
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05] | || [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05] | ||
| Строка 60: | Строка 60: | ||
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct] | | [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct] | ||
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more) | || Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more) | ||
|| | || | ||
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09] | || [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09] | ||
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06] | || [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06] | ||
Версия от 09:48, 7 сентября 2026
General Information
First lecture on Monday Sept 7 as described below. Afterwards, lectures will probably on Mondays 9h30, see this sheet.
Lectures: on Monday 13h -- 14h20 in Pokrovkaya Room TBA and in zoom by Bruno Bauwens
Seminars: on Monday 14h40 -- 16h online in Zoom by Nikita Lukianenko.
For questions about materials and last minute practical issues, join the telegram group. The course is similar to last year.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| 16 Sep | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| 23 Sep | The standard optimal algorithm. The perceptron algorithm. | ch02 ch03 | prob02 | sol02 | |
| 30 Sep | Prediction with expert advice. Recap probability theory (seminar). | ch04 ch05 | prob03 Upd 7 Oct | sol03 | |
| Part 2. Distribution independent risk bounds | |||||
| 07 Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | ch06 | prob04 | sol04 | |
| 14 Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ch07 ch08 | prob05 | sol05 | |
| 21 Oct | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | ch09 | prob06 | sol06 | |
| 23 Oct | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | sol07 |
| Part 3. Margin risk bounds with applications | |||||
| 06 Nov | Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to old recording for SVM stuff). | sl08 | ch12 ch13 | prob08 | sol08 |
| 11 Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | sol09 |
| 18 Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | sol10 |
| Part 4. Neural nets | |||||
| 25 Nov | Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ch16 | See next | ||
| 02 Dec | Lazy training and the neural tangent kernel in overparameterized nets. | ch17 | prob11 | sol11 | |
| 09 Dec | Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets. | ch18 | Consult 15.12 | ||
| 16 Dec | Colloquium Rules and questions. Select a timeslot. |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.3 * [avg score of intermediate exams] + 0.35 * [score of 2 colloquiums] + 0.2 * [score on the exam] + bonus from quizzes.
There are are 2 colloquiums, 1 during each of the sessions. There are 3 intermediate exams, at the end of September, during the session at the end of Okt, at the end of Nov.
At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.
Autogrades: if you only need 6/10 on the final exam to have the maximal 10/10 for the course, this will be given automatically.
Problems exam
Date: Saturday 20.12, 13h-17h, room D203
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
-- Example of an exam (a bit easier, during COVID).
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default.
Office hours
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible