Statistical learning theory 2026: различия между версиями
Перейти к навигации
Перейти к поиску
Bauwens (обсуждение | вклад) Нет описания правки |
Bauwens (обсуждение | вклад) Нет описания правки |
||
| Строка 15: | Строка 15: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions | ! Video !! Old Vid !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions | ||
|- | |- | ||
| | | | ||
|| ''Part 1. Online learning'' | || ''Part 1. Online learning'' | ||
|- | |- | ||
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw | | [https://rutube.ru/video/private/9cc77dbd7e6a6d30d42ca6ee7d8de266/?p=OgbKo3FyR5fcul6Vzp7fJA 07 Sept] | ||
|| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw oldrec 01] | |||
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | || Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | ||
|| [https://www.dropbox.com/scl/fi/tfwv5w7064yxl05ssggnl/01slides_all.pdf?rlkey=g1ymron26rxw4kmjjht9sdjd0&st=5wm2lbl5&dl=0 sl01] | || [https://www.dropbox.com/scl/fi/tfwv5w7064yxl05ssggnl/01slides_all.pdf?rlkey=g1ymron26rxw4kmjjht9sdjd0&st=5wm2lbl5&dl=0 sl01] | ||
| Строка 27: | Строка 28: | ||
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01] | || [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01] | ||
|- | |- | ||
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe | | 14 Sept | ||
|| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe oldrec 02] | |||
|| The standard optimal algorithm. The perceptron algorithm. | || The standard optimal algorithm. The perceptron algorithm. | ||
|| | || | ||
| Строка 34: | Строка 36: | ||
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02] | || [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02] | ||
|- | |- | ||
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A | | 21 Sept | ||
|| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A oldrec 03] | |||
|| Prediction with expert advice. Recap probability theory (seminar). | || Prediction with expert advice. Recap probability theory (seminar). | ||
|| | || | ||
| Строка 42: | Строка 45: | ||
|- | |- | ||
| | | | ||
|| | |||
|| ''Part 2. Distribution independent risk bounds'' | || ''Part 2. Distribution independent risk bounds'' | ||
|- | |- | ||
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ | | 28 Sept | ||
|| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ oldrec 04] | |||
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | || Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | ||
|| | || | ||
| Строка 50: | Строка 55: | ||
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] | || [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] | ||
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04] | || [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04] | ||
|- | |- | ||
| [https://www.youtube.com/watch?v=8J5B9CCy-ws | | 05 Oct | ||
|| [https://www.youtube.com/watch?v=8J5B9CCy-ws oldrec 05] | |||
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | || Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ||
|| | || | ||
| Строка 58: | Строка 64: | ||
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05] | || [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05] | ||
|- | |- | ||
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 | | 12 Oct | ||
|| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 oldrec 06] | |||
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more) | || Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more) | ||
|| | || | ||
| Строка 65: | Строка 72: | ||
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06] | || [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06] | ||
|- | |- | ||
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ | | 19 Oct | ||
|| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ oldrec 07] | |||
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | || Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | ||
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07] | || [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07] | ||
| Строка 73: | Строка 81: | ||
|- | |- | ||
| | | | ||
|| | |||
|| ''Part 3. Margin risk bounds with applications'' | || ''Part 3. Margin risk bounds with applications'' | ||
|- | |- | ||
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ | | 02 Nov | ||
|| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ oldrec 08] | |||
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff). | || Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff). | ||
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08] | || [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08] | ||
| Строка 82: | Строка 92: | ||
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08] | || [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08] | ||
|- | |- | ||
| [https://youtube.com/live/77-rZFzX2O8 | | 09 Nov | ||
|| [https://youtube.com/live/77-rZFzX2O8 oldrec 09] | |||
|| Kernels: RKHS, representer theorem, risk bounds | || Kernels: RKHS, representer theorem, risk bounds | ||
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09] | || [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09] | ||
| Строка 88: | Строка 99: | ||
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09] | || [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09] | ||
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09] | || [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09] | ||
|- | |- | ||
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg | | 16 Nov | ||
|| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg oldrec 10] | |||
|| AdaBoost and the margin hypothesis | || AdaBoost and the margin hypothesis | ||
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | || [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | ||
| Строка 95: | Строка 107: | ||
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10] | || [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10] | ||
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10] | || [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10] | ||
|- | |- | ||
| | | | ||
|| | |||
|| ''Part 4. Neural nets'' | || ''Part 4. Neural nets'' | ||
|- | |- | ||
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw | | 23 Nov | ||
|| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw oldrec 11] | |||
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | || Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ||
|| | || | ||
| Строка 106: | Строка 120: | ||
|| | || | ||
|- | |- | ||
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de | | 30 Nov | ||
|| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de oldrec 12] | |||
|| Lazy training and the neural tangent kernel in overparameterized nets. | || Lazy training and the neural tangent kernel in overparameterized nets. | ||
|| | || | ||
| Строка 113: | Строка 128: | ||
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] | || [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] | ||
|- | |- | ||
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ | | 07 Dec | ||
|| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ oldrec 13] | |||
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets. | || Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets. | ||
|| | || | ||
| Строка 120: | Строка 136: | ||
|| | || | ||
|- | |- | ||
| | | | ||
|| | |||
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot]. | || Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot]. | ||
|} | |} | ||
Версия от 15:02, 7 сентября 2026
General Information
First lecture on Monday Sept 7 as described below. Afterwards, lectures will probably on Mondays 9h30, see this sheet.
Lectures: on Monday 13h -- 14h20 in Pokrovkaya Room TBA and in zoom by Bruno Bauwens
Seminars: on Monday 14h40 -- 16h online in Zoom by Nikita Lukianenko.
For questions about materials and last minute practical issues, join the telegram group. The course is similar to last year.
Course materials
| Video | Old Vid | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|---|
| Part 1. Online learning | ||||||
| 07 Sept | oldrec 01 | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| 14 Sept | oldrec 02 | The standard optimal algorithm. The perceptron algorithm. | ch02 ch03 | prob02 | sol02 | |
| 21 Sept | oldrec 03 | Prediction with expert advice. Recap probability theory (seminar). | ch04 ch05 | prob03 Upd 7 Oct | sol03 | |
| Part 2. Distribution independent risk bounds | ||||||
| 28 Sept | oldrec 04 | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | ch06 | prob04 | sol04 | |
| 05 Oct | oldrec 05 | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ch07 ch08 | prob05 | sol05 | |
| 12 Oct | oldrec 06 | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | ch09 | prob06 | sol06 | |
| 19 Oct | oldrec 07 | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | sol07 |
| Part 3. Margin risk bounds with applications | ||||||
| 02 Nov | oldrec 08 | Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to old recording for SVM stuff). | sl08 | ch12 ch13 | prob08 | sol08 |
| 09 Nov | oldrec 09 | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | sol09 |
| 16 Nov | oldrec 10 | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | sol10 |
| Part 4. Neural nets | ||||||
| 23 Nov | oldrec 11 | Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ch16 | See next | ||
| 30 Nov | oldrec 12 | Lazy training and the neural tangent kernel in overparameterized nets. | ch17 | prob11 | sol11 | |
| 07 Dec | oldrec 13 | Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets. | ch18 | Consult 15.12 | ||
| Colloquium Rules and questions. Select a timeslot. |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.25 * ([score colloquium 1] + [score of exam 1] + [score on colloquium 2] + [score on exam 2]) + bonus from quizzes.
There is a colloquium + exam during the session at the end of October. There is also a colloquium+exam during the session in December.
At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. There are no autogrades.
Problems exam
Date: Saturday 20.12, 13h-17h, room D203
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
-- Example of an exam (a bit easier, during COVID).
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default.
Office hours
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible