https://wiki.cs.hse.ru/api.php?action=feedcontributions&feedformat=atom&user=BrbauwensWiki - Факультет компьютерных наук - Вклад [ru]2026-09-21T00:23:28ZВкладMediaWiki 1.43.9https://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=96907Statistical learning theory 20252026-07-10T02:21:28Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12, 13h-17h, room D203<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2024/25&diff=96906Statistical learning theory 2024/252026-07-10T02:20:41Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesday 9h30--10h50 in room M302 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2023/24 last year].<br />
<br />
[https://docs.google.com/spreadsheets/d/1eo4OvNObJicoY-sfMTMUg7dhjbgcpfkGDs9dIehpUA0/edit?usp=sharing Results]<br />
<br />
== Problems exam ==<br />
<br />
Friday 20 December 11h-14h, D507<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, before the lecture. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit HW from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
[https://classroom.google.com/c/NzE5NzA4OTg1ODA4?cjc=imgrl43 Classroom] to submit homeworks. You may submit in English or Russian, as latex or as pictures. Results [https://docs.google.com/spreadsheets/d/1k9hivwCzCp3YcR-1n4WnQow94zyQBCjVcgWYaq-PHx8/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://www.youtube.com/watch?v=N_JUBxw3sZo 21 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/nbxsehlcl8hqodcaho7sg/01slides_all.pdf?rlkey=7u4smvn3jaofhscwrddh6mcoy&st=yb9esz0d&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/svgelu3iwijls092ehqqf/00book_intro.pdf?rlkey=jxdya4290kfc0hfl06b0y7k4b&st=lnv8chxf&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/luee4if0mrd4f440q69hd/01sem.pdf?rlkey=8702taq325mvb4ifh15stvvto&st=sq946cf3&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://www.youtube.com/watch?v=gQm1G3Ep-5s 24 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/darlkflu0p8idh1smsvqc/02sem.pdf?rlkey=9rxky51dscu0d1pvh0h3iun1i&st=whkfpp78&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/d2wuka77bu18j9plivwl5/02sol.pdf?rlkey=yp2eprgxpc7r2antyidjd8qiw&dl=0 sol02]<br />
|-<br />
| [https://www.youtube.com/watch?v=Fk1-QI9PRAI 01 Oct]<br />
|| Kernel perceptron algorithm. Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/bkuydm0u3xonnld8qlbl3/03sem.pdf?rlkey=xg2e9sbpe8c2071pxgcohlcab&st=ezxf2zgq&dl=0 prob03]<br />
|| [https://www.dropbox.com/scl/fi/wjksi4t5r4ng894uiaj8b/03sol.pdf?rlkey=madshl3vupmwkuyzs44ut23ry&st=caroyl3r&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://www.youtube.com/watch?v=ycfYXvmKF0I 08 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/x12se5y3heqtyfzo7qx30/04sem.pdf?rlkey=0hd5hphnbj90jc24nqsw63ka7&st=1zie6tp0&dl=0 prob04] ''update 12.10''<br />
|| [https://www.dropbox.com/scl/fi/g6j0n39zhm1he8kfena8d/04sol.pdf?rlkey=hcg1cr6s4cca9ekqua67ehlhf&st=81bpsm1a&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 15 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/1n9jdc70ia7vu957mls02/05sem.pdf?rlkey=8x89v3fkm1q61b4frirb9nqke&st=7pfvhuq6&dl=0 prob05]<br />
|| <!-- [https://www.dropbox.com/scl/fi/jzm82hqbnzp7931gz8jd2/05sol.pdf?rlkey=o04gco2huwqo4m7rrtp0yd9gl&st=6f0uh0q4&dl=0 sol05] --><br />
|-<br />
| [https://www.youtube.com/watch?v=zHau8Br_UFQ 22 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/15y2x2pq3pp77144nzee5/06sem.pdf?rlkey=72zoca4wgs472df4izvq2dd3t&st=5m9u4q2u&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/w8kc0izfc12sqjyd8hfou/06sol.pdf?rlkey=a09f6yx9e0ifohus9vt2ybthd&st=09qmm3m6&dl=0 sol06]<br />
|-<br />
| [https://youtube.com/live/G5fglRAaXMo 05 Nov]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/701h3asvj5a6kj7d9p1tm/07sem.pdf?rlkey=dsnhc90gp0nd7jqgy3oicds4i&st=fu4nf10i&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/kd3osu95m7bmilv6z6bxm/07sol.pdf?rlkey=9ycz3obscp65uc05pg2dt3zww&st=9d8g3jkf&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://www.youtube.com/watch?v=oU2AzubDXeo 12 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and neural nets with dropout regularization<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/ekrdaba2gzpxdp58yvfwo/08sem.pdf?rlkey=vsljva82ekk6ol6k7w1g87pz6&st=146i9y67&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/fcu1kbczqnxjbvtjpxst7/08sol.pdf?rlkey=irlhu14q6d12poymmc25xmh6q&st=pt7euz9i&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 19 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/t7jv4gulwbdluc278sadi/09sem.pdf?rlkey=wzitr8cwastoq5koyvpsj252o&st=cdik5cp7&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/2pxx6ctc7qv4xpvc4esla/09sol.pdf?rlkey=dg9pncbr6d294gz5me3efzrwp&st=v49ksm24&dl=0 sol09]<br />
|- <br />
| [https://www.youtube.com/watch?v=OgiaWrWh_WA 26 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/y3mbpbcoau67i1nfjg7lr/10sem.pdf?rlkey=mfye4kcfgm9gf6aos6z8nd6q4&st=n1btlv8c&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/5lbthnkjkn35y68ohmhm4/10sol.pdf?rlkey=0w0twp97ohfrlcsspnzfg0wgh&st=74hhghgd&dl=0 sol10]<br />
|- <br />
| [https://youtube.com/live/DUgksR6gOQ8 03 Dec]<br />
|| Losses of neural nets are not locally convex. Gradient descent with stable gradients. ([https://www.youtube.com/watch?v=ygVHVW3y3wM Old recording] about Hessians)<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/ipsngdfvo4bvhofxh4377/16book_lossLandscapeNeuralNet.pdf?rlkey=3018bx9wczc4rpu7xq0wxdc2q&st=64mz3r2p&dl=0 ch16]<br />
|| [https://www.dropbox.com/scl/fi/dc86iowe91nlzf3fu1h71/11sem.pdf?rlkey=87a7uqqpy4n39bcm3dxbsidew&st=t1gemioe&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://youtube.com/live/URjcCXEMPv4 10 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| <br />
||<br />
|-<br />
| 17 Dec<br />
|| Colloquium 9h30 - 12h30 (room D725) and 18h10 - 21h (different building Старая Басманная А-125). [https://www.dropbox.com/scl/fi/e2692ns95pg0kj0m4e0wo/colloqQuest.pdf?rlkey=peey4u0dxz0vohv39a3oc67ft&st=c87t9kqu&dl=0 Rules and questions.] Reserve in [https://docs.google.com/spreadsheets/d/17pJaioWm3Vo2aYB2J3msTyxj9NSovbZHOC_w6Cxvsj8/edit?usp=sharing shedule.]<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Bruno Bauwens: Tuesday 12h -- 20h. Friday 15h -- 17h30. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2026&diff=96905Statistical learning theory 20262026-07-03T16:48:03Z<p>Brbauwens: Новая страница: «== General Information == Lectures: on ?? in Pokrovkaya, (weekly, for 80 min) see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevX…»</p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on ?? in Pokrovkaya, (weekly, for 80 min) see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on ?? (weekly for 80 min) online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the telegram group [to do]. The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.3 * [avg score of intermediate exams] + 0.35 * [score of 2 colloquiums] + 0.2 * [score on the exam] + bonus from quizzes.<br />
<br />
There are are 2 colloquiums, 1 during each of the sessions. <br />
There are 3 intermediate exams, at the end of September, during the session at the end of Okt, at the end of Nov. <br />
<br />
At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the final exam to have the maximal 10/10 for the course, this will be given automatically.<br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12, 13h-17h, room D203<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Wiki_%D0%A4%D0%9A%D0%9D&diff=96904Wiki ФКН2026-07-03T16:40:23Z<p>Brbauwens: Added statistical learning theory course for next year 2026-27</p>
<hr />
<div>__NOTOC__<br />
= Учебные курсы факультета компьютерных наук =<br />
<br />
{| class="wikitable"<br />
|-<br />
! colspan="2" | <div style="text-align: center;">'''Навигация'''</div><br />
|-<br />
! [[#bachelors|Курсы бакалавриата ФКН]]<br />
| <div style="text-align:center">[[#AMI|ПМИ]] · [[#SE|ПИ]] · [[#DSBA|ПАД]] · [[#compds|КНАД]] · [[#EDA|ЭАД]] · [[#DRIP|ДРИП]] · [[#electives|майноры и факультативы]]</div><br />
|-<br />
! colspan="2" | [[#DataCulture|Курсы в рамках проекта Data Culture]] · [[#masters|Курсы магистратуры ФКН]] · [[#other|Курсы других факультетов]] · [[#archive|Архив]]<br />
|}<br />
<br />
== <span id="bachelors">Курсы за 2025/26 учебный год</span> ==<br />
{| class="wikitable courses"<br />
|-<br />
<br />
! width="21%" | 1 курс !! width="21%" | 2 курс !! width="21%" | 3 курс !! width="21%" | 4 курс !! rowspan="2" | '''<span id="electives">майноры и факультативы</span>'''<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="AMI">ПМИ</span>''' <br />
|-<br />
|<br />
<br />
'''М+'''<br />
<br />
[[Линейная_алгебра_и_геометрия_на_ПМИ_2025/2026_(пилотный_поток) | Линейная алгебра и геометрия (пилотный поток)]]<br />
<br />
[[Математический_анализ_1_2025/26_(пилотный_поток) | Математический анализ-1 (пилотный поток)]]<br />
<br />
[[Дискретная математика_на_ПМИ_2025/2026_(пилотный_поток) | Дискретная математика (пилотный поток)]]<br />
<br />
[[Теория_чисел_(пилотный_поток)_2025/26 | Теория чисел (пилотный поток)]]<br />
<br />
[[Алгебра_на_ПМИ_2025/2026_(пилотный_поток) | Алгебра (пилотный поток)]]<br />
<br />
'''М'''<br />
<br />
[[Линейная_алгебра_и_геометрия_на_ПМИ_2025/2026_(основной_поток) | Линейная алгебра и геометрия (основной поток)]]<br />
<br />
[[Математический_анализ_1_2025/26_(основной_поток) | Математический анализ-1 (ПМИ основной поток)]]<br />
<br />
[[DM1PMIbase-2025-26 | Дискретная математика (ПМИ основной поток)]]<br />
<br />
[[Теория_чисел_(основной_поток)_2025/26 | Теория чисел (основной поток)]]<br />
<br />
[[Алгебра_на_ПМИ_2025/2026_(основной_поток) | Алгебра (основной поток)]]<br />
<br />
'''П'''<br />
<br />
[[Язык программирования Python 2025/26 (основной поток) ]]<br />
<br />
[[Алгоритмы и структуры данных 1 (основной поток) (4 модуль) 2025/2026]]<br />
<br />
<br />
'''П+'''<br />
<br />
[[Язык программирования C++ (пилотный поток) ]]<br />
<br />
[[Алгоритмы и структуры данных пилотный поток 2025/2026 | Алгоритмы и структуры данных (пилотный поток)]]<br />
<br />
||<br />
<br />
'''М+'''<br />
<br />
[[Математический_Анализ_2_на_ПМИ_2025/26_(пилотный_поток) | Математический анализ 2 (пилотный поток)]]<br />
<br />
[[Теория_вероятностей_2025/26_(пилотный_поток) | Теория вероятностей (пилотный поток)]]<br />
<br />
[[Математическая_статистика-1_2025/26_(пилотный_поток) | Математическая статистика-1 (пилотный поток)]]<br />
<br />
'''М'''<br />
<br />
[[Теория_вероятностей_на_ПМИ_2025/2026_(основной_поток) | Теория вероятностей (основной поток)]]<br />
<br />
[[Математический_анализ_-_2_(основной_поток)_ПМИ_и_ЭАД_2025/2026 | Математический анализ-2 (ПМИ + ЭАД)]]<br />
<br />
[[Математическая_статистика_2025/26_(основной_поток) | Математическая статистика (основной поток)]]<br />
<br />
&nbsp;<br />
<br />
'''П'''<br />
<br />
[[Мachine Learning 1 | Машинное обучение 1]]<br />
<br />
[[Язык_программирования_Python_(углубленный_курс) | Язык программирования Python (углубленный курс)]]<br />
&nbsp;<br />
<br />
[[CAOS-2025/26 | Архитектура компьютеров и операционные системы]]<br />
<br />
[[Инструменты_промышленной_разработки | Инструменты промышленной разработки]]<br />
<br />
[[Алгоритмы_и_структуры_данных_2_2025/26 | Алгоритмы и структуры данных 2 2025/26]]<br />
<br />
[[Язык_программирования_Go | Язык программирования Go]]<br />
<br />
<br />
[[Комплексный анализ 2025/26]]<br />
<br />
[[Функциональный анализ 2025/26]]<br />
<br />
[[ Дискретная_математика_2_2025/26 | Дискретная математика 2 2025/26 ]]<br />
<br />
[[Основы матричных вычислений 2025/26]]<br />
<br />
[[Дифференциальные уравнения 2025/2026]]<br />
<br />
<br />
||<br />
<br />
[[ PE101-25-26 | Performance engineering 101 ]]<br />
<br />
[[ Стохастический_анализ_(весна_2026) | Стохастический анализ ]]<br />
<br />
[[Функциональное_программирование_2025/2026 | Функциональное программирование]]<br />
<br />
[[ Основы_тензорных_вычислений_(2025/26) | Основы тензорных вычислений ]]<br />
<br />
[[Рекомендательные системы 2025/26 | Рекомендательные системы]]<br />
<br />
[[Глубинное обучение 1 25/26 | Введение в глубинное обучение]]<br />
<br />
[[ Types_25 | Типы в языках программирования ]]<br />
<br />
[[ Безопасность_компьютерных_систем_25/26 | Безопасность компьютерных систем ]]<br />
<br />
[[ Моделирование временных рядов 2025/26 | Моделирование временных рядов ]]<br />
<br />
[[ Случайные_процессы_приложения_2025/26 | Случайные процессы и их приложения ]] <br />
<br />
'''ПМИ / МОП'''<br />
<br />
[[Машинное_обучение_1_25/26 | Машинное обучение 1]]<br />
<br />
[[Математическая_статистика_2_2025/2026 | Математическая статистика 2]]<br />
<br />
[[ML_Research_Seminar_1 | НИС Машинное Обучение и Приложения 1]]<br />
<br />
[[Машинное_обучение_2/2025_2026 | Машинное обучение 2]]<br />
<br />
'''ПМИ / РС'''<br />
<br />
[https://docs.google.com/document/d/1sHfi9_m4hx4y0iIgQEueMCr4qokLxh2Bvyzw_MdE2iI/edit?usp=sharing Распределенные системы]<br />
<br />
[https://docs.google.com/document/d/1Hqu7iVY0FpYVujjNpIClkIMOQy32YWJpJHZKPSrOtmA/edit?usp=sharing НИС Распределенные системы]<br />
<br />
'''ПМИ / ТИ'''<br />
<br />
[[ NIS-TCS-25-26 | НИС Теоретическая информатика ]]<br />
<br />
[[KKTI-25-26 | Комбинаторные конструкции в теоретической информатике]]<br />
<br />
&nbsp;<br />
<br />
||<br />
<br />
[[ PE101-25-26 | Performance engineering 101 ]]<br />
<br />
[http://wiki.cs.hse.ru/%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D1%82%D0%B8%D0%B2%D0%BD%D1%8B%D0%B5_%D0%BC%D0%BE%D0%B4%D0%B5%D0%BB%D0%B8_%D0%BD%D0%B0_%D0%BE%D1%81%D0%BD%D0%BE%D0%B2%D0%B5_%D0%B4%D0%B8%D1%84%D1%84%D1%83%D0%B7%D0%B8%D0%B8_(25/26) Генеративные модели на основе диффузии 25/26]<br />
<br />
[[Большие_языковые_модели_25_26 | Большие языковые модели]]<br />
<br />
[[Theory_of_computation_2025 | Theory of computation]]<br />
<br />
[[Statistical_learning_theory_2025 | Statistical learning theory]]<br />
<br />
[[Statistical_learning_theory_2026 | Statistical learning next year]]<br />
<br />
[[Теория_и_практика_онлайн-экспериментов_25/26 | Теория и практика онлайн-экспериментов]]<br />
<br />
[[Современный_NLP_и_большие_языковые_модели_26 | Современный NLP и большие языковые модели]]<br />
<br />
[[Эффективные_системы_глубинного_обучения_25/26 | Эффективные системы глубинного обучения 25/26]]<br />
<br />
[[ Haskell_26 | Промышленное программирование на Haskell ]]<br />
<br />
[[ zkSNARK_26 | Протоколы доказательств с нулевым разглашением ]]<br />
<br />
[[ Развёртывание_ML-моделей_в_высоконагруженных_системах_26 | Развёртывание ML-моделей в высоконагруженных системах ]]<br />
<br />
'''ПМИ / МОП'''<br />
<br />
[http://wiki.cs.hse.ru/%D0%93%D0%BB%D1%83%D0%B1%D0%B8%D0%BD%D0%BD%D0%BE%D0%B5_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D0%B5_2_2025 Глубинное обучение 2]<br />
<br />
[[ML_Research_Seminar_2 | НИС Машинное Обучение и Приложения 2]]<br />
<br />
[[LSML 2025/2026 | Машинное обучение для больших данных]]<br />
<br />
'''ПМИ / РС'''<br />
<br />
[https://docs.google.com/document/d/1Q2gx-6ugSi9intAW6ZSsLZNrEvl7S5ClyAhn8EjJnRM/edit?usp=sharing НИС Распределенные системы 2]<br />
<br />
[https://gitlab.velkerr.ru/velkerr/msbdp-2026/-/blob/main/README.md?ref_type=heads Методы и системы обработки больших данных]<br />
<br />
'''ПМИ / ТИ'''<br />
<br />
[[ NIS-TCS-25-26 | НИС Теоретическая информатика ]]<br />
<br />
[[ OWF-25-26 | Односторонние функции и их применения ]]<br />
<br />
[[ ConvApprox26 | Выпуклое программирование и аппроксимационные алгоритмы ]]<br />
<br />
[[AT-25-26 | Теория автоматов, формальные языки, регулярные выражения]]<br />
<br />
&nbsp;<br />
<br />
| rowspan="11" | <br />
<!-- майноры и факультативы --><br />
<br />
[[Майнор_Биоинформатика_1_год_2025/26|Биоинформатика 1 год 2025/26]]<br />
<br />
[[Майнор_Биоинформатика_2_год_2025/26|Биоинформатика 2 год 2025/26]]<br />
<br />
[[Введение_в_программирование_25/26|Введение в программирование. Питон 25/26]]<br />
<br />
[[Введение_в_базы_данных_25/26|Введение в базы данных 25/26]]<br />
<br />
[[Основы_глубинного_обучения_25/26|Основы глубинного обучения 25/26]]<br />
<br />
[[Основы_машинного_обучения/2026|Основы машинного обучения 25/26]]<br />
<br />
[[Прикладные_задачи_анализа_данных/2026|Прикладные задачи анализа данных 25/26]]<br />
<br />
[[Kolmogorov_complexity_fall2025|Introduction to Kolmogorov complexity]]<br />
<br />
[[Complexity_theory_2026|Теория вычислений]]<br />
<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="SE">ПИ</span>''' <br />
|-<br />
|<br />
<br />
[[Алгебра_ПИ_2025-2026|Алгебра 25/26]]<br />
<br />
[[Дискретная_математика_25/26|Дискретная математика 25/26]]<br />
<br />
[[Математический_анализ_25/26|Математический анализ 25/26]]<br />
<br />
<br />
|<br />
[[НИС_Методы_и_алгоритмы_защиты_информации_25/26|НИС Методы и алгоритмы защиты информации 25/26]]<br />
<br />
[[Теория_вероятностей_25/26|Теория вероятностей 25/26]]<br />
<br />
[[Математическая_статистика_25/26 | Математическая статистика 2025/26]]<br />
|<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="DSBA">ПАД</span>''' <br />
|-<br />
|<br />
'''1st year DSBA 2025/2026'''<br />
<br />
[[Calculus 1 DSBA 2025/2026 | Calculus 1 (modules 1-4)]]<br />
<br />
[[C++ Programming Language DSBA 2025/2026 | C++ Programming Language (modules 1-3)]]<br />
<br />
[[English DSBA 2025/2026 | English Language (modules 1-4)]]<br />
<br />
[[LAaG DSBA 2025/2026 | Linear Algebra and Geometry (modules 1-4)]]<br />
<br />
[[Discrete Mathematics 1 DSBA 2025/2026 | Discrete Mathematics 1 (modules 1-3)]]<br />
<br />
[[Russian History DSBA 2025/2026 | Russian History (modules 1-4)]]<br />
<br />
[[Economics DSBA 2025/2026 | Economics (modules 2-3)]]<br />
<br />
[[Foundations of Russian Statehood DSBA 2025/2026 | Foundations of Russian Statehood (module 3)]]<br />
<br />
[[Algebra DSBA 2025/2026 | Algebra (module 4)]]<br />
<br />
[[Algorithms and Data Structures 1 DSBA 2025/2026 | Algorithms and Data Structures 1 (module 4)]]<br />
<br />
[[Python for Data Science DSBA 2025/2026 | Python for Data Science (module 4)]]<br />
<br />
||<br />
'''2nd year DSBA 2025/2026'''<br />
<br />
[[Algorithms and Data Structures DSBA 2025/2026 | Algorithms and Data Structures (modules 1-3)]]<br />
<br />
[[Discrete Mathematics 2 DSBA 2025/2026 | Discrete Mathematics 2 (modules 1-2)]]<br />
<br />
[[Calculus 2 DSBA 2025/2026 | Calculus 2 (modules 1-2)]]<br />
<br />
[[Probability Theory DSBA 2025/2026 | Probability Theory (modules 1-2)]]<br />
<br />
[[Introduction to Micro and Macroeconomics DSBA 2025/2026 | Introduction to Micro and Macroeconomics (modules 1-4)]]<br />
<br />
[[Computer Architecture and Operating Systems DSBA 2025/2026 | Computer Architecture and Operating Systems (modules 3-4)]]<br />
<br />
[[Mathematical Statistics DSBA 2025/2026 | Mathematical Statistics (modules 3-4)]]<br />
<br />
[[Machine Learning 1 DSBA 2025/2026 modules 3-4 | Machine Learning 1 (modules 3-4)]]<br />
<br />
[[Differential Equations DSBA 2025/2026 | Differential Equations (modules (3-4)]]<br />
<br />
'''Minors'''<br />
<br />
[[Business and Management in Global Context DSBA 2025/2026 | Business and Management in Global Context (modules 1-4)]]<br />
<br />
[[Introduction to Finance DSBA 2025/2026 | Introduction to Finance (modules 1-4)]]<br />
<br />
||<br />
'''3rd year DSBA 2025/2026'''<br />
<br />
[[Databases DSBA 2025/2026 | Databases (modules 1-2)]]<br />
<br />
[[Optimization Methods DSBA 2025/2026 | Optimization Methods (modules 1-2)]]<br />
<br />
[[Machine Learning 1 DSBA 2025/2026 | Machine Learning 1 (modules 1-2)]]<br />
<br />
[[Stochastic processes and applications DSBA 2025/2026 | Stochastic processes and applications (modules 1-2)]]<br />
<br />
[[Time Series Analysis DSBA 2025/2026 | Time Series Analysis (modules 3-4)]]<br />
<br />
[[Deep Learning DSBA 2025/2026 | Deep Learning (modules 3-4)]]<br />
<br />
'''Minors'''<br />
<br />
[[Introduction to Entrepreneurship DSBA 2025/2026 | Introduction to Entrepreneurship (business minor, modules 1-2)]]<br />
<br />
[[Information Systems Management DSBA 2025/2026 | Information Systems Management (business minor, modules 3-4)]]<br />
<br />
[[Econometrics DSBA 2025/2026 | Elements of Econometrics (finance minor, modules 1-4)]]<br />
<br />
'''Specialization Data Science in Business'''<br />
<br />
[[Research Seminar "Data Analysis in Business" DSBA 2025/2026 | Research Seminar "Data Analysis in Business“ (modules 1-4)]]<br />
<br />
[[Data Analysis in Business DSBA 2025/2026 | Data Analysis in Business (modules 1-2)]]<br />
<br />
[[Investment Management for Business DSBA 2025/2026 | Investment Management for Business (modules 3-4)]]<br />
<br />
'''Specialization Data Science in Finance'''<br />
<br />
[[Research Seminar "Data Science in Financial Economics" DSBA 2025/2026 | Research Seminar "Data Science in Financial Economics" (modules 1-4)]]<br />
<br />
[[Financial Mathematics DSBA 2025/2026 | Financial Mathematics (modules 1-2)]]<br />
<br />
[[Risk Management in Bank DSBA 2025/2026 | Risk Management in Bank (modules 3-4)]]<br />
<br />
'''Specialization Data Analysis in Applied Research'''<br />
<br />
[[Research Seminar "Data Science in Applied Research" DSBA 2025/2026 | Research Seminar "Data Science in Applied Research" (modules 1-4)]]<br />
<br />
[[Methods of Mathematical Modeling DSBA 2025/2026 | Methods of Mathematical Modeling (modules 1-2)]]<br />
<br />
[[Applied Statistics for Machine Learning DSBA 2025/2026 | Applied Statistics for Machine Learning (modules 3-4)]]<br />
<br />
'''Electives (modules 3-4)'''<br />
<br />
[[Operations Research and Game Theory DSBA 2025/2026 | Operations Research and Game Theory]]<br />
<br />
[[Advanced Statistical Methods DSBA 2025/2026 | Advanced Statistical Methods]]<br />
<br />
[[Recommender Systems DSBA 2025/2026 | Recommender Systems]]<br />
<br />
||<br />
'''4th year DSBA 2025/2026'''<br />
<br />
[[Strategy DSBA 2025/2026 | Strategy (modules 1-3)]]<br />
<br />
[[Statistical Methods for Market Research DSBA 2025/2026 | Statistical Methods for Market Research (modules 2-3)]]<br />
<br />
'''Electives block 1 (modules 1-3)'''<br />
<br />
[[Core Management Concepts DSBA 2025/2026 | Core Management Concepts]]<br />
<br />
[[Asset Pricing and Financial Markets DSBA 2025/2026 | Asset Pricing and Financial Markets]]<br />
<br />
'''Electives block 2 (modules 1-2)'''<br />
<br />
[[Introduction to the Study of Language for Computer Scientists DSBA 2025/2026 | Introduction to the Study of Language for Computer Scientists]]<br />
<br />
[[Quantitative Finance DSBA 2025/2026 | Quantitative Finance]]<br />
<br />
[[Information Security Risk Management DSBA 2025/2026 | Information Security Risk Management]]<br />
<br />
[[Generative Models in Machine Learning DSBA 2025/2026 | Generative Models in Machine Learning]]<br />
<br />
[[Natural Language Processing DSBA 2025/2026 | Natural Language Processing]]<br />
<br />
'''Electives block 3 (module 3)'''<br />
<br />
[[Information Systems DSBA 2025/2026 | Information Systems]]<br />
<br />
[[Cоmputer Vision DSBA 2025/2026 | Cоmputer Vision]]<br />
<br />
'''Research Seminar Electives (modules 1-3)'''<br />
<br />
[[Research Seminar "Data Analysis in the Natural Sciences" DSBA 2025/2026 | Research Seminar "Data Analysis in the Natural Sciences"]]<br />
<br />
[[Research Seminar "Data Science in Financial Markets" DSBA 2025/2026 | Research Seminar "Data Science in Financial Markets"]]<br />
<br />
[[Research Seminar "Data analysis in complex systems" DSBA 2025/2026 | Research Seminar "Data analysis in complex systems"]]<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="compds">КНАД</span>''' <br />
|-<br />
|<br />
[[Программирование на С++ КНАД ВСН 25-26]]<br />
<br />
[[Дискретная_Математика_КНАД_2025/26 | Дискретная математика 2025/26 (КНАД)]]<br />
<br />
[[Программирование_на_Python_КНАД_25/26 | Программирование на Python 25/26 (КНАД)]]<br />
<br />
[[Python_для_сбора_и_анализа_данных_КНАД_25/26 | Python для сбора и анализа данных КНАД 25/26]]<br />
<br />
[[Алгоритмы и структуры данных - 1 2025/2026 2 модуль (КНАД)]]<br />
<br />
[[Алгоритмы и структуры данных 2025/2026 4 модуль КНАД]]<br />
<br />
[[Линейная алгебра КНАД 25/26]]<br />
<br />
[[Математический Анализ КНАД 25/26]]<br />
<br />
[[ИПР КНАД 25/26]]<br />
<br />
||<br />
[[Алгебра КНАД 2025/2026 | Алгебра]]<br />
<br />
[[Алгоритмы_и_структуры_данных_2_КНАД_25/26 | Алгоритмы и структуры данных-2]]<br />
<br />
[[Математический_анализ_2_КНАД_2025/26 | Математический анализ-2 2025/26 (КНАД)]]<br />
<br />
[[Теория_вероятностей_КНАД_2025/26 | Теория вероятностей 2025/26 (КНАД)]]<br />
<br />
[[ACOS_COMPDS_2025/26 | Архитектура Компьютера и Операционные Системы]]<br />
<br />
[[Математическая_статистика_КНАД_2025/26 | Математическая статистика 2025/26 (КНАД)]]<br />
<br />
<br />
||<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="EDA">ЭАД</span>''' <br />
|-<br />
|<br />
[[Алгоритмы и структуры данных-1 2025/2026 2 модуль (ЭАД)]]<br />
<br />
[[Алгоритмы и структуры данных-1 2025/2026 4 модуль (ЭАД)]]<br />
<br />
[[Программирование на С++ ЭАД 25/26]]<br />
<br />
[[Язык программирования Python 2025/26 (ЭАД) ]]<br />
<br />
[[DM1EAD-2025-26 | Дискретная математика 2025/26 (ЭАД)]]<br />
<br />
[[Линейная_алгебра_и_геометрия_на_ПМИ_2025/2026_(основной_поток) | Линейная алгебра и геометрия]]<br />
<br />
[[Математический_анализ_1_2025/26_(основной_поток)_ЭАД | Математический анализ-1 (ЭАД)]]<br />
||<br />
<br />
[[Алгоритмы и структуры данных 2 ЭАД 25/26 | Алгоритмы и структуры данных-2]]<br />
<br />
||<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="DRIP">ДРИП</span>''' <br />
|-<br />
|<br />
[[Алгоритмы и структуры данных-1 (ДРИП)]]<br />
<br />
[[Ddip2529 | Линейная алгебра и геометрия]]<br />
<br />
[[DM_DRIP-2025-26 | Дискретная математика 2025/26 (ДРИП)]]<br />
<br />
||<br />
[[Теория_вероятностей_ДРИП_25/56 | Теория вероятностей 2025/26 (ДРИП)]]<br />
<br />
||<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
|-<br />
| colspan="4" style="text-align: center;" | '''<span id="DRIP">РИЦП</span>''' <br />
|-<br />
|<br />
[[Математика для компьютерной графики (РИЦП)]]<br />
<br />
<br />
||<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
||<br />
&nbsp;<br />
<br />
|}<br />
<br />
== <span id="DataCulture">Курсы в рамках проекта [https://www.hse.ru/dataculture/ Data Culture]</span> ==<br />
<br />
{| class="wikitable"<br />
|- <br />
! Осенний семестр !! Весенний семестр<br />
|-<br />
| <br />
|<br />
|}<br />
<br />
== <span id="masters">Курсы магистратуры ФКН</span> ==<br />
{| class="wikitable" <br />
|-<br />
! Ссылка !! Дисциплина !! Год обучения<br />
|-<br />
| [[ NIS-TCS-25-26 | НИС Теоретическая информатика ]] || НИС ТИ || СКН, 1-2 год<br />
|-<br />
| [[MOTV_2025 | Mathematical foundations of probability theory]] || Math of Machine Learning, MML || 1 year<br />
|-<br />
| [[MC_2025 | Markov Chains]] || Math of Machine Learning, MML || 1 year<br />
|-<br />
| [[Sample_2026 | Sampling and Generative Modeling ]] || Math of Machine Learning, MML || 1 year<br />
|}<br />
<br />
== <span id="other">Курсы других факультетов</span> ==<br />
<br />
{| class="wikitable" <br />
|-<br />
! Дисциплина !! Курс !! Период<br />
|-<br />
| [[icef-scalc-2025-fall|Stochastic Calculus Fall 2025]] ||icef, master 1 year || 2 module<br />
|-<br />
| [[smef_metrics_2025-26|Эконометрика (продвинутый курс) 2025-26]] || фэн, магистратура, 1 курс || 3-4 модуль<br />
|-<br />
|}<br />
<br />
== Факультативы ==<br />
<br />
{| class="wikitable" <br />
|-<br />
! Дисциплина !! Период<br />
|-<br />
| [[Аналитическая_теория_чисел:_приложения_комплексного_анализа_25/26 | Аналитическая теория чисел: приложения комплексного анализа 25/26]] || 1-2 модуль<br />
|-<br />
|}<br />
<br />
= Архив до 2024/25 учебного года включительно =<br />
[[Wiki ФКН/Архив]]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95735Theory of computation 20252026-03-09T17:22:03Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/89836244481?pwd=oJ7zni9Iv1BNVKaJ9voOnZyzfPT4ci.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades].<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1KIi6sFhS6M0BWr4xhU6v44BJyi53L7kF/view?usp=drive_link problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task previous year.](Will be updated.) [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|- <br />
|| 01.04 || I do not have a good programming exercise. Everyone gets 7/10. If you want more, then do a colloquium on April 1st (or earlier). Write me in telegram. || [https://drive.google.com/file/d/1jiUIt0fm3QNSCnSwU0i1vCiOAwlx0yqL/view?usp=drive_link Colloquium questions] <br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|-<br />
| Seminar 12 || [https://rutube.ru/video/private/b2baba2d5e37a15765dc7e0681b96c4c/?p=12yI6HG1IkPFfHYy9fI22w link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95734Theory of computation 20252026-03-09T17:21:31Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/89836244481?pwd=oJ7zni9Iv1BNVKaJ9voOnZyzfPT4ci.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades].<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1KIi6sFhS6M0BWr4xhU6v44BJyi53L7kF/view?usp=drive_link problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task previous year.](Will be updated.) [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|- <br />
|| 01.04 || I do not have a good programming exercise. Everyone gets 7/10. If you want more, then do a colloquium on April 1st (or earlier). Write me in telegram. | [https://drive.google.com/file/d/1jiUIt0fm3QNSCnSwU0i1vCiOAwlx0yqL/view?usp=drive_link Colloquium questions] <br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|-<br />
| Seminar 12 || [https://rutube.ru/video/private/b2baba2d5e37a15765dc7e0681b96c4c/?p=12yI6HG1IkPFfHYy9fI22w link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95582Theory of computation 20252026-02-27T11:15:17Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1KIi6sFhS6M0BWr4xhU6v44BJyi53L7kF/view?usp=drive_link problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task previous year.](Will be updated.) [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95581Theory of computation 20252026-02-27T11:14:48Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1KIi6sFhS6M0BWr4xhU6v44BJyi53L7kF/view?usp=drive_link list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task previous year.](Will be updated.) [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95580Theory of computation 20252026-02-27T11:10:33Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95414Theory of computation 20252026-02-17T15:08:48Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya room S834 and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens].<br />
<br />
Seminars: Friday 14h40 - 16h00 in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari (on 13.03 also in S834). <br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95413Theory of computation 20252026-02-17T15:04:13Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. FTP algorithms for planar graphs. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Problems that are FPT on graphs with small treewidth. (Or something needed in the project.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=95412Theory of computation 20252026-02-17T14:48:14Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 27.02] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 06.03] || Discussion of the programming project. Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 13.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94334Theory of computation 20252025-12-21T14:25:15Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
All [https://docs.google.com/spreadsheets/d/1saxWb6gGXJV7VL7uQGvg8YO6xS21uIgpBk4C8wwimvE/edit?usp=sharing grades]. <br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94332Statistical learning theory 20252025-12-21T08:40:48Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12, 13h-17h, room D203<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94330Statistical learning theory 20252025-12-20T14:05:15Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
[https://www.dropbox.com/scl/fi/cmxyn0t1k0hutvngdcq5m/exam.pdf?rlkey=8miw4h0r0f2gnv8pww6kxv4ee&st=crpb7zcq&dl=0 Tasks]<br />
<br />
Date: Saturday 20.12, 13h-17h, room D203<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94324Statistical learning theory 20252025-12-19T16:25:30Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12, 13h-17h, room D203<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC, it is a computer room), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94311Theory of computation 20252025-12-18T16:09:46Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
2 or 3 copies of each of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (If you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94310Theory of computation 20252025-12-18T16:08:42Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
Dec 23 at 9h30 in D203. <br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
A 2 or 3 copies of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94288Theory of computation 20252025-12-17T16:12:36Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
A 2 or 3 copies of the books by Sipser's, Arora&Barak, Mertens&Moore, will be available, as well as a few prints of chapters 7 & 8 of Sipser's book. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94287Statistical learning theory 20252025-12-17T16:02:44Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
If you have a passing grade without attending the exam, you may skip the exam and I will mark you as present by default. <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94255Statistical learning theory 20252025-12-15T15:22:31Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: Saturday 20.12<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 or 5 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
-- [https://www.dropbox.com/scl/fi/nsmp6azkhh63s5x0chkby/exampleExam.pdf?rlkey=b95kbp6ujm6d4gsn8yp2blho2&st=t1ehcbap&dl=0 Example] of an exam (a bit easier, during COVID). <br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94145Statistical learning theory 20252025-12-09T15:44:55Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| [https://rutube.ru/video/2433020134d99071e0a591a23b595f00/ 09 Dec]<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94139Statistical learning theory 20252025-12-09T11:21:36Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| 09 Dec<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: Tuesday 15h-21h Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94138Statistical learning theory 20252025-12-09T11:20:04Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| 09 Dec<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| Consult 15.12<br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94137Statistical learning theory 20252025-12-09T11:19:28Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| 09 Dec<br />
|| Finnish previous lecture. Optional: a label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94136Statistical learning theory 20252025-12-09T11:18:51Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel in overparameterized nets. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| 09 Dec<br />
|| Label dependent risk bound for overparameterized nets.<br />
|| <br />
|| [https://www.dropbox.com/scl/fi/vqt63hgp15ea0tqrsy8li/18book_fineGrainedAnalyses.pdf?rlkey=ufsc4tev9h7w8lev13ww8ntyo&st=v03nuk8x&dl=0 ch18]<br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94126Theory of computation 20252025-12-08T17:27:51Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] upd 8.12<br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94096Theory of computation 20252025-12-07T10:47:55Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/757b101d76445586b9f5e752f6028e33/?p=enzOqSoF4YmMdI3rnqEO9A recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] <br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] This year's [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2 rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|-<br />
| Seminar 11 || [https://rutube.ru/video/private/5536256c687688c3d824a0bd6daba00e/?p=0Vn3ddYFyVunqH09vaiU-A link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94093Kolmogorov complexity fall20252025-12-06T20:31:30Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/h23o9y87rrybderjirri3/algStatNotes.pdf?rlkey=u2gq3vzfxy4xfo2q524keiyez&st=xe5pfn0c&dl=0 ch09] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 12.12 || Optional lecture: 1-way functions exist if and only if with symmetry of information holds for pK^t-complexity holds with high probability for poly time sampled x and y. From [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf this] paper. || || ||<br />
|- <br />
|| 24.12 || 17h-20h Exam and colloquium. [https://www.dropbox.com/scl/fi/9bhuvect79u9pxnxmf87j/col.pdf?rlkey=pizqnk66a4vv0vcaypqg8nzyd&st=2vx46jm7&dl=0 questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94073Kolmogorov complexity fall20252025-12-05T16:59:43Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 12.12 || Optional lecture: 1-way functions exist if and only if with symmetry of information holds for pK^t-complexity holds with high probability for poly time sampled x and y. From [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf this] paper. || || ||<br />
|- <br />
|| 24.12 || 17h-20h Exam and colloquium. [https://www.dropbox.com/scl/fi/9bhuvect79u9pxnxmf87j/col.pdf?rlkey=pizqnk66a4vv0vcaypqg8nzyd&st=2vx46jm7&dl=0 questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94071Kolmogorov complexity fall20252025-12-05T16:59:20Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || Optional lecture: 1-way functions exist if and only if with symmetry of information holds for pK^t-complexity holds with high probability for poly time sampled x and y. From [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf this] paper. || || ||<br />
|- <br />
|| 24.12 || 17h-20h Exam and colloquium. [https://www.dropbox.com/scl/fi/9bhuvect79u9pxnxmf87j/col.pdf?rlkey=pizqnk66a4vv0vcaypqg8nzyd&st=2vx46jm7&dl=0 questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94070Kolmogorov complexity fall20252025-12-05T16:58:32Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || Optional lecture: 1-way functions exist if and only if with symmetry of information holds for pK^t-complexity holds with high probability for poly time sampled x and y [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf paper]. || || ||<br />
|- <br />
|| 24.12 || 17h-20h Exam and colloquium. [https://www.dropbox.com/scl/fi/9bhuvect79u9pxnxmf87j/col.pdf?rlkey=pizqnk66a4vv0vcaypqg8nzyd&st=2vx46jm7&dl=0 questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94069Kolmogorov complexity fall20252025-12-05T16:55:54Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || Optional lecture: 1-way functions exist if and only symmetry of information holds for pK^t-complexity, [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf paper]. || || ||<br />
|- <br />
|| 24.12 || 17h-20h Exam and colloquium. [https://www.dropbox.com/scl/fi/9bhuvect79u9pxnxmf87j/col.pdf?rlkey=pizqnk66a4vv0vcaypqg8nzyd&st=2vx46jm7&dl=0 questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94068Kolmogorov complexity fall20252025-12-05T16:51:41Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || Optional lecture. 1-way functions exist if and only symmetry of information holds for pK^t-complexity, [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf paper]. || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=94067Kolmogorov complexity fall20252025-12-05T16:51:17Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || 1-way functions exist if and only symmetry of information holds for pK^t-complexity, [https://wrap.warwick.ac.uk/id/eprint/174719/1/WRAP-A-duality-between-one-way-functions-and-average-case-symmetry-of-information-Oliveira-2023.pdf paper]. || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=94061Theory of computation 20252025-12-05T15:11:19Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/18c6e2b7e1222f3411afbc933f317631/?p=9KNQCTTbSP-nKD-gg6tDOw recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] <br />
|-<br />
|| [https://rutube.ru/video/fdfba34cca0e6d23b7382e883f138aa2/ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] Last year's [https://youtube.com/live/E4aNlvbYoLQ rec]. || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94043Statistical learning theory 20252025-12-04T18:13:31Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/8b415o6xl07jkzbf9ey01/17book_implicitRegularization.pdf?rlkey=byvcf6qg0ogqm9ipq4tznkfhl&st=qin9wihv&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11]<br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=94016Statistical learning theory 20252025-12-02T15:19:02Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://rutube.ru/video/eec59ac42390d487b0b9e20258ddc7de 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --><br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=93961Kolmogorov complexity fall20252025-11-29T11:45:11Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| [https://drive.google.com/file/d/10rYr0v8MRHO_ldiV3d1xYM6DXOAC_zEO/view?usp=drive_link 28.11] || Time bounded (decision) complexity, hash functions and language compression, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || TBA, (something related to cryptography), optional || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=93960Theory of computation 20252025-11-29T11:40:00Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] This year's [https://rutube.ru/video/private/18c6e2b7e1222f3411afbc933f317631/?p=9KNQCTTbSP-nKD-gg6tDOw recording]. || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] <br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=93959Theory of computation 20252025-11-29T11:39:12Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] <br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] (This year's [https://rutube.ru/video/private/18c6e2b7e1222f3411afbc933f317631/?p=9KNQCTTbSP-nKD-gg6tDOw recording].)|| [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Theory_of_computation_2025&diff=93958Theory of computation 20252025-11-29T11:12:37Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures: Friday 13h00 - 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. Starting 19.09.<br />
<br />
Seminars: Friday 14h40 - 16h00 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room, and in [https://us06web.zoom.us/j/84205097860?pwd=PQpzn1Oqf9G2fjO2vwcUVsMdCnRUuc.1 zoom] by Prof. Subin Pulari<br />
<br />
Telegram group for announcements and discussions [https://t.me/+7BOrITRebjk5MTc0 invite link.] The course is similar to [http://wiki.cs.hse.ru/Theory_of_computation_2024 last year's one].<br />
<br />
For students programming engineering this course is called "computational complexity theory" and the course has an extra part in the 3rd module. <br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture. Submit in pdf or fotos of handwritten text in [https://classroom.google.com/c/ODEwMjk4NDQxMTU1?cjc=n5dttaev google class]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
Late policy: 1 homework can be submitted at most 24h late without explanations.<br />
<br />
All homework scores are available in the following Google Sheet: [https://docs.google.com/spreadsheets/d/1QvyOh7kk7W-IhgMNcyFJ5NUNj9Tw-9z12ora_ZBW8eA/edit?usp=sharing link]<br />
<br />
= Course Materials =<br />
<br />
The main reference is Sipser's book "Introduction to the theory of computation", chapters 3, 4, 7–9.<br />
<br />
If you need some background in math, consider<br />
<!-- [http://www.cs.elte.hu/~lovasz/dmbook.ps Lecture notes: Discrete Mathematics], L. Lovasz, K. Vesztergombi and --><br />
[http://rubtsov.su/public/DM-HSE-Draft.pdf Лекции по дискретной математике].<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Problem list<br />
|-<br />
|| [https://www.youtube.com/watch?v=-VIr385nKVk 19.09] || Turing machines, multitape Turing machines, connection between them. Universal Turing machine. Examples. Time and space complexity. Complexity classes P, PSPACE, EXP. (Recording from previous year.) [https://drive.google.com/file/d/1Xby_fNyeVwZtVY6yfVJLBYNCaZVbOP61/view?usp=drive_link Notes]|| [https://drive.google.com/file/d/1bf3hhE82uw5D0bQGajXHl1-FhhyG-XnY/view?usp=drive_link problem list 1]<br />
|-<br />
|-<br />
|| [https://www.youtube.com/watch?v=tnq5BkcGfk0 26.09] || Undecidability of the Halting problem. Time and space hierarchy theorems. See notes above. || [https://drive.google.com/file/d/1iyba3kM98oJCz5yvKmWRb0oOpIAhFPef/view?usp=drive_link problem list 2] <br />
|-<br />
|| [https://www.youtube.com/watch?v=VR_A62kFXH8 03.10] || Complexity class NP. Examples. Non-deterministic machines and another definition of NP. Polynomial reductions. NP-hardness and NP-completeness. [https://drive.google.com/file/d/1chzfYehNELzNkDMDjmRo9gfIXv_9FYeG/view?usp=drive_link Notes], [https://drive.google.com/file/d/1leJcc3UqC-WIGGN3G7Zm2Y0jt84q13o1/view?usp=drive_link tex]. || [https://drive.google.com/file/d/17AF1Bj5I_S0XzwXkM4IeOugxX2Kr3i2u/view?usp=drive_link problem list 3]<br />
|-<br />
|| [https://rutube.ru/video/585774c956d226dde80929ca0b39f13e 10.10] || NP-completenes of independent-set, NAE-3SAT, 3colorability, subsetsum, knapsack problem. Notes above. || [https://drive.google.com/file/d/1dVUT2KZ9hAyGBZITDJh3xtfLb992aX9L/view?usp=drive_link problem list 4] upd 10.10<br />
|-<br />
|| [https://rutube.ru/video/53f86ac57c92d812edb493b280e31a07 17.10] || Circuits 1: examples and all functions have exponential circuits. Classes P/poly, AC^i and NC^i. Some functions have exponential circuit complexity. <!-- NC1 = Boolean formulas of polynomial size. Addition in AC0. Multiplication is in NC1. P is in P/poly. 3SAT is NP-complete.--> [https://drive.google.com/file/d/1Cqr4A6ohPLIaAVocG3jE6GYsNz4XgA4r/view?usp=drive_link Notes.pdf] upd 28.10, [https://drive.google.com/file/d/1rzL0-_fKq9ZbSu_zjTUUiM_cQNUGUX05/view?usp=drive_link tex]. || [https://drive.google.com/file/d/1uB12iMZptqzcoLpF6hZT-8X1Xh9-1nm4/view?usp=drive_link problem list 5]<br />
|-<br />
|| 23.10 || Circuits 2: NC0 = functions that depend on a constant number of inputs. P is in P/poly. 3SAT is NP-complete. Seminar: addition in AC0. Multiplication is in NC1. Notes above. Last year's [https://www.youtube.com/watch?v=Jqy89FPbFj4 video]. || see list 5<br />
|-<br />
|| [https://drive.google.com/file/d/19vsMFBQfun7mf9URNF-F2R9JWbPlNN81/view?usp=drive_link 24.10] || Directed Reachability is in SPACE(log^2 n). TQBF problem, its PSPACE-completeness. PSPACE = NPSPACE. NSPACE(s(n)) is in SPACE(s(n)^2) for space constructible s.|| [https://drive.google.com/file/d/1j9XCMFYwsqcqIzGgKGSgDpJvmtz4-Wx8/view?usp=drive_link problem list 6] upd 28.10<br />
|-<br />
|| [https://youtube.com/live/W_uZuQXm53c 07.11] || Oracle computation definitions. There exists an oracle ''A'' for which P<sup>''A''</sup> = NP<sup>''A''</sup>. There is an oracle B such that P<sup>''B''</sup> is not equal to NP<sup>''B''</sup>. || [https://drive.google.com/file/d/1v8dh4EAeAtq1et4ya_Q1VBO77_m3a6WS/view?usp=drive_link problem list 7]<br />
|- <br />
|| [https://youtube.com/live/ALO6r52wuIU 14.11] || Probabilistic computation. Probabilistic machines, the class BPP, invariance of the definition BPP for different thresholds, RP, coRP, PP, ZPP. BPP is in P/poly. Most of it is also [https://www.youtube.com/watch?v=YSMgVbOqB-8&list=PLm3J0oaFux3YL5vLXpzOyJiLtqLp6dCW2&index=23 here] and [https://www.cs.cmu.edu/afs/cs/academic/class/15859-f04/www/scribes/lec2.pdf scribe1] [https://lucatrevisan.github.io/cs278-04/notes/lecture08.pdf scribe2] || [https://drive.google.com/file/d/1pMn1rzOKjnmjvW4X1WJA-jauQv08YFIh/view?usp=drive_link problem list 8] upd 20.11<br />
|-<br />
|| [https://rutube.ru/video/private/82810d471de9883b9215d0c9a3235c44/?p=8fTMLwkOD-WvpiQM1ACE0A 21.11] || Approximation algorithms. Definition c-approximation algorithm. 2-approximation for vertex cover and greedy vertex cover is not optimal. (ln n + 1)-approximation for set cover. PTAS for the makespan problem. Based on [https://www.youtube.com/watch?v=MEz1J9wY2iM&pp=ygUYYXBwcm94aW1hdGlvbiBhbGdvcml0aG1z MIT lecture].|| <br />
[https://drive.google.com/file/d/1TPC6D_81PrklhjlTfrfnEzE4_5vm3vPS/view?usp=drive_link problem list 9]<br />
|-<br />
|| [https://youtube.com/live/r9rP3ZojeqY 28.11] || Parameterized complexity: The classes FPT and XP. Kernelization. Examples for vertex cover. [https://drive.google.com/file/d/1W9SU24HW0r5QhugzmrghpkzJUFuC8whq/view?usp=sharing Notes.] || [https://drive.google.com/file/d/12GnrF5dpVZjMsHfG4Y6jmT3P6jsc8jjm/view?usp=drive_link problem list 10] <br />
|-<br />
|| [https://youtube.com/live/E4aNlvbYoLQ 05.12] || Parameterized complexity: W-hierarchy, hardness from the exponential time hypothesis. [https://drive.google.com/file/d/1f-085Gr7E01HepUR-brGJDet-fXJzUwb/view?usp=sharing presentation] [https://drive.google.com/file/d/1OIw7h31N2tt-npC0NrvVgNhYGw_HlJa_/view?usp=sharing Notes] || [https://drive.google.com/file/d/1xhUYMsj0iV2TMZfR8FtkxWUNlcCjdS-N/view?usp=sharing problem list 11] <br />
|- <br />
|| 12.12 || ''Colloquium.'' Also on 19.12. [https://drive.google.com/file/d/1no8CbHKJluGRx7FROpSS7BqXdjOVgdT1/view?usp=sharing Rules and questions.] (Same as last year.) Reserve a [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing slot]. || [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam] <br />
|}<br />
<br />
Artem Parfenov's [https://drive.google.com/file/d/16_ZP0X9wwza1RsrXZZGQsx5Igwk_vY9r/view?usp=sharing lecture summaries] [https://drive.google.com/drive/folders/1XsNL2B69akd3A9qaOEgC09NiZXLP8Wbz?usp=drive_link source] (Disclaimer: I did not check them):<br />
<br />
{| class="wikitable"<br />
|-<br />
! Date !! Software engineering: parameterized complexity, FPT algorithms !! Problem list<br />
|-<br />
|| [https://youtube.com/live/xDaTpuVF0Es 05.03] || Recap from last lecture. More examples of kernels: linear programming kernel for vertex cover problem. [https://www.dropbox.com/scl/fi/x5gwdae4ny4u3zixoy4pw/student_vc_branching.ipynb?rlkey=5oowq1u3jhy490s84ogojg1s7&dl=0 programming task.] [https://www.dropbox.com/s/zzzaulpmzql7x7u/parameterizedComplexity.pdf?dl=0 presentation].|| [https://drive.google.com/file/d/1sXA5F4gUZ-KvigQspz0ScVLOA38ZcxHb/view?usp=sharing problem list 12] <br />
|-<br />
|| [https://youtube.com/live/TBX8sWx7nO0 12.03] || Linear programming kernel for VC, color coding, dynamic programming. Colorcoding [https://drive.google.com/file/d/1RMoIpd6p_L1XXOVeV36g0pdJdbtltbPF/view?usp=sharing from slide 39] || [https://drive.google.com/file/d/16wjIGqbS5UvEUZCxuUfGROUxk2RlLqZl/view?usp=sharing problem list 13] <br />
|-<br />
|| 19.03 || Optional: problems that are FPT on graphs with small treewidth.(No recording, sorry.) || [https://drive.google.com/file/d/1mYkshv6q-c_5nY5qfHgosKxBd1zDDvGK/view?usp=sharing problem list 14]<br />
|}<br />
<br />
{| class="wikitable"<br />
|-<br />
! Seminars (2025) !! Recording link<br />
|-<br />
| Seminar 4 || [https://rutube.ru/video/private/6f4f908ef52b8af8e5bace950d8a9c0c/?p=5GnphSR3t1Jf4a9Mpqg2Zw link]<br />
|-<br />
| Seminar 5 || Part I: [https://rutube.ru/video/private/992b26e2a8e6c95c662d8d246c7f3e0d/?p=uYcEmJungpW_YPkyLlGMOA link] <br> Part II: [https://rutube.ru/video/private/9d17a54647262d3c63187aad06fab0b8/?p=8WfJJBBJUYFfE9tQkBJ1MA link]<br />
|-<br />
| Seminar 6 || [https://rutube.ru/video/private/6aa2f287bc39727ca82f4c713d2cc463/?p=2s6opSEa5_CjGYmJI3sOLg link]<br />
|-<br />
| Seminar 7 || [https://rutube.ru/video/private/88af4a5c6eb2699db4475dbcf88884b7/?p=k_1vuYG_fWnOAQld9-0aCQ link]<br />
|-<br />
| Seminar 8 || [https://rutube.ru/video/private/8978247dc6dcaaa8ef0e04ac3f4333af/?p=EGFH7-YuCIEiH2BYyKUqrg link]<br />
|-<br />
| Seminar 9 || [https://rutube.ru/video/private/2bfd76c0422f371bc506328c35c9e670/?p=3TXJi4AYqVMD1Idt8HyFmg link]<br />
|-<br />
| Seminar 10 || [https://rutube.ru/video/private/1ab525cc4cdeef5d66b736d6bce5887b/?p=Tvxh3q1LON2Y9O_AvXck1Q link]<br />
|}<br />
<br />
<br />
<br />
[https://www.youtube.com/playlist?list=PL8EKo81hBCTGfNPpZlIe7USiEPpZGYDte Recordings last year]<br />
<br />
= Exam =<br />
<br />
5 or 6 questions with the same difficulty as the homework questions. You have 3 hours time. <br />
<br />
Each year, 1 of the questions is to prove that some problem is NP-complete. Do not forget to say why the problem is in NP. <br />
<br />
Copies of Sipser's book, Arora&Barak, Mertens&Moore, will be available. (I you have these books or printed parts of them, please bring it.) Also, personal handwritten notes are allowed, but nothing else. [https://www.dropbox.com/s/37gdsbv2it8omnm/tc-sample-exam.pdf?dl=0 Sample exam]. <br />
<br />
<br />
= Project (for PI students) =<br />
<br />
During the 3rd module Januari till March 2024 there are projects where you need to implement algorithms from parameterized complexity. (For example, for the vertex cover algorithm and disjoint paths problems.) A grader will check whether your algorithm reaches certain time limits. <br />
<br />
There are 3 tasks: 2 of them about branching and kernelization, 1 task about linear programming bounds. See the table with lectures. The tasks have equal weight for the grade. <br />
<br />
Deadline March 31st, 23h59. <br />
<br />
<br />
= Additional reading = <br />
<br />
Recall that the most important book for our course is ''Sipser, Introduction to the theory of computation'' 3rd edition, 2013, chapters 3, 4, 7–9. This book is intended for Bachelor students. <br />
<br />
The following book is popular with students theoretical computer science, because it contains most materials of our course in a concise way. Moreover, it presents many important advanced topics. I find the style of some proofs rather technical, but I like the topics in this book. <br />
<br />
''S. Arora and B. Barak, Computational Complexity: A Modern Approach, 2009'' <br />
<br />
The course materials can also be found in various chapters of the following massive book (700 pages). It starts at beginning bachelor level and ends at an advanced master level. It is written in a pleasant style with excellent examples.<br />
<br />
''C. Moore and S. Mertens, The nature of computation, 2011.''<br />
<br />
This book gives an introduction to important recent research directions in computational hardness. It also studies specific topics (games and planar problems) in huge detail.<br />
<br />
''E. Demaine, W Gasarch, Haijaghayi, Computational intractability: a guide to lower bounds, 2023'' [https://hardness.mit.edu/ current draft]<br />
<br />
This is an advanced textbook with background on parameterized algorithms.<br />
<br />
''M. Cygan, F. Fomin and 6 others, Parameterized algorithms, 2016''<br />
<br />
<br />
= Grading =<br />
<br />
For AMI students:<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
For PI students (the course is called "computational complexity" and takes 3 modules). There are 2 scores for this course. The first one, given in December is calculated by the above formula (but probably it does not mean anything, I will ask about it). The second score is given below, and it is the one that will be in the diploma. It includes a programming project. The assignment and grader, will be set up by the end of Februari, the deadline is the end of March. <br />
<br />
Final score = 0.3 * [score homework] + 0.3 * [score colloquium] + 0.2 * [score exam] + 0.2 * [score project] <br><br />
<br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 0.5 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. <br />
<br />
Subin Pulari: Please contact via telegram or mail [mailto:spulari@hse.ru spulari@hse.ru]</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=93948Kolmogorov complexity fall20252025-11-28T17:00:25Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| 28.11 || Time bounded complexity, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/3izk9t67v07rm4r354wnl/07sem.pdf?rlkey=rzd9slcmorkdy0n5bcd0kf3cv&st=edujzruy&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || TBA, (something related to cryptography), optional || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=93947Kolmogorov complexity fall20252025-11-28T16:57:21Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| 28.11 || Time bounded complexity, P=NP implies symmetry of time-bounded complexity. || || [https://www.dropbox.com/scl/fi/m0tzh6qocbxtr4rz3ai5x/07sem.pdf?rlkey=iy7glkqwkbciax7n5ttdfkmo0&st=qpbh8cih&dl=0 sem07] ||<br />
|- <br />
|| 05.12 || TBA, (something related to cryptography), optional || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=93933Statistical learning theory 20252025-11-27T19:49:17Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://youtube.com/live/URjcCXEMPv4 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --><br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/4k5s7yrarztkj9xhh8gh2/colloqQuest.pdf?rlkey=36tqup19jwjs89x7y5qdgvsxr&st=iww3fo1v&dl=0 Rules and questions]. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=93908Kolmogorov complexity fall20252025-11-26T13:26:41Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://drive.google.com/file/d/1ulkswiIOvB-mxok_sUwffNzvw1Vy_4tf/view?usp=drive_link 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| 28.11 || Time bounded complexity, P=NP implies symmetry of time-bounded complexity. || || ||<br />
|- <br />
|| 05.12 || TBA, (something related to cryptography), optional || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Kolmogorov_complexity_fall2025&diff=93907Kolmogorov complexity fall20252025-11-26T12:45:49Z<p>Brbauwens: </p>
<hr />
<div><br />
= Classes =<br />
<br />
Lectures + seminar: Friday 18h10 -- 21h00 in Pokrovkaya, the room is on line 20 [https://docs.google.com/spreadsheets/d/1nrzlctbhbJ6sBk0DGq_RZz-P2Tk3Ocu9mXyP6UzzgOo/edit?gid=0#gid=0 here] and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom]. The teacher is [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]. <br />
<br />
Telegram group for announcements and discussions [https://t.me/+DwgZDmhRORgxZjg0 invite link.] <br />
<br />
<br />
= Course Materials =<br />
<br />
<br />
{| class="wikitable"<br />
|-<br />
! Rec !! Summary !! Notes !! Problem list !! Solutions<br />
|-<br />
|| [https://rutube.ru/video/a7c6eee70cf19f9c3fa364e6421e3246 10.10] || Course overview, (universal) Turing machines, computable and non-computable sets and functions. [https://www.dropbox.com/scl/fi/0fxjeu8h1ekawcueto4yn/01slides.pdf?rlkey=ww3ec2ngtoysexwoc6wahcfg4&st=7t55t5bi&dl=0 slides.pdf] || [https://www.dropbox.com/scl/fi/n1vhfvtfagv2ji6x30exn/00notes.pdf?rlkey=pvd79gi91567mut35n87j1vfo&st=ypsaw36s&dl=0 ch00] [https://www.dropbox.com/scl/fi/cm2xfnegccktu3s7h00wy/01notes.pdf?rlkey=87ctjsio681ee821tzlomufqp&st=jvdlwdnn&dl=0 ch01]|| [https://www.dropbox.com/scl/fi/7qwg0cdxm41vp9awrr6iw/01sem.pdf?rlkey=ix23k6otcnc0ym2byrc4rgs1g&st=im58g130&dl=0 sem01] || [https://www.dropbox.com/scl/fi/ptqijcgwf70yfca3kcwxs/01sol.pdf?rlkey=7cchyt7p75bqylz75mj3ehg9n&st=ursmcybk&dl=0 sol01]<br />
|-<br />
|| [https://youtube.com/live/gxtqX9ZhJfQ 17.10] || Plain Kolmogorov complexity, simple properties, upper bounds, symmetry of information (see also [https://arxiv.org/pdf/1504.04955 notes]) || [https://www.dropbox.com/scl/fi/4ujoo1ppjxyib8ehz66il/04slides.pdf?rlkey=gz6npiztww9fcui2pgktxp6pz&st=09bw895o&dl=0 05sl] || [https://www.dropbox.com/scl/fi/4it73pfmwjy1t7m2mqhkw/02sem.pdf?rlkey=qr5mt2mmqjot19vg8pmycs1ek&st=avpkd9qv&dl=0 sem02] upd 03.11<br />
|| [https://www.dropbox.com/scl/fi/o5fff9ikngnayzhefpxbz/02sol.pdf?rlkey=softuys65jaxhq70l77we9508&st=2s8v90ba&dl=0 sol02]<br />
|-<br />
|| <!-- [https://www.youtube.com/watch?v=p6vKnnPfVOM 25.10]--> [https://rutube.ru/video/865c221a0b1d7431680b0236b6116a82/ 25.10] || Optimality of Solomonoff induction. See Vitanyi chapter 5.1--5.2 for background. [https://www.dropbox.com/scl/fi/rgdv6lhsjyauupmi319pw/02slides.pdf?rlkey=86etkg0bnbxx7debnvo5d8i2q&st=etj740r0&dl=0 slides.pdf]|| [https://www.dropbox.com/scl/fi/hhkk87q66ebx54vtswkge/02notes.pdf?rlkey=argdpb3l8ky75lj1g7n94175e&st=rqfszhit&dl=0 ch02] [https://www.dropbox.com/scl/fi/wjihhy6iuoxjuh9gje6b7/03notes.pdf?rlkey=03s29gprfjej7qwrvju8icaus&st=hlhvwri5&dl=0 ch03]|| [https://www.dropbox.com/scl/fi/x3chzhcrl3qzakuhyrik6/03sem.pdf?rlkey=mbcbobwn73fsj0a1e3c424x4a&st=qd9siz5a&dl=0 sem03] || [https://www.dropbox.com/scl/fi/maro4fk0h5fr2uvncf58g/03sol.pdf?rlkey=nevucf12rz3ywvwwp3f9sly3s&st=i8bcrhbd&dl=0 sol03]<br />
<!-- |-<br />
|| 24.10 || Minimizing loss according to unknown computable measure (non-responsive systems). Merhav-Feder and Helinger bounds for Bayesian mixtures. || [https://www.dropbox.com/scl/fi/isxpy6t5v9remzwlvt5e2/04notes.pdf?rlkey=gxhs69b00tcye4y8mmv4x5qyx&st=xalcy9a4&dl=0 ch04] || see notes ch04 || --> <br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=sharing 07.11] || Minimizing loss in responsive systems. Self-optimizing strategies in sets of environments. If there exists a self-optimizing strategy, then a Bayesian mixture over all environments in a countable set is self optimizing. A variant with algorithmic probability. || [https://www.dropbox.com/scl/fi/i97a1auk5x10aqomx6its/05notes.pdf?rlkey=kcmufbn1w70m48n4lvbp5yp54&st=rswmru4w&dl=0 ch05] || [https://www.dropbox.com/scl/fi/7ovfpljxe5p59vfhthzrk/04sem.pdf?rlkey=18urcicegj875oiu7x21sj2gi&st=p8nergg7&dl=0 sem04]<br />
|- <br />
|| [https://drive.google.com/file/d/11rPr-3Z7BkEUZ9bGelnHX19QXC_SyW54/view?usp=drive_link 14.11] || Prefix Kolmogorov complexity 2 definitions. Precise symmetry of information (proof next time). || [https://www.dropbox.com/scl/fi/zu84z1smw9m0rdns54ohr/05slides.pdf?rlkey=fakwr6sbl3fadwbydsecsz4mb&st=nyuu4wmz&dl=0 06sl] || [https://www.dropbox.com/scl/fi/1a05vzsn449cnvycqxvmr/05sem.pdf?rlkey=re2kkl2cagxz9tqtos05liipv&st=gzuzg3sq&dl=0 sem05] || <br />
|-<br />
|| [https://youtube.com/live/Uh-hXNLegRw 21.11] || Algorithmic statistics [https://arxiv.org/pdf/1607.08077 overview paper]. Seminar: incompressibility method || [https://www.dropbox.com/scl/fi/07vwgskncnceo5gnl3g1m/07slides.pdf?rlkey=7yuluul6anwd7vpm72tagpqek&st=nscvj3g3&dl=0 07sl] || [https://www.dropbox.com/scl/fi/zmb0hajmclre451fvvsnj/06sem.pdf?rlkey=becx5d4auwhgpjfaewatgmkej&st=4t3e0xzt&dl=0 sem06] || <br />
<!--|-<br />
|| [https://youtube.com/live/o8GS-4fCACk 28.11] || Algorithmic statistics, (notes later) || || [https://www.dropbox.com/scl/fi/gap1istjpb9xe80nfu5mb/08sem.pdf?rlkey=blefd1fodq4x0zhommsnp0h6i&st=rzaqhybe&dl=0 sem08] || --><br />
|-<br />
|| 28.11 || Time bounded complexity, P=NP implies symmetry of time-bounded complexity. || || ||<br />
|- <br />
|| 05.12 || TBA, (something related to cryptography), optional || || ||<br />
|- <br />
|| 12.12 || Colloquium. [https://www.dropbox.com/scl/fi/9pc3gsvprmwzuynxauis0/col.pdf?rlkey=rbyhl3ydyhfqjxrpmb1yyy5dg&st=vo7x6nwr&dl=0 Previous year's questions and rules]. || || ||<br />
|}<br />
<br />
<br />
<br />
<br />
= Homeworks =<br />
<br />
Deadlines: every 2 weeks, before the lecture at 18h00. Submit in pdf or fotos of handwritten to brbauwens@gmail.com with the subject line starting with KOLM-HW. Link with results will be [https://docs.google.com/spreadsheets/d/1_vO5Q0wGkM0kU1OHNiP8lz24M4i-toLC2AmQg7tuRVE/edit?usp=sharing here]. <br />
<br />
Tasks are in the problem lists from the seminar. Deadlines: problem lists 1 and 2: at the start of 3rd lecture, lists 3 and 4 at the start of the 5th lecture, etc.<br />
<br />
<br />
= References = <br />
<br />
See notes00.pdf above.<br />
<br />
= Grading =<br />
<br />
Final score = 0.35 * [score homework] + 0.35 * [score colloquium] + 0.3 * [score exam] <br><br />
<br />
Some homework assignments contain extra problems. Each solution of an extra problem will give 1 extra points on the final exam (which is graded out of 10). There will be around 10 extra problems. Rounding is applied only when the final score is transferred to the official grade. Arithmetic rounding is used. Autogrades. If only 6/10 for the exam is needed to get a final score of 10/10, then this will be given automatically. <br />
<br />
<br />
= Colloquium and exam =<br />
<br />
Colloquium: in the middel of December, a list with about 10 questions will be provided. <br />
<br />
Exam: Problems similar to the homework. You may use main references, lecture notes, and handwritten notes. <br />
<br />
<br />
= Office hours =<br />
<br />
Bruno Bauwens: Tuesday 15h -- 21h. Friday 15h -- 18h. Contact in telegram for other moments</div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=93899Statistical learning theory 20252025-11-25T21:33:08Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 hours late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16]<br />
|| See next<br />
|| <br />
|-<br />
| [https://youtube.com/live/URjcCXEMPv4 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --><br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/e2692ns95pg0kj0m4e0wo/colloqQuest.pdf?rlkey=peey4u0dxz0vohv39a3oc67ft&st=c87t9kqu&dl=0 Rules and questions] previous year. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=93877Statistical learning theory 20252025-11-25T14:06:10Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://rutube.ru/video/private/0880b08b276d73b298680531eb2f50ec/?p=Q0Yuxh016gjGtW8887Dnpw 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/ipsngdfvo4bvhofxh4377/16book_lossLandscapeNeuralNet.pdf?rlkey=3018bx9wczc4rpu7xq0wxdc2q&st=64mz3r2p&dl=0 ch16]<br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --><br />
|-<br />
| [https://youtube.com/live/URjcCXEMPv4 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| <br />
||<br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/e2692ns95pg0kj0m4e0wo/colloqQuest.pdf?rlkey=peey4u0dxz0vohv39a3oc67ft&st=c87t9kqu&dl=0 Rules and questions] previous year. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwenshttps://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2025&diff=93876Statistical learning theory 20252025-11-25T14:03:32Z<p>Brbauwens: </p>
<hr />
<div>== General Information ==<br />
<br />
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see [https://docs.google.com/spreadsheets/d/1EAbqb8wf48evEi5Bf2M0xmZXQSrpd_FJbsnevXDQxaQ/edit?gid=614347250#gid=614347250 here] for the room a few hours before the lecture, and in [https://us02web.zoom.us/j/82300259484?pwd=NWxXekxBeE5yMm9UTmwvLzNNNGlnUT09 zoom] by [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens]<br />
<br />
Seminars: on Tuesdays 14h40 -- 16h00 online in [https://us06web.zoom.us/j/85239566702?pwd=y4uhpPrdjSVKOS2LkDIcKCzBXtCbFb.1 Zoom] by [https://www.hse.ru/org/persons/225553845/ Nikita Lukianenko].<br />
<br />
Please join the [https://t.me/+JX7BTfez1sYyODFk telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2024/25 last year].<br />
<br />
<br />
== Homeworks ==<br />
<br />
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)<br />
<br />
Before 3rd lecture, submit homework from problem lists 1 and 2. <br />
Before 5th lecture, from lists 3 and 4. Etc.<br />
<br />
Submit homeworks in [https://classroom.google.com/c/ODE1MTM3NzkzNjMz?cjc=rfmqiyik google class]. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results [https://docs.google.com/spreadsheets/d/1ef5FanXBoxh7xO0rpvU8xJfxBtmfmXev1onvyvN27UQ/edit?usp=sharing are here].<br />
<br />
Late policy: 1 homework can be submitted at most 24 late without explanations.<br />
<br />
== Course materials ==<br />
<br />
{| class="wikitable"<br />
|-<br />
! Video !! Summary !! Slides !! Lecture notes !! Problem list !! Solutions<br />
|-<br />
| <br />
|| ''Part 1. Online learning'' <br />
|-<br />
| [https://rutube.ru/video/private/73eda00904d975a1860151ad0203b529/?p=TSJicLPrAw5Tz1Q67j9jKw 16 Sep]<br />
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. <br />
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01]<br />
|| [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] <br />
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01]<br />
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01]<br />
|-<br />
| [https://rutube.ru/video/8d4e5fd67a791b8f0b46603c2dd4cffe 23 Sep]<br />
|| The standard optimal algorithm. The perceptron algorithm. <br />
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] <br />
|| [https://www.dropbox.com/scl/fi/9016w6j87oclagapah8dt/02book_sequentialOptimalAlgorithm.pdf?rlkey=r729ir0a47ncqip8rooq9txxo&st=zx2tu8gp&dl=0 ch02] [https://www.dropbox.com/scl/fi/iwclbc321iv4k9fmljwpb/03book_perceptron.pdf?rlkey=9v27bt1b9qc2q382l6lwyrkic&st=ni0n8482&dl=0 ch03]<br />
|| [https://www.dropbox.com/scl/fi/ytjjgu9fcjcqm1a0h015q/02sem.pdf?rlkey=oj132041fc6g3i5tbjezlj7fv&st=q3lr807c&dl=0 prob02]<br />
|| [https://www.dropbox.com/scl/fi/0a00da9ls8bsb2zuhgor1/02sol.pdf?rlkey=x9wkfvqfu9j6x6nnf9im24w0x&st=xg427cef&dl=0 sol02]<br />
|-<br />
| [https://rutube.ru/video/private/1182e518bc57807ba63fcb1e4d34a2df/?p=7Oox56EgTf8lCAUwLPir2A 30 Sep]<br />
|| Prediction with expert advice. Recap probability theory (seminar). <br />
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03]<br />
|| [https://www.dropbox.com/scl/fi/7pn3dyf2890p9zuyxleyl/04book_predictionWithExperts.pdf?rlkey=0capmeeu6pwp9wz2mhi0t5h58&st=f4c4n9wo&dl=0 ch04] [https://www.dropbox.com/scl/fi/cx7hsxzwg2f8ep4qcuefc/05book_introProbability.pdf?rlkey=rfq0y9cgzqvl1dlxkccc3qebv&dl=0 ch05]<br />
|| [https://www.dropbox.com/scl/fi/vgqy4yp5dl6ip6ydunm69/03sem.pdf?rlkey=cgmdzvg4dn2eesspy0196l2v5&st=n6864cld&dl=0 prob03] Upd 7 Oct<br />
|| [https://www.dropbox.com/scl/fi/rwq6u32ld68bcdz3mk4cl/03sol.pdf?rlkey=2b8q1vih0byz6ipu1s2tbfzet&st=8ea131vr&dl=0 sol03]<br />
|-<br />
| <br />
|| ''Part 2. Distribution independent risk bounds'' <br />
|-<br />
| [https://rutube.ru/video/c1efb4a0af21e2309d9353e5cc5616fa/ 07 Oct]<br />
|| Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. <br />
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]<br />
|| [https://www.dropbox.com/s/nh4puyv7nst4ems/06book_sampleComplexity.pdf?dl=0 ch06] <br />
|| [https://www.dropbox.com/scl/fi/3g1r1gqfsilr2xuf0s5wc/04sem.pdf?rlkey=7rtmzxsynqf4340duzsqof2k0&st=uod3vu0z&dl=0 prob04] <br />
|| [https://www.dropbox.com/scl/fi/kbjujf1wazsirzg9cno6p/04sol.pdf?rlkey=lcbilxedmhi1toghgenaiy2nu&st=c29aqpjw&dl=0 sol04]<br />
|- <br />
| [https://www.youtube.com/watch?v=8J5B9CCy-ws 14 Oct]<br />
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions <br />
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]<br />
|| [https://www.dropbox.com/s/eurz2vkvt1wa5zm/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/scl/fi/50oxlmjkx59hjrq82yqvx/08book_VCdimension.pdf?rlkey=5dtlcis378kqu24ttko6s7zpf&dl=0 ch08]<br />
|| [https://www.dropbox.com/scl/fi/jhcildh8546dr0u9kkz5n/05sem.pdf?rlkey=l8wz1fyl2svmcu2w8tbd0rzjh&st=5lipe8jy&dl=0 prob05]<br />
|| [https://www.dropbox.com/scl/fi/70z0umh656y5kt45vsbt9/05sol.pdf?rlkey=6jpu3usc6uri7duj78l9wr8wq&st=w8gjtn1i&dl=0 sol05]<br />
|-<br />
| [https://rutube.ru/video/e50aeb873359ba63e4df48d52fa8cb67 21 Oct]<br />
|| Risk decomposition and the fundamental theorem of statistical learning theory (previous [https://www.youtube.com/watch?v=zHau8Br_UFQ recording] covers more)<br />
|| [https://www.dropbox.com/s/0p8r5wgjy1hlku2/06slides.pdf?dl=0 sl06]<br />
|| [https://www.dropbox.com/scl/fi/th4r5t2gm29en4hejareq/09book_riskBounds.pdf?rlkey=4ox3f26kygxorxft8jlijuf0f&st=fg0fdyx2&dl=0 ch09]<br />
|| [https://www.dropbox.com/scl/fi/orntdt6b6u8y4b408dkfz/06sem.pdf?rlkey=3j6sajyqdvtfph49ao73mj6ly&st=i5j8itjr&dl=0 prob06]<br />
|| [https://www.dropbox.com/scl/fi/53b0wro00u20tkpqb6206/06sol.pdf?rlkey=u1hpkkrrz96ol7i49ifdrhpud&st=bbs9e6yr&dl=0 sol06]<br />
|-<br />
| [https://rutube.ru/video/fb0bb984a0461b9e5b9ea2e6973f8252/ 23 Oct]<br />
|| Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. <br />
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]<br />
|| [https://www.dropbox.com/scl/fi/ohtmf1fwsu9c6vkrj6e5a/10book_measureConcentration.pdf?rlkey=dqsgskp8slui6xoq9c7tx680b&dl=0 ch10] [https://www.dropbox.com/s/hfrvhebbsskbk6g/11book_RademacherComplexity.pdf?dl=0 ch11]<br />
|| [https://www.dropbox.com/scl/fi/spcnkcm5opa432mqeh95s/07sem.pdf?rlkey=roht5zgpdaatx4h9dy24lryoi&st=wu6ubcdi&dl=0 prob07]<br />
|| [https://www.dropbox.com/scl/fi/icn16qpdb5bfyb46nn72x/07sol.pdf?rlkey=smcllfyfub5jsk530i2igdg3w&st=efos9f1f&dl=0 sol07]<br />
|-<br />
| <br />
|| ''Part 3. Margin risk bounds with applications'' <br />
|-<br />
| [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ 06 Nov]<br />
|| Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to [https://rutube.ru/video/43cde71591a0ee78e27d4818daddc313/ old recording] for SVM stuff).<br />
|| [https://www.dropbox.com/s/oo1qny9busp3axn/08slides.pdf?dl=0 sl08]<br />
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13]<br />
|| [https://www.dropbox.com/scl/fi/kb8njw77xqohcul0meeik/08sem.pdf?rlkey=c0cewf8l34skxcanhtbc29wrk&st=kd7d535j&dl=0 prob08]<br />
|| [https://www.dropbox.com/scl/fi/17idg180n115gu2cd5qqu/08sol.pdf?rlkey=mo07u28tszof7l3fzxahj8cm1&st=oy758osr&dl=0 sol08]<br />
|-<br />
| [https://youtube.com/live/77-rZFzX2O8 11 Nov]<br />
|| Kernels: RKHS, representer theorem, risk bounds<br />
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09]<br />
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14]<br />
|| [https://www.dropbox.com/scl/fi/72ze9ki1si6v9wnaw6ioi/09sem.pdf?rlkey=5junztvzv8rxguts9lv4xyh7g&st=5xplcduo&dl=0 prob09]<br />
|| [https://www.dropbox.com/scl/fi/pod6vizxg2y6gjhmz9kip/09sol.pdf?rlkey=jvdyl6d0z4yw7stdgszfsan7z&st=8coxi1hv&dl=0 sol09]<br />
|- <br />
| [https://rutube.ru/video/private/5ffc7d4c3dbcf8f60f7f553b87fd555a/?p=eYr_yyIUJ2yHdB7CFygpbg 18 Nov]<br />
|| AdaBoost and the margin hypothesis<br />
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10]<br />
|| [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15]<br />
|| [https://www.dropbox.com/scl/fi/hhsmkjr0g5fwk56wicdok/10sem.pdf?rlkey=c6rntxw1zhs2fe9tajmdhvlsa&st=b1wi8bwd&dl=0 prob10]<br />
|| [https://www.dropbox.com/scl/fi/petas7wseh2p1igpjjonk/10sol.pdf?rlkey=zubelzltmtgyvxwpfssm4vdvi&st=geg7yxi1&dl=0 sol10]<br />
|- <br />
|<br />
|| ''Part 4. Neural nets'' <br />
|- <br />
| [https://youtube.com/live/DUgksR6gOQ8 25 Nov]<br />
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/ipsngdfvo4bvhofxh4377/16book_lossLandscapeNeuralNet.pdf?rlkey=3018bx9wczc4rpu7xq0wxdc2q&st=64mz3r2p&dl=0 ch16]<br />
|| [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11]<br />
|| <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --><br />
|-<br />
| [https://youtube.com/live/URjcCXEMPv4 02 Dec]<br />
|| Lazy training and the neural tangent kernel. <br />
|| <br />
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] <br />
|| <br />
||<br />
|-<br />
| 09 Dec<br />
|| TBA<br />
|| <br />
|| <br />
|| <br />
||<br />
|-<br />
| 16 Dec<br />
|| Colloquium [https://www.dropbox.com/scl/fi/e2692ns95pg0kj0m4e0wo/colloqQuest.pdf?rlkey=peey4u0dxz0vohv39a3oc67ft&st=c87t9kqu&dl=0 Rules and questions] previous year. Select a [https://docs.google.com/spreadsheets/d/1aaetd-Mh9Y_OaJXYprfsrWz7wkt5fvg12VJRpqkqMTc/edit?usp=sharing timeslot].<br />
|}<br />
<br />
<!-- Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.--><br />
<br />
The lectures in October and November are based on the book:<br />
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. <br />
<br />
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:<br />
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.<br />
<br />
== Grading formula ==<br />
<br />
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.<br />
<br />
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade. <br />
<br />
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used. <br />
<br />
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes. <br />
<br />
== Colloquium ==<br />
<br />
[https://www.dropbox.com/scl/fi/80u1zfr34nt1il8q0avxs/colloqQuest.pdf?rlkey=n8y51ykull9urd0cryv8435nr&dl=0 Rules and questions from last year.] <br />
<br />
Date: TBA<br />
<br />
== Problems exam ==<br />
<br />
Date: TBA<br><br />
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br><br />
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) <br />
<br />
About questions<br><br />
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br><br />
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. <br />
<br />
<br />
== Office hours ==<br />
<br />
Bruno Bauwens: TBA. Better send me an email in advance.<br />
<br />
Nikita Lukianenko: Write in Telegram, the time is flexible <br />
<br />
<!--<br />
== Russian texts ==<br />
<br />
The following links might help students who have trouble with English. A [http://www.machinelearning.ru/wiki/images/d/d9/Voron-2011-tnop.pdf lecture] on VC-dimensions was given by K. Vorontsov.<br />
A [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) course] on Statistical Learning Theory by Nikita Zhivotovsky is given at MIPT. Some short description about PAC learning on p136 in the [http://gen.lib.rus.ec/search.php?req=%D0%9D%D0%B0%D1%83%D0%BA%D0%B0+%D0%B8+%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE+%D0%BF%D0%BE%D1%81%D1%82%D1%80%D0%BE%D0%B5%D0%BD%D0%B8%D1%8F+%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC%D0%BE%D0%B2%2C+%D0%BA%D0%BE%D1%82%D0%BE%D1%80%D1%8B%D0%B5+%D0%B8%D0%B7%D0%B2%D0%BB%D0%B5%D0%BA%D0%B0%D1%8E%D1%82+%D0%B7%D0%BD%D0%B0%D0%BD%D0%B8%D1%8F+%D0%B8%D0%B7+%D0%B4%D0%B0%D0%BD%D0%BD%D1%8B%D1%85&lg_topic=libgen&open=0&view=simple&res=25&phrase=0&column=def book] <br />
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru] <br />
you can find brief and clear definitions.<br />
--></div>Brbauwens