Обучение с подкреплением (ИИ24, 7 модуль): различия между версиями
Перейти к навигации
Перейти к поиску
Ссылки неделя 7 |
Ссылки на GH материалы |
||
| Строка 29: | Строка 29: | ||
! Занятие !! Тема !! Дата !! Дополнительные материалы | ! Занятие !! Тема !! Дата !! Дополнительные материалы | ||
|- | |- | ||
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || Intro to RL, Dynamic Programming || 10/01/2026 || | | style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 || | ||
|- | |- | ||
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 || | | style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 || | ||
|- | |- | ||
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]] | | style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]] | ||
|- | |- | ||
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]] | | style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]] | ||
|- | |- | ||
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]] | | style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]] | ||
[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]] | [[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]] | ||
| Строка 53: | Строка 53: | ||
[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]] | [[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]] | ||
|- | |- | ||
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]] | | style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]] | ||
[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]] | [[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]] | ||
|- | |- | ||
| style="background:#eaecf0;" | '''7''' [[ Youtube]] [[ VKVideo]] || Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]] | | style="background:#eaecf0;" | '''7''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]] | ||
[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]] | [[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]] | ||
| Строка 65: | Строка 65: | ||
[[https://github.com/tinkoff-ai/CORL CORL Library]] | [[https://github.com/tinkoff-ai/CORL CORL Library]] | ||
|- | |- | ||
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || Multi-Armed Bandits || 28/02/2026 || | | style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 || | ||
|- | |- | ||
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || Model-based RL: AlphaZero and friends || 07/03/2026 || | | style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || | ||
|- | |- | ||
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || RL in a context of LLMs || 14/03/2026 || | | style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 || | ||
|- | |- | ||
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || Practical RL || 21/03/2026 || | | style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 || | ||
|- | |- | ||
|} | |} | ||
Версия от 13:00, 21 февраля 2026
О курсе
Занятия проводятся в Zoom по субботам с 13:00 МСК
Контакты
Чат курса в TG: link
Преподаватель: Сергей Лактионов, Вячеслав Бучков
| Ассистент | Контакты |
|---|---|
Материалы курса
Ссылка на плейлист курса на YouTube: [YouTube-playlist]
Ссылка на плейлист курса в VKVideo: [VKVideo-playlist]
Ссылка на GitHub с материалами курса: [GitHub repository]
| Занятие | Тема | Дата | Дополнительные материалы |
|---|---|---|---|
| 1 [YouTube] [VKVideo] | [Материалы] Intro to RL, Dynamic Programming | 10/01/2026 | |
| 2 [YouTube] [VKVideo] | [Материалы] Model-Free Tabular RL: Q-Learning, SARSA | 17/01/2026 | |
| 3 [YouTube] [VKVideo] | [Материалы] Intro to Deep RL: DQN, RAINBOW and beyond | 24/01/2026 | [DQN], [DRQN], [RAINBOW], [APE-X] |
| 4 [YouTube] [VKVideo] | [Материалы] Policy-Based Methods: Policy Gradient, REINFORCE, A2C | 31/01/2026 | [Policy Gradient], [Actor-Critic], [REINFORCE in 2024] |
| 5 [YouTube] [VKVideo] | [Материалы] Advanced Policy-Based: TRPO, PPO and beyond | 07/02/2026 | [TRPO], [PPO], [TRPO vs PPO]
[37 implementation details of PPO] [Wasserstein distance вместо KL] |
| 6 [YouTube] [VKVideo] | [Материалы] Continuous Control: DDPG, SAC and beyond | 14/02/2026 | [DDPG], [TD3], [SAC], [TQC]
[MuJoCo] |
| 7 Youtube VKVideo | [Материалы] Offline RL | 21/02/2026 | [Offline RL Tutorial, [A Survey on Offline RL] |
| 8 Youtube VKVideo | Материалы Multi-Armed Bandits | 28/02/2026 | |
| 9 Youtube VKVideo | Материалы Model-based RL: AlphaZero and friends | 07/03/2026 | |
| 10 Youtube VKVideo | Материалы RL in a context of LLMs | 14/03/2026 | |
| 11 Youtube VKVideo | Материалы Practical RL | 21/03/2026 |
Формула оценивания
Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где
- HW - сумма баллов за (как минимум) 5 ДЗ;
- RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
- TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).
Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.
Домашние задания
- HW-1 "Value- and policy-iteration algorithms" (2 балла) | Мягкий дедлайн - 23/01/26, Жёсткий - 30/01/26 | [Ноутбук]
- HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (3 балла) | Мягкий дедлайн -
31/01/2604.02.2026 23:59, Жёсткий -07/02/2611.02.2026 23:59 | [Ноутбук] - HW-3 "DQN Implementation" (6 баллов) | Мягкий дедлайн - 20/02/26, Жёсткий - 27/02/26 | [Ноутбук]