Обучение с подкреплением (ИИ24, 7 модуль): различия между версиями

Материал из Wiki - Факультет компьютерных наук
Перейти к навигации Перейти к поиску
← Предыдущая правкаСледующая правка →
HW5 + Paper Deadline
Материалы 10 + Ссылки 9 + Апдейт ссылок
Строка 29: Строка 29:
  ! Занятие !! Тема !! Дата !! Дополнительные материалы
  ! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming  || 10/01/2026 ||  
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming  || 10/01/2026 ||  
|-
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||  
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||  
|-
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]


[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]
[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]
Строка 53: Строка 53:
[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]


[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]


[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]
[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]
Строка 65: Строка 65:
[[https://github.com/tinkoff-ai/CORL CORL Library]]
[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]


[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||  
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||  
|-
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||  
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||  

Версия от 12:51, 14 марта 2026

О курсе

Занятия проводятся в Zoom по субботам с 13:00 МСК

Контакты

Чат курса в TG: link

Преподаватель: Сергей Лактионов, Вячеслав Бучков

Ассистент Контакты

Материалы курса

Ссылка на плейлист курса на YouTube: [YouTube-playlist]

Ссылка на плейлист курса в VKVideo: [VKVideo-playlist]

Ссылка на GitHub с материалами курса: [GitHub repository]

Занятие Тема Дата Дополнительные материалы
1 [YouTube] [VKVideo] [Материалы] Intro to RL, Dynamic Programming 10/01/2026
2 [YouTube] [VKVideo] [Материалы] Model-Free Tabular RL: Q-Learning, SARSA 17/01/2026
3 [YouTube] [VKVideo] [Материалы] Intro to Deep RL: DQN, RAINBOW and beyond 24/01/2026 [DQN], [DRQN], [RAINBOW], [APE-X]
4 [YouTube] [VKVideo] [Материалы] Policy-Based Methods: Policy Gradient, REINFORCE, A2C 31/01/2026 [Policy Gradient], [Actor-Critic], [REINFORCE in 2024]
5 [YouTube] [VKVideo] [Материалы] Advanced Policy-Based: TRPO, PPO and beyond 07/02/2026 [TRPO], [PPO], [TRPO vs PPO]

[37 implementation details of PPO]

[Wasserstein distance вместо KL]

[Sinkhorn distance вместо KL]

[Improvement Lower Bound в TRPO]

[TV Distance]

[Sigmoid Soft-Clipping]

[Soft-Clipping in LLM]

6 [YouTube] [VKVideo] [Материалы] Continuous Control: DDPG, SAC and beyond 14/02/2026 [DDPG], [TD3], [SAC], [TQC]

[MuJoCo]

7 [Youtube] [VKVideo] [Материалы] Offline RL 21/02/2026 [Offline RL Tutorial, [A Survey on Offline RL]

[IQL], [CQL], [ReBRAC]

[Decision Transformers], [Trajectory Transformers]

[CORL Library]

8 [Youtube] [VKVideo] [Материалы] Multi-Armed Bandits 28/02/2026 [Neural UCB] [Neural Thompson Sampling]
9 [Youtube] [VKVideo] [Материалы] Model-based RL: AlphaZero and friends 07/03/2026 [AlphaZero], [MuZero], [EfficientZero]

[World Models], [Dreamer-V1]

10 Youtube VKVideo [Материалы] RL in a context of LLMs 14/03/2026
11 Youtube VKVideo Материалы Practical RL 21/03/2026

Формула оценивания

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

  • HW - сумма баллов за (как минимум) 5 ДЗ;
  • RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
  • TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

Домашние задания

  1. HW-1 "Value- and policy-iteration algorithms" (2 балла) | Мягкий дедлайн - 23/01/26 23:59, Жёсткий - 30/01/26 23:59 | [Ноутбук]
  2. HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (3 балла) | Мягкий дедлайн - 31/01/26 23:59 04.02.2026 23:59, Жёсткий - 07/02/26 23:59 11.02.2026 23:59 | [Ноутбук]
  3. HW-3 "DQN Implementation" (6 баллов) | Мягкий дедлайн - 20/02/26 23:59, Жёсткий - 27/02/26 23:59 01.03.2026 23:59 | [Ноутбук]
  4. HW-4 "PPO Implementation" (5 баллов) | Мягкий дедлайн - 08/03/26 23:59, Жёсткий - 15/03/26 23:59 | [Ноутбук]
  5. HW-5 "SAC Implementation" (5 баллов) | Мягкий дедлайн - 15/03/26 23:59, Жёсткий - 22/03/26 23:59 | [Ноутбук]

Дедлайн по согласованию статьи - 15/03/26 23:59

Литература

  1. Reinforcement Learning: An Introduction by R.Sutton and A.Barto
  2. Practical RL course by YSDA
  3. David Silver's course
  4. Sergey Levine's course
  5. Reinforcement Learning Textbook (in Russian)
Источник — https://wiki.cs.hse.ru/index.php?title=Обучение_с_подкреплением_(ИИ24,_7_модуль)&oldid=95773