Wiki - Факультет компьютерных наук - Вклад [ru]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-29T13:26:19Z

Ekantonistova 2: Дедлайны

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[https://www.youtube.com/watch?v=QBkqOB65WPg Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239912?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[https://www.youtube.com/watch?v=tn9C5VbqsY0 Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239939?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week11_practical_rl Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

Дедлайн по сдаче статьи - <strike>21/03/26 23:59</strike> <strike>24/03/26 23:59</strike> '''25/03/26 8:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-29T13:25:30Z

Ekantonistova 2: Ссылки Лекция 11

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[https://www.youtube.com/watch?v=QBkqOB65WPg Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239912?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[https://www.youtube.com/watch?v=tn9C5VbqsY0 Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239939?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week11_practical_rl Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

Дедлайн по сдаче статьи - <strike>21/03/26 23:59</strike> '''24/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-22T02:55:55Z

Ekantonistova 2: Видео Неделя 10 + Материалы Неделя 11

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[https://www.youtube.com/watch?v=QBkqOB65WPg Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239912?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week11_practical_rl Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

Дедлайн по сдаче статьи - <strike>21/03/26 23:59</strike> '''24/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-22T02:53:04Z

Ekantonistova 2: Опечатка

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

Дедлайн по сдаче статьи - <strike>21/03/26 23:59</strike> '''24/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-22T02:52:40Z

Ekantonistova 2: Дедлайны по статье

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''
Дедлайн по сдаче статьи - <strike>21/03/26 23:59</strike> '''24/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-14T12:51:38Z

Ekantonistova 2: Материалы 10 + Ссылки 9 + Апдейт ссылок

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0 YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[https://www.youtube.com/watch?v=bE_rLccqGXI Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239891?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week10_rl_for_llm Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-09T13:57:24Z

Ekantonistova 2: HW5 + Paper Deadline

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]
# HW-5 "SAC Implementation" (''5 баллов'') | '''Мягкий дедлайн - 15/03/26 23:59''', Жёсткий - 22/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/2026/hw5/sac.ipynb Ноутбук]]

Дедлайн по согласованию статьи - '''15/03/26 23:59'''

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-07T12:42:25Z

Ekantonistova 2: Материалы 8 + Записи 8 + Материалы 9

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[https://www.youtube.com/watch?v=roVyJyAGCtM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239873?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week8_bandits Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week9_model_based_rl Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 || [[https://arxiv.org/abs/1712.01815 AlphaZero]], [[https://deepmind.google/discover/blog/muzero-mastering-go-chess-shogi-and-atari-without-rules/ MuZero]], [[https://arxiv.org/abs/2111.00210 EfficientZero]]

[[https://worldmodels.github.io/ World Models]], [[https://blog.research.google/2020/03/introducing-dreamer-scalable.html Dreamer-V1]]
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-01T11:07:02Z

Ekantonistova 2: Неправильный дедлайн HW4

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 15/03/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-01T11:06:10Z

Ekantonistova 2: Опечатка

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', Жёсткий - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-01T11:05:32Z

Ekantonistova 2: HW3 Deadline Extension

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', '''Жёсткий''' - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - <strike>27/02/26 23:59</strike> '''01.03.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-03-01T11:04:28Z

Ekantonistova 2: Видео неделя 7 + Материалы неделя 8

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[https://www.youtube.com/watch?v=z705MwrjrEU&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB Youtube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239857?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 || [[https://arxiv.org/abs/1911.04462 Neural UCB]] [[https://arxiv.org/abs/2010.00827 Neural Thompson Sampling]]
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', '''Жёсткий''' - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-22T14:53:03Z

Ekantonistova 2: HW4

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 ||
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26 23:59''', Жёсткий - 30/01/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26 23:59</strike> '''04.02.2026 23:59''', '''Жёсткий''' - <strike>07/02/26 23:59</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]
# HW-4 "PPO Implementation" (''5 баллов'') | '''Мягкий дедлайн - 08/03/26 23:59''', Жёсткий - 27/02/26 23:59 | [[https://github.com/laktionov/RL-course/blob/main/hw4/ppo.ipynb Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-21T13:00:17Z

Ekantonistova 2: Ссылки на GH материалы

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week1_intro_dynamic_programming Материалы]] Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week2_model_free_rl Материалы]] Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week3_dqn Материалы]] Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week4_policy_based Материалы]] Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week5_advanced_policy_based Материалы]] Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week6_continuous_control Материалы]] Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[ Youtube]] [[ VKVideo]] || [[https://github.com/laktionov/RL-course/tree/2026/week7_offline_rl Материалы]] Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Multi-Armed Bandits || 28/02/2026 ||
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || [[Материалы]] Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26''', Жёсткий - 30/01/26 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26</strike> '''04.02.2026 23:59''', '''Жёсткий''' - <strike>07/02/26</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26''', Жёсткий - 27/02/26 | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-21T12:27:52Z

Ekantonistova 2: Ссылки неделя 7

==О курсе==

Занятия проводятся в [https://us06web.zoom.us/j/82360571226?pwd=QcdTZQvEba8tBWx2FrfK7v4P4k2Jra.1 Zoom] '''по субботам с 13:00 МСК'''

==Контакты==

Чат курса в TG: [https://t.me/+cl-7RP37Ulw4YTE6 link]

Преподаватель: Сергей Лактионов, Вячеслав Бучков

{| class="wikitable"
|-
! Ассистент !! Контакты
|-
| style="background:#eaecf0;" | ||
|-
| style="background:#eaecf0;" | ||
|}

==Материалы курса==
Ссылка на плейлист курса на YouTube: [[https://www.youtube.com/playlist?list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg YouTube-playlist]]

Ссылка на плейлист курса в VKVideo: [[https://vkvideo.ru/playlist/-227011779_68 VKVideo-playlist]]

Ссылка на GitHub с материалами курса: [[https://github.com/laktionov/RL-course/tree/2026 GitHub repository]]

{| class="wikitable"
|-
! Занятие !! Тема !! Дата !! Дополнительные материалы
|-
| style="background:#eaecf0;" | '''1''' [[https://www.youtube.com/watch?v=KLEcPmdR87U&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=5&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239732?linked=1 VKVideo]] || Intro to RL, Dynamic Programming || 10/01/2026 ||
|-
| style="background:#eaecf0;" | '''2''' [[https://www.youtube.com/watch?v=Uf-KHdRh3zs&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=4&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239762?linked=1 VKVideo]] || Model-Free Tabular RL: Q-Learning, SARSA || 17/01/2026 ||
|-
| style="background:#eaecf0;" | '''3''' [[https://www.youtube.com/watch?v=1wqbiJEB5ok&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=3&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239789?linked=1 VKVideo]] || Intro to Deep RL: DQN, RAINBOW and beyond || 24/01/2026 ||[[https://arxiv.org/pdf/1507.06527.pdf#:~:text=The%20resulting%20Deep%20Recurrent%20Q,equivalents%20featuring%20flickering%20game%20screens. DQN]], [[https://arxiv.org/abs/1507.06527 DRQN]], [[https://arxiv.org/abs/1710.02298 RAINBOW]], [[https://arxiv.org/abs/1803.00933 APE-X]]
|-
| style="background:#eaecf0;" | '''4''' [[https://www.youtube.com/watch?v=Je_20lKuBSM&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=2&pp=iAQB0gcJCYcKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239808?linked=1 VKVideo]] || Policy-Based Methods: Policy Gradient, REINFORCE, A2C || 31/01/2026 || [[https://papers.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Policy Gradient]], [[https://arxiv.org/abs/1602.01783 Actor-Critic]], [[https://arxiv.org/abs/2402.14740 REINFORCE in 2024]]
|-
| style="background:#eaecf0;" | '''5''' [[https://www.youtube.com/watch?v=8KU9nnp1PMo&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239827?linked=1 VKVideo]] || Advanced Policy-Based: TRPO, PPO and beyond || 07/02/2026 || [[https://arxiv.org/pdf/1502.05477.pdf TRPO]], [[https://arxiv.org/pdf/1707.06347.pdf PPO]], [[https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html TRPO vs PPO]]

[[https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ 37 implementation details of PPO]]

[[https://arxiv.org/abs/2010.05380 Wasserstein distance вместо KL]]

[[https://openreview.net/forum?id=Mlwe37htstv Sinkhorn distance вместо KL]]

[[https://arxiv.org/pdf/1705.10528 Improvement Lower Bound в TRPO]]

[[https://arxiv.org/abs/2401.16025 TV Distance]]

[[https://arxiv.org/abs/2205.10047 Sigmoid Soft-Clipping]]

[[https://arxiv.org/abs/2511.20347 Soft-Clipping in LLM]]
|-
| style="background:#eaecf0;" | '''6''' [[https://www.youtube.com/watch?v=ujESxbK1uI0&list=PLmA-1xX7IuzDe8CEWijYwsgmdHXyaEQsg&index=1&pp=iAQB0gcJCaIKAYcqIYzv YouTube]] [[https://vkvideo.ru/playlist/-227011779_68/video-227011779_456239845?linked=1 VKVideo]] || Continuous Control: DDPG, SAC and beyond || 14/02/2026 || [[https://arxiv.org/abs/1509.02971 DDPG]], [[https://arxiv.org/abs/1802.09477 TD3]], [[https://arxiv.org/abs/1801.01290 SAC]], [[https://arxiv.org/abs/2005.04269 TQC]]

[[https://gymnasium.farama.org/environments/mujoco/ MuJoCo]]
|-
| style="background:#eaecf0;" | '''7''' [[ Youtube]] [[ VKVideo]] || Offline RL || 21/02/2026 || [[https://arxiv.org/abs/2005.01643 Offline RL Tutorial], [[https://arxiv.org/abs/2203.01387 A Survey on Offline RL]]

[[https://arxiv.org/abs/2110.06169 IQL]], [[https://arxiv.org/abs/2006.04779 CQL]], [[https://arxiv.org/abs/2305.09836 ReBRAC]]

[[https://arxiv.org/pdf/2106.01345.pdf Decision Transformers]], [[https://arxiv.org/abs/2106.02039 Trajectory Transformers]]

[[https://github.com/tinkoff-ai/CORL CORL Library]]
|-
| style="background:#eaecf0;" | '''8''' [[ Youtube]] [[ VKVideo]] || Multi-Armed Bandits || 28/02/2026 ||
|-
| style="background:#eaecf0;" | '''9''' [[ Youtube]] [[ VKVideo]] || Model-based RL: AlphaZero and friends || 07/03/2026 ||
|-
| style="background:#eaecf0;" | '''10''' [[ Youtube]] [[ VKVideo]] || RL in a context of LLMs || 14/03/2026 ||
|-
| style="background:#eaecf0;" | '''11''' [[ Youtube]] [[ VKVideo]] || Practical RL || 21/03/2026 ||
|-
|}

==Формула оценивания==

Оценка = МИН(10, 10*(0.65*HW + 0.10*TA + 0.25*RC)), где

* HW - сумма баллов за (как минимум) 5 ДЗ;
* RC - оценка за презентацию статьи, посвященной новым алгоритмам или неожиданным применениям RL-парадигмы в индустрии;
* TA - сумма баллов за еженедельные квизы (суммарно 10 квизов).

Для каждого домашнего задания есть мягкий дедлайн, сдача после которого в течение недели до жёсткого дедлайна оценивается со штрафом 5% от оценки за ДЗ за каждый день просрочки.

== Домашние задания ==
# HW-1 "Value- and policy-iteration algorithms" (''2 балла'') | '''Мягкий дедлайн - 23/01/26''', Жёсткий - 30/01/26 | [[https://github.com/laktionov/RL-course/tree/2026/hw1 Ноутбук]]
# HW-2 "SARSA(\lambda) and EV-SARSA(\lambda)" (''3 балла'') | '''Мягкий дедлайн''' - <strike>31/01/26</strike> '''04.02.2026 23:59''', '''Жёсткий''' - <strike>07/02/26</strike> '''11.02.2026 23:59''' | [[https://github.com/laktionov/RL-course/tree/2026/hw2 Ноутбук]]
# HW-3 "DQN Implementation" (''6 баллов'') | '''Мягкий дедлайн - 20/02/26''', Жёсткий - 27/02/26 | [[https://github.com/laktionov/RL-course/tree/2026/hw3 Ноутбук]]

== Литература ==
# [http://incompleteideas.net/book/the-book-2nd.html Reinforcement Learning: An Introduction by R.Sutton and A.Barto]
# [https://github.com/yandexdataschool/Practical_RL Practical RL course by YSDA]
# [https://www.davidsilver.uk/teaching/ David Silver's course]
# [https://rail.eecs.berkeley.edu/deeprlcourse/ Sergey Levine's course]
# [https://arxiv.org/abs/2201.09746 Reinforcement Learning Textbook (in Russian)]

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-14T13:42:17Z

Ekantonistova 2: Опечатка TG

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-14T13:42:01Z

Ekantonistova 2: Опечатка TG

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-14T13:40:25Z

Ekantonistova 2: Дедлайны ДЗ2

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-14T13:15:36Z

Ekantonistova 2: Лекция 6

Обучение с подкреплением (ИИ24, 7 модуль)

2026-02-14T13:12:48Z