强化学习的数学原理

0. Preview

• Bilibili: 强化学习的数学原理

• GitHub: MathFoundationRL

1. Basic Concepts

1.1 Markov Decision Process (MDP)

• Sets

  • State: $\mathcal{S}$

  • Action: $\mathcal{A}(s)$

  • Reward: $\mathcal{R}(s,a)$

• Probability distribution

  • State transition: $p(s'|s,a)$

  • Reward: $p(r|s,a)$

• Policy: $\pi(a|s)$

• Markov property

2. State Value and Bellman Equation

2.1 State Value

• State value: $v_\pi(s)=\mathbb{E}_\pi[G_t|S_t=s]$

  where $G_t=\sum_{k=0}^\infty \gamma^k R_{t+k+1}$

2.2 Bellman Equation

• Bellman equation: $v_\pi(s)=\sum_a \pi(a|s)\sum_{s'}[\underbrace{\sum_r p(r|s,a)r}_{immediate\ reward} + \underbrace{\gamma \sum_{s'} p(s'|s,a)v_\pi(s')}_{future\ reward}]$

  • Matrix form: $v_\pi=r_\pi + \gamma P_\pi v_\pi$