177 lines
4.9 KiB
Markdown
177 lines
4.9 KiB
Markdown
# CSE510 Deep Reinforcement Learning (Lecture 16)
|
|
|
|
## Deterministic Policy Gradient (DPG)
|
|
|
|
### Learning Deterministic Policies
|
|
|
|
- Deterministic policy gradients [Silver et al., ICML 2014]
|
|
- Explicitly learn a deterministic policy.
|
|
- $a = \mu_\theta(s)$
|
|
- Advantages
|
|
- Existing optimal deterministic policy for MDPs
|
|
- Naturally dealing with a continuous action space
|
|
- Expected to be more efficient than learning stochastic policies
|
|
- Computing stochastic gradient requires more samples, as it integrates over both state and action space.
|
|
- Deterministic gradient is preferable as it integrates over state space only.
|
|
|
|
### Deterministic Policy Gradient
|
|
|
|
The objective function is:
|
|
|
|
$$
|
|
J(\theta)=\int_{s\in S} \rho^{\mu}(s) r(s,\mu_\theta(s)) ds
|
|
$$
|
|
|
|
where $\rho^{\mu}(s)$ is the stationary distribution under the behavior policy $\mu_\theta(s)$.
|
|
|
|
The policy gradient from the standard policy gradient theorem is:
|
|
|
|
$$
|
|
\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta Q^{\mu_\theta}(s,a)]=\mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
|
|
$$
|
|
|
|
#### Issues for DPG
|
|
|
|
The formulations up to now can only use on-policy data.
|
|
|
|
Deterministic policy can hardly guarantee sufficient
|
|
exploration.
|
|
|
|
- Solution: Off-policy training using a stochastic behavior policy.
|
|
|
|
#### Off-Policy Deterministic Policy Gradient (Off-DPG)
|
|
|
|
Use a stochastic behavior policy $\beta(a|s)$. The modified objective function is:
|
|
|
|
$$
|
|
J(\mu_\theta)=\int_{s\in S} \rho^{\beta}(s) Q^{\mu_\theta}(s,\beta(s)) ds
|
|
$$
|
|
|
|
The gradients are:
|
|
|
|
$$
|
|
\begin{aligned}
|
|
\nabla_\theta J(\mu_\theta) &\approx \int_{s\in S} \rho^{\beta}(s) \nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)} ds\\
|
|
&= \mathbb{E}_{s\sim \rho^{\beta}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
|
|
\end{aligned}
|
|
$$
|
|
|
|
Importance sampling is avoided in the actor due to the absence of integral over actions.
|
|
|
|
#### Policy Evaluation in DPG
|
|
|
|
Importance sampling can also be avoided in the critic.
|
|
|
|
Gradient TD-like algorithm can be directly applied to the critic.
|
|
|
|
$$
|
|
\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2
|
|
$$
|
|
|
|
#### Off-Policy Deterministic Actor-Critic
|
|
|
|
$$
|
|
\delta_t=r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)
|
|
$$
|
|
|
|
$$
|
|
w_{t+1} = w_t + \alpha_w \delta_t \nabla_w Q^w(s_t,a_t)
|
|
$$
|
|
|
|
$$
|
|
\theta_{t+1} = \theta_t + \alpha_\theta \nabla_\theta \mu_\theta(s_t) \nabla_a Q^{\mu_\theta}(s_t,a_t)\vert_{a=\mu_\theta(s_t)}
|
|
$$
|
|
|
|
### Deep Deterministic Policy Gradient (DDPG)
|
|
|
|
Insights from DQN + Deterministic Policy Gradients
|
|
|
|
- Use a replay buffer
|
|
- Critic is updated every timestep (Sample from buffer, minibatch):
|
|
|
|
$$
|
|
\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2
|
|
$$
|
|
|
|
Actor is updated every timestep:
|
|
|
|
$$
|
|
\nabla_a Q(s_t,a;w)|_{a=\mu_\theta(s_t)} \nabla_\theta \mu_\theta(s_t)
|
|
$$
|
|
|
|
Smoothing target updated at every timestep:
|
|
|
|
$$
|
|
w_{t+1} = \tau w_t + (1-\tau) w_{t+1}
|
|
$$
|
|
|
|
$$
|
|
\theta_{t+1} = \tau \theta_t + (1-\tau) \theta_{t+1}
|
|
$$
|
|
|
|
Exploration: add noise to the action selection: $a_t = \mu_\theta(s_t) + \mathcal{N}_t$
|
|
|
|
Batch normalization used for training networks
|
|
|
|
### Extension of DDPG
|
|
|
|
Overestimation bias is an issue of Q-learning in which the maximization of a noisy value estimate
|
|
|
|
$$
|
|
DDPG:\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
|
|
$$
|
|
|
|
#### Double DQN is not enough
|
|
|
|
Because the slow-changing policy in an actor-critic setting
|
|
|
|
- the current and target value estimates remain too similar to avoid maximization bias.
|
|
- Target value of Double DQN: $r_t + \gamma Q^w'(s_{t+1},\mu_\theta(s_{t+1}))$
|
|
|
|
#### TD3: Twin Delayed Deep Deterministic policy gradient
|
|
|
|
Address overestimation bias:
|
|
|
|
- Double Q-learning is unbiased in tabular settings, but still slight overestimation with function approximation.
|
|
|
|
$$
|
|
y_1 = r + \gamma Q^{\theta_2'}(s', \pi_{\phi_1}(s'))
|
|
$$
|
|
$$
|
|
y_2 = r + \gamma Q^{\theta_1'}(s', \pi_{\phi_2}(s'))
|
|
$$
|
|
It is possible that $Q^{\theta_2}(s, \pi_{\phi_1}(s)) > Q^{\theta_1}(s, \pi_{\phi_1}(s))$
|
|
|
|
Clipped double Q-learning:
|
|
|
|
$$
|
|
y_1 = r + \gamma \min_{i=1,2} Q^{\theta_i'}(s', \pi_{\phi_i}(s'))
|
|
$$
|
|
|
|
High-variance estimates provide a noisy gradient.
|
|
|
|
Techniques in TD3 to reduce the variance:
|
|
|
|
- Update the policy at a lower frequency than the value network.
|
|
- Smoothing the value estimate:
|
|
$$
|
|
y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]
|
|
$$
|
|
|
|
Update target:
|
|
|
|
$$
|
|
y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]
|
|
$$
|
|
|
|
where $\epsilon\sim clip(\mathcal{N}(0, \sigma), -c, c)$
|
|
|
|
#### Other methods
|
|
|
|
- Generalizable Episode Memory for Deep Reinforcement Learning
|
|
- Distributed Distributional Deep Deterministic Policy Gradient
|
|
- Distributional critic
|
|
- N-step returns are used to update the critic
|
|
- Multiple distributed parallel actors
|
|
- Prioritized experience replay
|
|
- |