4.9 KiB
CSE510 Deep Reinforcement Learning (Lecture 16)
Deterministic Policy Gradient (DPG)
Learning Deterministic Policies
- Deterministic policy gradients [Silver et al., ICML 2014]
- Explicitly learn a deterministic policy.
a = \mu_\theta(s)
- Advantages
- Existing optimal deterministic policy for MDPs
- Naturally dealing with a continuous action space
- Expected to be more efficient than learning stochastic policies
- Computing stochastic gradient requires more samples, as it integrates over both state and action space.
- Deterministic gradient is preferable as it integrates over state space only.
Deterministic Policy Gradient
The objective function is:
J(\theta)=\int_{s\in S} \rho^{\mu}(s) r(s,\mu_\theta(s)) ds
where \rho^{\mu}(s) is the stationary distribution under the behavior policy \mu_\theta(s).
The policy gradient from the standard policy gradient theorem is:
\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta Q^{\mu_\theta}(s,a)]=\mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
Issues for DPG
The formulations up to now can only use on-policy data.
Deterministic policy can hardly guarantee sufficient exploration.
- Solution: Off-policy training using a stochastic behavior policy.
Off-Policy Deterministic Policy Gradient (Off-DPG)
Use a stochastic behavior policy \beta(a|s). The modified objective function is:
J(\mu_\theta)=\int_{s\in S} \rho^{\beta}(s) Q^{\mu_\theta}(s,\beta(s)) ds
The gradients are:
\begin{aligned}
\nabla_\theta J(\mu_\theta) &\approx \int_{s\in S} \rho^{\beta}(s) \nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)} ds\\
&= \mathbb{E}_{s\sim \rho^{\beta}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
\end{aligned}
Importance sampling is avoided in the actor due to the absence of integral over actions.
Policy Evaluation in DPG
Importance sampling can also be avoided in the critic.
Gradient TD-like algorithm can be directly applied to the critic.
\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2
Off-Policy Deterministic Actor-Critic
\delta_t=r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)
w_{t+1} = w_t + \alpha_w \delta_t \nabla_w Q^w(s_t,a_t)
\theta_{t+1} = \theta_t + \alpha_\theta \nabla_\theta \mu_\theta(s_t) \nabla_a Q^{\mu_\theta}(s_t,a_t)\vert_{a=\mu_\theta(s_t)}
Deep Deterministic Policy Gradient (DDPG)
Insights from DQN + Deterministic Policy Gradients
- Use a replay buffer
- Critic is updated every timestep (Sample from buffer, minibatch):
\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2
Actor is updated every timestep:
\nabla_a Q(s_t,a;w)|_{a=\mu_\theta(s_t)} \nabla_\theta \mu_\theta(s_t)
Smoothing target updated at every timestep:
w_{t+1} = \tau w_t + (1-\tau) w_{t+1}
\theta_{t+1} = \tau \theta_t + (1-\tau) \theta_{t+1}
Exploration: add noise to the action selection: a_t = \mu_\theta(s_t) + \mathcal{N}_t
Batch normalization used for training networks
Extension of DDPG
Overestimation bias is an issue of Q-learning in which the maximization of a noisy value estimate
DDPG:\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]
Double DQN is not enough
Because the slow-changing policy in an actor-critic setting
- the current and target value estimates remain too similar to avoid maximization bias.
- Target value of Double DQN:
r_t + \gamma Q^w'(s_{t+1},\mu_\theta(s_{t+1}))
TD3: Twin Delayed Deep Deterministic policy gradient
Address overestimation bias:
- Double Q-learning is unbiased in tabular settings, but still slight overestimation with function approximation.
y_1 = r + \gamma Q^{\theta_2'}(s', \pi_{\phi_1}(s'))
y_2 = r + \gamma Q^{\theta_1'}(s', \pi_{\phi_2}(s'))
It is possible that Q^{\theta_2}(s, \pi_{\phi_1}(s)) > Q^{\theta_1}(s, \pi_{\phi_1}(s))
Clipped double Q-learning:
y_1 = r + \gamma \min_{i=1,2} Q^{\theta_i'}(s', \pi_{\phi_i}(s'))
High-variance estimates provide a noisy gradient.
Techniques in TD3 to reduce the variance:
- Update the policy at a lower frequency than the value network.
- Smoothing the value estimate:
y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]
Update target:
y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]
where \epsilon\sim clip(\mathcal{N}(0, \sigma), -c, c)
Other methods
- Generalizable Episode Memory for Deep Reinforcement Learning
- Distributed Distributional Deep Deterministic Policy Gradient
- Distributional critic
- N-step returns are used to update the critic
- Multiple distributed parallel actors
- Prioritized experience replay