66 lines
2.5 KiB
Markdown
66 lines
2.5 KiB
Markdown
# CSE510 Deep Reinforcement Learning (Lecture 18)
|
|
|
|
## Model-based RL framework
|
|
|
|
Model Learning with High-Dimensional Observations
|
|
|
|
- Learning model in a latent space with observation reconstruction
|
|
- Learning model in a latent space without observation reconstruction
|
|
- Learning model in the observation space (i.e., videos)
|
|
|
|
### Naive approach:
|
|
|
|
If we knew $f(s_t,a_t)=s_{t+1}$, we could use the tools from last week. (or $p(s_{t+1}| s_t, a_t)$ in the stochastic case)
|
|
|
|
So we can learn $f(s_t,a_t)$ from data, and _then_ plan through it.
|
|
|
|
Model-based reinforcement learning version **0.5**:
|
|
|
|
1. Run base polity $\pi_0$ (e.g. random policy) to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
|
|
2. Learn dynamics model $f(s_t,a_t)$ to minimize $\sum_{i}\|f(s_i,a_i)-s_{i+1}\|^2$
|
|
3. Plan through $f(s_t,a_t)$ to choose action $a_t$
|
|
|
|
Sometime, it does work!
|
|
|
|
- Essentially how system identification works in classical robotics
|
|
- Some care should be taken to design a good base policy
|
|
- Particularly effective if we can hand-engineer a dynamics representation using our knowledge of physics, and fit just a few parameters
|
|
|
|
However, Distribution mismatch problem becomes worse as we use more
|
|
expressive model classes.
|
|
|
|
Version 0.5: collect random samples, train dynamics, plan
|
|
|
|
- Pro: simple, no iterative procedure
|
|
- Con: distribution mismatch problem
|
|
|
|
Version 1.0: iteratively collect data, replan, collect data
|
|
|
|
- Pro: simple, solves distribution mismatch
|
|
- Con: open loop plan might perform poorly, esp. in stochastic domains
|
|
|
|
Version 1.5: iteratively collect data using MPC (replan at each step)
|
|
|
|
- Pro: robust to small model errors
|
|
- Con: computationally expensive, but have a planning algorithm available
|
|
|
|
Version 2.0: backpropagate directly into policy
|
|
|
|
- Pro: computationally cheap at runtime
|
|
- Con: can be numerically unstable, especially in stochastic domains
|
|
- Solution: model-free RL + model-based RL
|
|
|
|
Final version:
|
|
|
|
1. Run base polity $\pi_0$ (e.g. random policy) to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
|
|
2. Learn dynamics model $f(s_t,a_t)$ to minimize $\sum_{i}\|f(s_i,a_i)-s_{i+1}\|^2$
|
|
3. Backpropagate through $f(s_t,a_t)$ into the policy to optimized $\pi_\theta(s_t,a_t)$
|
|
4. Run the policy $\pi_\theta(s_t,a_t)$ to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
|
|
5. Goto 2
|
|
|
|
## Model Learning with High-Dimensional Observations
|
|
|
|
- Learning model in a latent space with observation reconstruction
|
|
- Learning model in a latent space without observation reconstruction
|
|
|