102 lines
2.5 KiB
Markdown
102 lines
2.5 KiB
Markdown
# CSE510 Deep Reinforcement Learning (Lecture 25)
|
|
|
|
> Restore human intelligence
|
|
|
|
## Linear Value Factorization
|
|
|
|
[link to paper](https://arxiv.org/abs/2006.00587)
|
|
|
|
### Why Linear Factorization works?
|
|
|
|
- Multi-agent reinforcement learning are mostly emprical
|
|
- Theoretical Model: Factored Multi-Agent Fitted Q-Iteration (FMA-FQI)
|
|
|
|
#### Theorem 1
|
|
|
|
It realize **Counterfactual** credit assignment mechanism.
|
|
|
|
Agent $i$:
|
|
|
|
$$
|
|
Q_i^{(t+1)}(s,a_i)=\mathbb{E}_{a_{-i}'}\left[y^{(t)}(s,a_i\oplus a_{-i}')\right]-\frac{n-1}{n}\mathbb{E}_{a'}\left[y^{(t)}(s,a')\right]
|
|
$$
|
|
|
|
Here $\mathbb{E}_{a_{-i}'}\left[y^{(t)}(s,a_i\oplus a_{-i}')\right]$ is the evaluation of $a_i$.
|
|
|
|
and $\mathbb{E}_{a'}\left[y^{(t)}(s,a')\right]$ is the baseline
|
|
|
|
The target $Q$-value: $y^{(t)}(s,a)=r+\gamma\max_{a'}Q_{tot}^{(t)}(s',a')$
|
|
|
|
#### Theorem 2
|
|
|
|
it has local convergence with on-policy training
|
|
|
|
##### Limitations of Linear Factorization
|
|
|
|
Linear: $Q_{tot}(s,a)=\sum_{i=1}^{n}Q_{i}(s,a_i)$
|
|
|
|
Limited Representation: Suboptimal (Prisoner's Dilemma)
|
|
|
|
|a_2\a_2| Action 1 | Action 2 |
|
|
|---|---|---|
|
|
|Action 1| **8** | -12 |
|
|
|Action 2| -12 | 0 |
|
|
|
|
After linear factorization:
|
|
|
|
|a_2\a_2| Action 1 | Action 2 |
|
|
|---|---|---|
|
|
|Action 1| -6.5 | -5 |
|
|
|Action 2| -5 | **-3.5** |
|
|
|
|
#### Theorem 3
|
|
|
|
it may diverge with off-policy training
|
|
|
|
### Perfect Alignment: IGM Factorization
|
|
|
|
- Individual-Global Maximization (IGM) Constraint
|
|
|
|
$$
|
|
\argmax_{a}Q_{tot}(s,a)=(\argmax_{a_1}Q_1(s,a_1), \dots, \argmax_{a_n}Q_n(s,a_n))
|
|
$$
|
|
|
|
- IGM Factorization: $Q_{tot} (s,a)=f(Q_1(s,a_1), \dots, Q_n(s,a_n))$
|
|
- Factorization function $f$ realizes all functions satsisfying IGM.
|
|
|
|
- FQI-IGM: Fitted Q-Iteration with IGM Factorization
|
|
|
|
#### Theorem 4
|
|
|
|
Convergence & optimality. FQI-IGM globally converges to the optimal value function in multi-agent MDPs.
|
|
|
|
### QPLEX: Multi-Agent Q-Learning with IGM Factorization
|
|
|
|
[link to paper](https://arxiv.org/pdf/2008.01062)
|
|
|
|
IGM: $\argmax_a Q_{tot}(s,a)=\begin{pamtrix}
|
|
\argmax_{a_1}Q_1(s,a_1) \\
|
|
\dots \\
|
|
\argmax_{a_n}Q_n(s,a_n)
|
|
\end{pmatrix}
|
|
$
|
|
|
|
Core idea:
|
|
|
|
- Fitting well the values of optimal actions
|
|
- Approximate the values of non-optimal actions
|
|
|
|
QPLEX Mixing Network:
|
|
|
|
$$
|
|
Q_{tot}(s,a)=\sum_{i=1}^{n}\max_{a_i'}Q_i(s,a_i')+\sum_{i=1}^{n} \lambda_i(s,a)(Q_i(s,a_i)-\max_{a_i'}Q_i(s,a_i'))
|
|
$$
|
|
|
|
Here $\sum_{i=1}^{n}\max_{a_i'}Q_i(s,a_i')$ is the baseline $\max_a Q_{tot}(s,a)$
|
|
|
|
And $Q_i(s,a_i)-\max_{a_i'}Q_i(s,a_i')$ is the "advantage".
|
|
|
|
Coefficients: $\lambda_i(s,a)>0$, **easily realized and learned with neural networks**
|
|
|
|
> Continue next time...
|