# CSE510 Deep Reinforcement Learning (Lecture 24)

## Cooperative Multi-Agent Reinforcement Learning (MARL)

This lecture introduces cooperative multi-agent reinforcement learning, focusing on formal models, value factorization, and modern algorithms such as QMIX and QPLEX.

## Multi-Agent Coordination Under Uncertainty

In cooperative MARL, multiple agents aim to maximize a shared team reward. The environment can be modeled using a Markov game or a Decentralized Partially Observable MDP (Dec-POMDP).

A transition is defined as:

$$
P(s' \mid s, a_{1}, \dots, a_{n})
$$

Parameter explanations:

- $s$: current global state.
- $s'$: next global state.
- $a_{i}$: action taken by agent $i$.
- $P(\cdot)$: environment transition function.

The shared return is:

$$
\mathbb{E}\left[\sum_{t=0}^{T} \gamma^{t} r_{t}\right]
$$

Parameter explanations:

- $\gamma$: discount factor.
- $T$: horizon length.
- $r_{t}$: shared team reward at time $t$.

### CTDE: Centralized Training, Decentralized Execution

Training uses global information (centralized), but execution uses local agent observations. This is critical for real-world deployment.

## Joint vs Factored Q-Learning

### Joint Q-Learning

In joint-action learning, one learns a full joint Q-function:

$$
Q_{tot}(s, a_{1}, \dots, a_{n})
$$

Parameter explanations:

- $Q_{tot}$: joint value for the entire team.
- $(a_{1}, \dots, a_{n})$: joint action vector across agents.

Problem:

- The joint action space grows exponentially in $n$.
- Learning is not scalable.

### Value Factorization

Instead of learning $Q_{tot}$ directly, we factorize it into individual utility functions:

$$
Q_{tot}(s, \mathbf{a}) = f(Q_{1}(s,a_{1}), \dots, Q_{n}(s,a_{n}))
$$

Parameter explanations:

- $\mathbf{a}$: joint action vector.
- $f(\cdot)$: mixing network combining individual Q-values.

The goal is to enable decentralized greedy action selection.

## Individual-Global-Max (IGM) Condition

The IGM condition enables decentralized optimal action selection:

$$
\arg\max_{\mathbf{a}} Q_{tot}(s,\mathbf{a})=

\big(\arg\max_{a_{1}} Q_{1}(s,a_{1}), \dots, \arg\max_{a_{n}} Q_{n}(s,a_{n})\big)
$$

Parameter explanations:

- $\arg\max_{\mathbf{a}}$: search for best joint action.
- $\arg\max_{a_{i}}$: best local action for agent $i$.
- $Q_{i}(s,a_{i})$: individual utility for agent $i$.

IGM makes decentralized execution optimal with respect to the learned factorized value.

### VDN (Value Decomposition Networks)

VDN assumes:

$$
Q_{tot}(s,\mathbf{a}) = \sum_{i=1}^{n} Q_{i}(s,a_{i})
$$

Parameter explanations:

- $Q_{i}(s,a_{i})$: value of agent $i$'s action.
- $\sum_{i=1}^{n}$: linear sum over agents.

Pros:

- Very simple, satisfies IGM.
- Fully decentralized execution.

Cons:

- Limited representation capacity.
- Cannot model non-linear teamwork interactions.

## QMIX: Monotonic Value Factorization

QMIX uses a state-conditioned mixing network enforcing monotonicity:

$$
\frac{\partial Q_{tot}}{\partial Q_{i}} \ge 0
$$

Parameter explanations:

- $\partial Q_{tot} / \partial Q_{i}$: gradient of global Q w.r.t. individual Q.
- $\ge 0$: ensures monotonicity required for IGM.

The mixing function is:

$$
Q_{tot}(s,\mathbf{a}) = f_{mix}(Q_{1}, \dots, Q_{n}; s)
$$

Parameter explanations:

- $f_{mix}$: neural network with non-negative weights.
- $s$: global state conditioning the mixing process.

Benefits:

- More expressive than VDN.
- Supports CTDE while keeping decentralized greedy execution.

## Theoretical Issues With Linear and Monotonic Factorization

Limitations:

- Linear models (VDN) cannot represent complex coordination.
- QMIX monotonicity limits representation power for tasks requiring non-monotonic interactions.
- Off-policy training can diverge in some factorizations.

## QPLEX: Duplex Dueling Multi-Agent Q-Learning

QPLEX introduces a dueling architecture that satisfies IGM while providing full representation capacity within the IGM class.

### QPLEX Advantage Factorization

QPLEX factorizes:

$$
Q_{tot}(s,\mathbf{a}) = \sum_{i=1}^{n} \lambda_{i}(s,\mathbf{a})\big(Q_{i}(s,a_{i}) - \max_{a'-{i}} Q_{i}(s,a'-{i})\big)

- \max_{\mathbf{a}} \sum_{i=1}^{n} Q_{i}(s,a_{i})
  $$

Parameter explanations:

- $\lambda_{i}(s,\mathbf{a})$: positive mixing coefficients.
- $Q_{i}(s,a_{i})$: individual utility.
- $\max_{a'-{i}} Q_{i}(s,a'-{i})$: per-agent baseline value.
- $\max_{\mathbf{a}}$: maximization over joint actions.

QPLEX Properties:

- Fully satisfies IGM.
- Has full representation capacity for all IGM-consistent Q-functions.
- Enables stable off-policy training.

## QPLEX Training Objective

QPLEX minimizes a TD loss over $Q_{tot}$:

$$
L = \mathbb{E}\Big[(r + \gamma \max_{\mathbf{a'}} Q_{tot}(s',\mathbf{a'}) - Q_{tot}(s,\mathbf{a}))^{2}\Big]
$$

Parameter explanations:

- $r$: shared team reward.
- $\gamma$: discount factor.
- $s'$: next state.
- $\mathbf{a'}$: next joint action evaluated by TD target.
- $Q_{tot}$: QPLEX global value estimate.

## Role of Credit Assignment

Credit assignment addresses: "Which agent contributed what to the team reward?"

Value factorization supports implicit credit assignment:

- Gradients into each $Q_{i}$ act as counterfactual signals.
- Dueling architectures allow each agent to learn its influence.
- QPLEX provides clean marginal contributions implicitly.

## Performance on SMAC Benchmarks

QPLEX outperforms:

- QTRAN
- QMIX
- VDN
- Other CTDE baselines

Key reasons:

- Effective realization of IGM.
- Strong representational capacity.
- Off-policy stability.

## Extensions: Diversity and Shared Parameter Learning

Parameter sharing encourages sample efficiency, but can cause homogeneous agent behavior.

Approaches such as CDS (Celebrating Diversity in Shared MARL) introduce:

- Identity-aware diversity.
- Information-based intrinsic rewards for agent differentiation.
- Balanced sharing vs agent specialization.

These techniques improve exploration and cooperation in complex multi-agent tasks.

## Summary of Lecture 24

Key points:

- Cooperative MARL requires scalable value decomposition.
- IGM enables decentralized action selection from centralized training.
- QMIX introduces monotonic non-linear factorization.
- QPLEX achieves full IGM representational capacity.
- Implicit credit assignment arises naturally from factorization.
- Diversity methods allow richer multi-agent coordination strategies.
