# CSE5313 Coding and information theory for data science (Lecture 21)

## Approximate Gradient Coding

### Exact gradient computation and approximate gradient computation

In the previous formulation, the gradient $\sum_i v_i$
is computed exactly.

- Accurate
- Requires $d \geq s + 1$ (high replication factor). 
- Need to know $s$ in advance!

However:

- Approximate gradient computation are very common!
  - E.g., stochastic gradient descent.
- Machine learning is inherently inaccurate.
  - Relies on biased data, unverified assumptions about model, etc.

Idea: If we relax the exact computation requirement, can have $d < s + 1$?

- No fixed $s$ anymore.

Approximate computation:

- Exact computation: $\nabla \triangleq v = \sum_i v_i = (1, \cdots, 1) v_1, \cdots, v_n^\top$.
⊤.
- Approximate computation: $\nabla \triangleq v = \sum_i v_i \approx u v_1, \cdots, v_n^\top$,
  - Where $d_2(u, \mathbb{I})$ is "small" ($d_2(u, v) =\sqrt{ \sum_i (u_i - v_i)^2}$).
  - Why?
- Lemma: Let $v_u = u (v_1, \cdots, v_n)^\top$. If $d_2(u, \mathbb{I}) \leq \epsilon$ then $d_2(v, v_u) \leq \epsilon \cdot \ell_{spec}(V)$.
  - $V$ is the matrix whose rows are the $v_i$'s.
  - $\ell_{spec}$ is the spectral norm (positive sqrt of maximum eigenvalue of $V^\top V$).
- Idea: Distribute $S_1, \cdots, S_n$ as before, and
  - as the master gets more and more responses,
  - it can reconstruct $u v_1, \cdots, v_n^\top$,
  - such that $d_2(u, \ell)$ gets smaller and smaller.

> [!NOTE]
> $d \geq s + 1$ no longer holds.
> $s$ no longer a parameter of the system, but $s = n - \#responses$ at any given time.

### Trivial Scheme

Off the bat, the "do nothing" approach:

- Send $S_i$ to worker $i$, i.e., $d = 1$.
- Worker $i$ replies with the $i$'th partial gradient $v_i$.
- The master averages up all the responses.

How good is that?

- For $u = \frac{n}{n-s} \cdot \mathbb{I}$, the factor $\frac{n}{n-s}$ corrects the $\frac{1}{n}$ in $v_i = \frac{1}{n} \cdot \nabla \text{ on } S_i$.
- Is this $\approx \sum_i v_i$? In other words, what is $d_2(\frac{n}{n-s} \cdot \mathbb{I}, \mathbb{I})$?

Trivial scheme: $\frac{n}{n-s} \cdot \mathbb{I}$ approximation.

Must do better than that!

### Roadmap

- Quick reminder from linear algebra.
  - Eigenvectors and orthogonality.
- Quick reminder from graph theory.
  - Adjacency matrix of a graph.
- Graph theoretic concept: expander graphs.
  - "Well connected" graphs.
  - Extensively studied.
- An approximate gradient coding scheme from expander graphs.

### Linear algebra - Reminder

- Let $A \in \mathbb{R}^{n \times n}$.
- If $A v = \lambda v$ then $\lambda$ is an eigenvalue and $v$ is an eigenvector.
- $v_1, \cdots, v_n \in \mathbb{R}^n$ are orthonormal:
  - $\|v_i\|_2 = 1$ for all $i$.
  - $v_i \cdot v_j^\top = 0$ for all $i \neq j$.
- Nice property: $\| \alpha_1 v_1 + \cdots + \alpha_n v_n \|_2^2 = \sqrt{\sum_i \alpha_i^2}$.
- $A$ is called symmetric if $A = A^\top$.
- Theorem: A **real and symmetric** matrix has an orthonormal basis of eigenvectors.
  - That is, there exists an orthonormal basis $v_1, \cdots, v_n$ such that $A v_i = \lambda_i v_i$ for some $\lambda_i$'s.

### Graph theory - Reminder

- Undirected graph $G = V, E$.
- $V$ is a vertex set, usually $n = 1,2, \cdots, n$.
- $E \subseteq \binom{V}{2}$ is an edge set (i.e., $E$ is a collection of subsets of $V$ of size two).
- Each edge $e \in E$ is of the form $e = (a, b)$ for some distinct $a, b \in V$.
- Spectral graph theory:
  - Analyze properties of graphs (combinatorial object) using matrices (algebraic object).
  - Specifically, for a graph $G$ let $A_G \in \{0,1\}^{n \times n}$ be the adjacency matrix of $G$.
  - $A_{i,j} = 1$ if and only if $\{i,j\} \in E$ (otherwise 0).
  - $A$ is real and symmetric.
  - Therefore, has an orthonormal basis of eigenvectors.

#### Some nice properties of adjacency matrices

- Let $G = (V, E)$ be $d$-regular, with adjacency matrix $A_G$ whose (real) eigenvalues $\lambda_1 \geq \cdots \geq \lambda_n$.
- Some theorems:
  - $\lambda_1 = d$.
  - $\lambda_n \geq -d$, equality if and only if $G$ is bipartite.
  - $A_G \mathbb{I}^\top = \lambda_1 \mathbb{I}^\top = d \mathbb{I}^\top$ (easy to show!).
    - Does that ring a bell? ;)
  - If $\lambda_1 = \lambda_2$ then $G$ is not connected.

#### Expander graphs - Intuition.

- An important family of graphs.
- Multiple applications in:
  - Algorithms, complexity theory, error correcting codes, etc.
- Intuition: A graph is called an expander if there are no "lonely small sets" of nodes.
- Every set of at most $n/2$ nodes is "well connected" to the remaining nodes in the graph.
- A bit more formally:
  - An infinite family of graphs $G_n$ $n=1$ $\infty$ (where $G_n$ has $n$ nodes) is called an **expander family**, if the "minimal connectedness" of small sets in $G_n$ does not go to zero with $n$.

#### Expander graphs - Definitions.

- All graphs in this lecture are $d$-regular, i.e., all nodes have the same degree $d$.
- For sets of nodes $S, T \subseteq V$, let $E(S, T)$ be the set of edges between $S$ and $T$. I.e., $E(S, T) = \{(i,j) \in E | i \in S \text{ and } j \in T\}$.
- For a set of nodes $S$ let:
  - $S^c = n \setminus S$ be its complement.
  - Let $\partial S = E(S, S^c)$ be the boundary of $S$.
  - I.e., the set of edges between $S$ and its complement $S^c$.
- The expansion parameter $h_G$ of $G$ is:
  - I.e., how many edges leave $S$, relative to its size.
  - How "well connected" $S$ is to the remaining nodes.

> [!NOTE]
> $h_G = \min_{S \subseteq V, |S| \leq n/2} \frac{\partial S}{|S|}$.

- An infinite family of $d$-regular graphs $(G_n)_{n=1}^\infty$ (where $G_n$ has $n$ nodes) is called an **expander family** if $h(G_n) \geq \epsilon$ for all $n$.
  - Same $d$ and same $\epsilon$ for all $n$.
- Expander families with large $\epsilon$ are hard to build explicitly.
- Example: (Lubotsky, Philips and Sarnak '88)
  - $V = \mathbb{Z}_p$ (prime).
  - Connect $x$ to $x + 1, x - 1, x^{-1}$.
  - $d = 3$, very small $\epsilon$.
- However, **random** graphs are expanders with high probability.

#### Expander graphs - Eigenvalues

- There is a strong connection between the expansion parameter of a graph and the eigenvalues $\lambda_1 \geq \cdots \geq \lambda_n$ of its adjacency matrix.
- Some theorems (no proof):
  - $\frac{d-\lambda_2}{2} \leq h_G \leq \sqrt{2d (d - \lambda_2)}$.
  - $d - \lambda_2$ is called the **spectral gap** of $G$.
    - If the spectral gap is large, $G$ is a good expander.
    - How large can it be?
  - Let $\lambda = \max \{|\lambda_2|, |\lambda_n|\}$. Then $\lambda \geq 2 d - 1 - o_n(1)$. (Alon-Boppana Theorem).
  - Graphs which achieve the Alon-Boppana bound (i.e., $\lambda \leq 2 d - 1$) are called **Ramanujan graphs**.
    - The "best" expanders.
    - Some construction are known.
    - Efficient construction of Ramanujan graphs for all parameters is very recent (2016).

#### Approximate GC from Expander Graphs

Back to approximate gradient coding.

- Let $d$ be any replication parameter.
- Let $G$ be an expander graph (i.e., taken from an infinite expander family $(G_n)_{n=1}^\infty$)
  - With eigenvalues $\lambda_1 \geq \cdots \geq \lambda_n$, and respective eigenvectors $w_1, \cdots, w_n$
  - Assume $\|w_1\|_2 =\| w_2\|_2 = \cdots = \|w_n\|_2 = 1$, and $w_i w_j^\top = 0$ for all $i \neq j$.
- Let the gradient coding matrix $B=\frac{1}{d} A_G$.
  - The eigenvalues of $B$ are $\mu_1 = 1\geq \mu_2 \geq \cdots \geq \mu_n$. where $\mu_i = \frac{\lambda_i}{d}$.
  - Let $\mu = \max \{|\mu_2|, |\mu_n|\}$.
  - $d$ nonzero entries in each row $\Rightarrow$ Replication factor $d$.
- Claim: For any number of stragglers $s$, we can get close to $\mathbb{I}$.
  - Much better than the trivial scheme.
  - Proximity is a function of $d$ and $\lambda$.
- For every $s$ and any set $\mathcal{K}$ of $n - s$ responses, we build an "decoding vector".
  - A function of $s$ and of the identities of the responding workers.
  - Will be used to linearly combine the $n - s$ responses to get the approximate gradient.
- Let $w_{\mathcal{K}} \in \mathbb{R}^n$ such that $(w_{\mathcal{K}})_i = \begin{cases} -1 & \text{if } i \notin \mathcal{K} \\ \frac{s}{n-s} & \text{if } i \in \mathcal{K} \end{cases}$.

Lemma 1: $w_{\mathcal{K}}$ is spanned by $w_2, \cdots, w_n$, the $n - 1$ last eigenvectors of $A_G$.

<details>
<summary>Proof</summary>

$w_2, \cdots, w_n$ are independent, and all orthogonal to $w_1 = \mathbb{I}$.

$\Rightarrow$ The span of $w_2, \cdots, w_n$ is exactly all vectors whose sum of entries is zero.

Sum of entries of $w_{\mathcal{K}}$ is zero $\Rightarrow$ $w_{\mathcal{K}}$ is in their span.

Corollary: $w_{\mathcal{K}} = \alpha_2 w_2 + \cdots + \alpha_n w_n$ for some $\alpha_i$'s in $\mathbb{R}$.
</details>

Lemma 2: From direct computation, the norm of $w_{\mathcal{K}} = \alpha_2 w_2 + \cdots + \alpha_n w_n$ is $\frac{ns}{n-s}$.

Corollary: $w_{\mathcal{K}}^2 = \sum_{i=2}^n \alpha_i^2 = \frac{ns}{n-s}$ (from Lemma 2 + orthonormality of $w_2, \cdots, w_n$).

The scheme:

- If the set of responses is $\mathcal{K}$, the decoding vector is $w_{\mathcal{K}} + \ell_2$.
- Notice that $\operatorname{supp}(w_{\mathcal{K}} + \ell_2) = \mathcal{K}$.
- The responses the master receives are the rows of $B v_1, \cdots, v_n^\top$ indexed by $\mathcal{K}$.
- $\Rightarrow$ The master can compute $w_{\mathcal{K}} + \ell_2 B v_1, \cdots, v_n^\top$.

Left to show: How close is $w_{\mathcal{K}} + \ell_2 B$ to $\ell_2$?

<details>
<summary>Proof</summary>

Recall that:

1. $w_{\mathcal{K}} = \alpha_2 w_2 + \cdots + \alpha_n w_n$.
2. $w_i$'s are eigenvectors of $A_G$ (with eigenvalues $\lambda_i$) and of $B = \frac{1}{d} A_G$ (with eigenvalues $\mu_i = \frac{\lambda_i}{d}$).

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = d_2 (\mathbb{I} + \alpha_2 w_2 + \cdots + \alpha_n w_n B, \mathbb{I})$ (from 1.)

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = d_2 (\mathbb{I} + \alpha_2 \mu_2 w_2 + \cdots + \alpha_n \mu_n w_n, \mathbb{I})$ (eigenvalues of $B$, and $\mu_1 = 1$)

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = d_2 (\mathbb{I} + \alpha_2 \mu_2 w_2 + \cdots + \alpha_n \mu_n w_n , \mathbb{I})$ (by def.).

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = \|\sum_{i=2}^n \alpha_i \mu_i w_i\|_2$

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = \|\sum_{i=2}^n \alpha_i \mu_i w_i\|_2$ (w_i's are orthonormal).

$d_2 (w_{\mathcal{K}} + \mathbb{I} B, \mathbb{I}) = \sqrt{\sum_{i=2}^n \alpha_i^2 \mu_i^2}$ (w_i's are orthonormal).

</details>

#### Improvement factor

Corollary: If $B = \frac{1}{d} A_G$ for a (d-regular) Ramanujan graph $G$,

- $\Rightarrow$ improvement factor $\approx \frac{2}{d}$.
- Some explicit constructions of Ramanujan graphs (Lubotsky, Philips and Sarnak '88)
  - with 2 $\frac{2}{d}$ $\approx$ 0.5!

### Recap

- Expander graph: A $d$-regular graph with no lonely small subsets of nodes.
  - Every subset with $\leq n/2$ has a large ratio $\partial S / S$ (not $\rightarrow 0$ with $n$).
  - Many constructions exist, random graph is expander w.h.p.
  - The expansion factor is determined by the spectral gap $d - \lambda_2$,
  - Where $\lambda = \max \{\lambda_2, \lambda_n\}$, and $\lambda_1 = d \geq \lambda_2 \geq \cdots \geq \lambda_n$ are the eigenvalues of $A_G$.
  - "Best" expander = Ramanujan graph = has $\lambda \leq 2 d - 1$.
- "Do nothing" approach: approximation $\frac{ns}{n-s}$.
- Approximate gradient coding:
  - Send $d$ subsets $S_{j1}, \cdots, S_{jd}$ to each node $i$, which returns a linear combination according to a coefficient matrix $B$.
  - Let $B = \frac{1}{d} A_G$, for $G$ a Ramanujan graph: approximation $\frac{\lambda}{d} \frac{ns}{n-s}$.
  - Up to 50% closer than "do nothing", at the price of higher computation load

>[!NOTE]
> Faster = more computation load.