# CSE5313 Coding and information theory for data science (Lecture 21)

## Gradient coding

### Intro to Statistical Machine Learning

Given by the learning problem:

**Unknown** target function $f:X\to Y$.

Training data $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^n$.

Learning algorithm: $\mathbb{A}$ and Hypothesis set $\mathcal{H}$.

Goal is to find $g\approx f$,

Common hypothesis sets:

- Linear classifiers:

$$
f(x)=sign (wx^\top)
$$

- Linear regressors:

$$
f(x)=wx^\top
$$

- Neural networks:
  - Concatenated linear classifiers (or differentiable approximation thereof).

The dataset $\mathcal{D}$ is taken from unknown distribution $D$.

#### Common approach – Empirical Risk Minimization

- Q: How to quantify $f\approx g$ ?
- A: Use a loss function.
  - A function which measures the deviation between $g$'s output and $f$'s output.
- Using the chosen loss function, define two measures:
  - True risk: $\mathbb{E}_{D}[\mathbb{L}(f,g)]$
  - Empirical risk: $\frac{1}{n}\sum_{i=1}^n \ell(g(x_i),y_i)$

Machine learning is over $\mathbb{R}$.

### Gradient Descent (Motivation)

Parameterize $g$ using some real vector $\vec{w}$, we want to minimize $ER(\vec{w})$.

Algorithm:

- Initialize $\vec{w}_0$.
- For $t=1,2,\cdots,T$:
  - Computer $\nabla_{\vec{w}} ER(\vec{w})$.
  - $\vec{w}\gets \vec{w}-\eta\nabla_{\vec{w}} ER(\vec{w})$
  - Terminate if some stop condition are met.

Bottleneck: Calculating $\nabla_{\vec{w}} ER(\vec{w})=\frac{1}{n}\sum_{i=1}^n \nabla_{\vec{w}} \ell(g(x_i),y_i)$.

Potentially, $O(PN)$ where $N$ is number of points, dimension of feature space.

Solution: Parallelize.

#### Distributed Gradient Descent

Idea: use a distributed system with **master** and **workers**.

Problem: Stragglers (slow servers, 5-6 slower than average time).

Potential Solutions:

- Wait for all servers:
  - Accurate, but slow
- Sum results without the slowest ones
  - Less accurate, but faster
- Introduce redundancy
  - Send each $\mathcal{D}_i$ to more than one server.
  - Each server receives more than one $\mathcal{D}_i$.
  - Each server sends a linear combination of the partials gradient to its $\mathcal{D}_i$.
  - The master decodes the sum of partial gradients from the linear combination.

### Problem setups

System setup: 1 master $M$, $n$ workers $W_1,\cdots,W_n$.

A dataset $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^n$.

Each server $j$:

- Receives some $d$ data points $\mathcal{D}_j=\{(x_{j,i},y_{j,i})\}_{i=1}^d$. where $d$ is the replication factor.
- Computes a certain vector $v_i$ from each $(x_{j,i},y_{j,i})$.
- Returns a linear combination $u_i=\sum_{j=1}^n \alpha_j v_j$. to the master.

The master:

- Waits for the first $n-s$ $u_i$'s to arrive ($s$ is the straggler tolerance factor).
- Linear combines the $u_i$'s to get the final gradient.

Goal: Retrieve $\sum_i v_i$ regardless of which $n-s$ workers responded.

- Computation of the full gradient that tolerates any $s$ straggler.

The $\alpha_{i,j}$'s are fixed (do not depend on data). These form a **gradient coding matrix** $B\in\mathbb{C}^{n\times n}$.

Row $i$ has $d$ non-zero entries $\alpha_{i,j}$. at some positions.

The $\lambda_i$'s:

- Might depend on the identity of the $n-s$ responses.
- Nevertheless must exists in any case.

Recall:

- The master must be able to recover $\sum_i v_i$ form any $n-s$ responses.

Let $\mathcal{K}$ be the indices of the responses.

Let $B_\mathcal{K}$ be the submatrix of $B$ indexed by $\mathcal{K}$.

Must have:

- For every $\mathcal{K}$ of size $n-s$ there exists coefficients $\lambda_1,\cdots,\lambda_{n-s}$ such that:

$$
(\lambda_1,\cdots,\lambda_{n-s})B_\mathcal{K}=(1,1,1,1,\cdots,1)=\mathbb{I}
$$

Then if $\mathcal{K}=\{i_1,\cdots,i_{n-s}\}$ responded,

$$
(\lambda_1 v_{i_1},\cdots,\lambda_{n-s} v_{i_{n-s}})\begin{pmatrix}
u_{i_1}\\
u_{i_2}\\
\vdots\\
u_{i_{n-s}}
\end{pmatrix}=\sum_i v_i
$$

#### Definition of gradient coding matrix.

For replication factor $d$ and straggler tolerance factor $s$: $B\in\mathbb{C}^{n\times n}$ is a gradient coding matrix if:

- $\mathbb{I}$ is in th span of any $n-s$ rows.
- Every row of $B$ contains at most $d$ nonzero elements.

Grading coding matrix implies gradient coding algorithm:

- The master sends $S_i$ to worker $i$. where $i=1,\cdots,n$ are the nonzero indices of row $i$.
- Worker $i$:
  - Computes $\mathcal{D}_{i,\ell}\to v_{i,\ell}$ for $\ell=1,\cdots,d$.
  - Sends $u_i=\sum_{j=1}^d \alpha_{i,j}v_{i,j}$ to the master.
- Let $\mathcal{K}=\{i_1,\cdots,i_{n-s}\}$ be the indices of the first $n-s$ responses.
- Since $\mathbb{I}$ is in the span of any $n-s$ rows of $B$, there exists $\lambda_1,\cdots,\lambda_{n-s}$ such that $(\lambda_1,\cdots,\lambda_{n-s})((u_{i_1},\cdots,u_{i_{n-s}}))=\sum_i v_i$.

#### Construction of Gradient Coding Matrices

Goal:

- For a given straggler tolerance parameter $s$, we wish to construct a gradient coding matrix $B$ with the smallest possible $d$.

- Tools:

I. Cyclic Reed-Solomon codes over the complex numbers.
II. Definition of $\mathcal{C}^\perp$ (dual of $\mathcal {C}$) and $\mathcal{C}^R$ (reverse of $\mathcal{C}$).
III. A simple lemma about MDS codes.

Recall: An $n,k$ Reed-Solomon code over a field $\mathcal{F}$ is as follows.

- Fix distinct $\alpha_1,\cdots,\alpha_n-1\in\mathcal{F}$.
- $\mathcal{C}=\{f(\alpha_1),f(\alpha_2),\cdots,f(\alpha_n-1)\}$.
- Dimension $k$ and minimum distance $n-k+1$ follow from $\mathcal{F}$ being a field.
- Also works for $\mathcal {F}=\mathbb{C}$.

### I. Cyclic Reed-Solomon codes over the complex numbers.

The following Reed-Solomon code over the complex numbers is called a cyclic code.

- Let $i=\sqrt{-1}$.
- For $j\in \{0,\cdots,n-1\}$, choose $a_j=e^{2\pi i j/n}$. The $a_j$'s are roots of unity of order $n$.
- Use these $a_j$'s to define a Reed-Solomon code as usual.

This code is cyclic:

- Let $c=(f_c(a_0),f_c(a_1),\cdots,f_c(a_{n-1}))$ for some $f_c(x)\in \mathbb{C}[x]$.
- Need to show that $c'=f_c(a_j)$ for all $j\in \{0,\cdots,n-1\}$.

$$
c'=(f_c(a_1),f_c(a_2),\cdots,f_c(a_{n-1}),f_c(a_0))=(f_c(a_0),f_c(a_1),\cdots,f_c(a_{n-1}))
$$

### II. Dual and Reversed Codes

- Let $\mathcal{C}=[n,k,d]_{\mathbb{F}}$ be an MDS code.

#### Definition for dual code of $\mathcal{C}$

The dual code of $\mathcal{C}$ is

$$
\mathcal{C}^\perp=\{c'\in \mathbb{F}^n|c'c^\top=0\text{ for all }c\in \mathcal{C}\}
$$

Claim: $\mathcal{C}^\perp$ is an $[n,n-k,k+1]_{\mathbb{F}}$ code.

#### Definition for reversed code of $\mathcal{C}$

The reversed code of $\mathcal{C}$ is

$$
\mathcal{C}^R=\{(c_{n-1},\cdots,c_0)|(c_0,\cdots,c_{n-1})\in \mathcal{C}\}
$$

We claim that if $\mathcal{C}$ is cyclic, then $\mathcal{C}^R$ is cyclic.

### III. Lemma about MDS codes

Let $\mathcal{C}=[n,k,n-k+1]_{\mathbb{F}}$ be an MDS code.

#### Lemma

For any subset $\mathcal{K}\subset \{0,\cdots,n-1\}$, of size $n-k+1$ there exists $c\in \mathcal{C}$ whose support (set of nonzero indices) is $\mathcal{K}$.

<details>
<summary>Proof</summary>

Let $G\in \mathbb{F}^{k\times n}$ be a generator matrix, and let $G_{\mathcal{K}^c}\in \mathbb{F}^{k\times (k-1)}$ be its restriction to columns not indexed by $\mathcal{K}$.

$G_{\mathcal{K}^c}$ has more rows than columns, so there exists $v\in \mathbb{F}^{k}$ such that $vG_{\mathcal{K}^c}=0$.

So $c=vG$ has at least $|\mathcal{K}^c|=k-1$ zeros inn entires indexed by $\mathcal{K}^c$.

The remaining $n-(k-1)=d$ entries of $c$, indexed by $\mathcal{K}$, must be nonzero.

Thus the suppose of $c$ is $\mathcal{K}$.

</details>

### Construct gradient coding matrix

Consider nay $n$ workers and $s$ stragglers.

Let $d=s+1$

Let $\mathcal{C}=[n,n-s]_{\mathbb{C}}$ be the cyclic RS code build by I.

Then by III, there exists $c\in \mathcal{c}$ whose support is the $n-(n-s)+1=s+1$ first entires.

Denote $c=(\beta_1,\cdots,\beta_{s+1},0,0,\cdots,0)$. for some nonzero $\beta_1,\cdots,\beta_{s+1}$.

Build:

$B\in \mathbb{C}^{n\times n}$ whose columns are all cyclic shifts of $c$.

We claim that $B$ is a gradient coding matrix.

$$
\begin{pmatrix}
\beta_1     & 0           &        & 0          & \beta_{s+1} & \beta_s& \cdots & \beta_2 \\
\vdots      & \beta_1     &        & \vdots     & 0           & \beta_{s+1} &  & \vdots \\
\beta_{s+1} & \vdots      &        &            & \vdots      & 0 &  & \beta_{s+1}\\
0           & \beta_{s+1} & \ddots & 0          &             &\vdots & \dots & 0\\
\vdots      & 0           & \ddots & \beta_1     & & & & &\\
            & \vdots      & \ddots & \vdots     & & & &0\\
0           & 0           &        & \beta_{s+1}& \beta_s& \beta_{s-1}& \cdots & \beta_1\\
\end{pmatrix}
$$

<details>
<summary>Proof</summary>

Every row is a codeword in $\mathcal{C}^R$.

- Specifically, a shift of $(0,\cdots,0,\beta_{s+1},\cdots,\beta_1)$.
- Then every row contains $\leq d=s+1$ nonzeros.

$\mathcal{I}$ is in the span of any $n-s$ rows of $B$.

- Observe that $\mathcal{I}\in \mathcal{C}$, (evaluate the polynomial $f(x)=1$ at $\alpha_1,\cdots,\alpha_n$).
- Then $\mathcal{I}\in \mathcal{C}^R$.
- Therefore, it suffices to show that any $n-s$ span $\mathcal{C}^R$.
- Since $\dim \mathcal{C}=\dim \mathcal{C}^R=n-s$, it suffices to show that any $n-s$ rows are independent.

Observe: The left most $n-s$ columns are linearly independent, and therefore span $\mathcal{C}$.

Assume for contradiction there exists $n-s$ dependent rows.

Then $\exists v\in \mathbb{C}^{n}$ such that $vB=0$.

$v$ is orthogonal to the basis of $\mathcal{C}$.

So $v\in \mathcal{C}^\perp$.

From II, $\mathcal{C}^\perp$ is an $[n,s]$ MDS code, and hence every $v\in \mathcal{C}^\perp$ is of Hamming weight $\geq n-s+1$.

This is a contradiction.

</details>

### Bound for gradient coding

We want $s$ to be large and $d$ to be small.

How small can $d$ with respect to $s$?

- A: Build a bipartite graph.
  - Left side: $n$ workers $W_1,\cdots,W_n$.
  - Right side: $n$ partial datasets $D_1,\cdots,D_n$.
  - Connect $W_i$ to $D_i$ if worker $i$ contains $D_i$.
    - Equivalently if $B_{i,i}\neq 0$.
  - $\deg (W_i) = d$ by definition.
  - $\deg (\mathcal{D}_j)\geq s+1$.
  - Sum degree on left $nd$ and right $\geq n(s+1)$.
  - So $d\geq s+1$.

We can break the lower bound using approximate computation.