# CSE5313 Coding and information theory for data science (Lecture 23)

## Coded Computing

### Motivation

Some facts:

- Moore's law is saturating.
  - Improving CPU performance is hard.
- Modern datasets are growing remarkably large.
  - E.g., TikTok, YouTube.
- Learning tasks are computationally heavy.
  - E.g., training neural networks.

Solution: Distributed Computing for Scalability

- Offloading computation tasks to multiple computation nodes.
- Gather and accumulate computation results.
- E.g., Apache Hadoop, Apache Spark, MapReduce.

### General Framework

- The system involves 1 master node and $P$ worker nodes.
- The master has a dataset $D$ and wants $f(D)$, where $f$ is some function.
- The master partitions $D=(D_1,\cdots,D_P)$, and sends $D_i$ to node $i$.
- Every node $i$ computes $g(D_i)$, where $g$ is somem function.
- Finally, the master collects $g(D_1),\cdots,g(D_P)$ and computes $f(D)=h(g(D_1),\cdots,g(D_P))$, where $h$ is some function.

#### Challenges

Stragglers

- Nodes that are significantly slower than the others.

Adversaries

- Nodes that return errounous results.
  - Computation/communication errors.
  - Adversarial attacks.

Privary

- Nodes may be curious about the dataset.

### Resemblance to communication channel

Suppose $f,g=\operatorname{id}$, and let $D=(D_1,\cdots,D_P)\in \mathbb{F}^p$ a message.

- $D_i$ is a field element
- $\mathbb{F}$ could be $\mathbb{R}$ or $\mathbb{C}$, $\mathbb{F}^q$.

Observation: This is a distributed storage system.

- An erasure - node that does not respond.
- An error - node that returns errounous results.

Solution:

- Add redundancy to the message
- Error-correcting codes.

### Coded Distributed Computing

- The master partitions $D$ and encodes it before sending to $P$ workers.
- Workers perform computations on coded data $\tilde{D}$ and generate coded results $g(\tilde{D})$.
- The master decode the coded results and obtain $f(D)=h(g(\tilde{D}))$.

### Outline

Matrix-Vector Multiplication

- MDS codes.
- Short-Dot codes.

Matrix-Matrix Multiplication

- Polynomial codes.
- MatDot codes.

Polynomial Evaluation

- Lagrange codes.
- Application to BLockchain.

### Trivial solution - replication

Why no straggler tolerance?

- We employ an individual worker node $i$ to compute $y_i=(a_i,\ldots,a_{iN})\cdot (x_1,\ldots,x_N)^T$.

Replicate the computation?

- Let $r+1$ nodes compute every $y_i$.

We need $P=rM+M$ worker nodes to tolerate $r$ erasures and $\lfloor \frac{r}{2}\rfloor$ adversaries.

### Use of MDS codes

Let $2|M$ and $P=3$.

Let $A_1,A_2$ be submatrices of $A$ such that $A=[A_1^\top|A_2^\top]^\top$.

- Worker node 1 conputes $A_1\cdot x$.
- Worker node 2 conputes $A_2\cdot x$.
- Worker node 3 conputes $(A_1+A_2)\cdot x$.

Observation: the results can be obtained from any two worker nodes.

Let $G\in \mathbb{F}^{M\times P}$ be the generator matrix of an $(P,M)$ MDS code.

The master node computes $F=G^\top A\in \mathbb{F}^{P\times N}$.

Every worker node $i$ computers $F_i\cdot x$.

- $F_i=(G^\top A)_i$ is the i-th row of $G^\top A$.

Notice that $Fx=G^\top A\cdot x=G^\top y$ is the codeword of $y$.

Node $i$ computes an entry in this codeword.

$1$ response = $1$ entyr of the codeword.

The master does **not** need all workers to respond to obtain $y$.

- The MDS property allows decoding from any $M$ $y_i$'s
- This scheme tolerates $P-M$ erasures, and the recovery threshold $K=M$.
- We need $P=r+M$ worker nodes to tolerate $r$ stragglers or $\frac{r}{2}$ adversaries.
  - With replication, we need $P=rM+M$ worker nodes.

#### Potential improvements for MDS codes

- The matrix $A$ is usually a (trained) model, and $x$ is the data (feature vector).
- $x$ is transmitted frequently, while the row of $A$ (or $G^\top A$) is communicated in advance.
- Every worker needs to receive the entire $x$ and compute the dot product.
- Communication-heavy
- Can we design a scheme that allows every node only receive only a part of $x$?

### Short-Dot codes

[link to paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8758338)

We want to create a matrix $F\in \mathbb{F}^{P\times M}$ from $A$ such that:

- Every node computes $F_i\cdot x$.
- Every $K$ rows linearly span the row space of $A$.
- Each row of $F$ contains at most $s$ non-zero entries.

In the MDS method, $F=G^\top A$.

- The recovery threshold $K=M$.
- Every worker node needs to receive $s=N$ symbols (the entire x)

No free lunch

Can we trade the recovery threshold $K$ for a smaller $s$?

- Every worker node receives less than $N$ symbols.
- The master will need more than $M$ responses to recover the computation result.

#### Construction of Short-Dot codes

Choose a super-regular matrix $B\in \mathbb{F}^{P\times K}$, where $P$ is the number of worker nodes.

- A matrix is supper-regular if every square submatrix is invertible.
- Lagrange/Cauchy matrix is super-regular (next lecture).

Create matrix $\tilde{A}$ by stacking some $Z\in \mathbb{F}^{(K-M)\times N}$ below matrix $A$.

Let $F=B\cdot \tilde{A}\in \mathbb{F}^{P\times N}$.

**Short-Dot**: create matrix $F\in \mathbb{F}^{P\times N}$ such that:

- Every $K$ rows linearly span the row space of $A$.
- Each row of $F$ contains at most $s=\frac{P-K+M}{P}$. $N$ non-zero entries (sparse).

#### Recovery of Short-Dot codes

Claim: Every $K$ rows of $F$ linearly span the row space of $A$.

<details>
<summary>Proof</summary>

Since $B$ is supper-regular, it is also MDS, i.e., every $K\times K$ submatrix of $B$ is invertible.

Hence, every row of $A$ can be represented as a linear combination of any $K$ rows of $F$.

That is, for every $\mathcal{X}\subseteq[P],|\mathcal{X}|=K$, we can have $\tilde{A}=(B^{\mathcal{X}})^{-1}F^{\mathcal{X}}$.

</details>

What about the sparsity of $F$?

- Want each row of $F$ to be sparse.

#### Sparsity of Short-Dot codes

Build $P\times P$ square matrix whose each row/column contains $P-K+M$ non-zero entries.

Concatenate $\frac{N}{P}$ such matrices and obtain 

[Missing slides 18]

We now investigate what 𝑍 should look like to construct such a matrix 𝐹.
• Recall that each column of 𝐹 must contains 𝐾 − 𝑀 zeros.
– They are indexed by the set 𝒰 ∈ 𝑃 , where |𝒰| = 𝐾 − 𝑀.
– Let 𝐵
𝒰 ∈ 𝔽
𝐾−𝑀 ×𝐾 be a submatrix of 𝐵 containing rows indexed by 𝒰.
• Since 𝐹 = 𝐵𝐴ሚ , it follows that 𝐹𝑗 = 𝐵𝐴ሚ
𝑗
, where 𝐹𝑗 and 𝐴ሚ
𝑗 are the 𝑗-th column of 𝐹 and 𝐴ሚ.
• Next, we have 𝐵
𝒰𝐴ሚ
𝑗 = 0 (𝐾−𝑀)×1.
• Split 𝐵
𝒰 = [𝐵 1,𝑀
𝒰
,𝐵[𝑀+1,𝐾]
𝒰 ], 𝐴ሚ
𝑗 = 𝐴𝑗
𝑇
, 𝑍𝑗
𝑇
𝑇
.
• 𝐵
𝒰𝐴ሚ
𝑗 = 𝐵 1,𝑀
𝒰 𝐴𝑗 + 𝐵[𝑀+1,𝐾]
𝒰 𝑍𝑗 = 0 (𝐾−𝑀)×1.
• 𝑍𝑗= −(𝐵 𝑀+1,𝐾
𝒰
)
−1 𝐵 1,𝑀
𝒰 𝐴𝑗
.
• Note that 𝐵 𝑀+1,𝐾
𝒰 ∈ 𝔽
𝐾−𝑀 × 𝐾−𝑀 is invertible.
– Since 𝐵 is super-regular.