268 lines
7.8 KiB
Markdown
268 lines
7.8 KiB
Markdown
# CSE5313 Coding and information theory for data science (Lecture 8)
|
||
|
||
## Review on Linear codes
|
||
|
||
|The code |Dimension $k$ (effective message length) | Minimum distance $d$ | Dual code | Minimum distance of dual code|
|
||
|---------|--------------|----------------------|-----------|-----------------------------|
|
||
|$\mathbb{F}^n$| $n$ | $1$ | $\{0\}$ | $0$|
|
||
|Parity code| $n-1$ | $2$ | Repetition code | $n$|
|
||
|Hamming code| $2^m-m-1$ | $3$ | Punctured Hadamard code | $2^{m-1}$|
|
||
|
||
## More on linear codes
|
||
|
||
### Extended Hamming code
|
||
|
||
Consider the Hamming code $[2^m-1,2^m-m-1,3]_{\mathbb{F}_2}$.
|
||
|
||
Extend it to a cod eof length $2^m$ by adding a parity bit.
|
||
|
||
Recall the hamming code $[7,4,3]_{2}$.
|
||
|
||
$$
|
||
H_{HAM}=
|
||
\begin{pmatrix}
|
||
1 & 0 & 0 & 1 & 1 & 0 & 1\\
|
||
0 & 1 & 0 & 1 & 0 & 1 & 1\\
|
||
0 & 0 & 1 & 0 & 1 & 1 & 1\\
|
||
\end{pmatrix}
|
||
$$
|
||
|
||
The extended Hamming code is:
|
||
|
||
$$
|
||
H_{EXT}=
|
||
\begin{pmatrix}
|
||
1 & 1 & 1 & 1 & 1 & 1 & 1 & 1\\
|
||
1 & 0 & 0 & 1 & 1 & 0 & 1 & 0\\
|
||
0 & 1 & 0 & 1 & 0 & 1 & 1 & 0\\
|
||
0 & 0 & 1 & 0 & 1 & 1 & 1 & 0\\
|
||
\end{pmatrix}
|
||
$$
|
||
|
||
The minimum distance of the extended Hamming code is 4.
|
||
|
||
<details>
|
||
<summary>Proof for minimum distance</summary>
|
||
|
||
It is sufficient to show that every 3 columns are linearly independent.
|
||
|
||
Using the [lemma for minimum distance](./CSE5313_L7#lemma-for-minimum-distance), we have that the minimum distance is 4.
|
||
|
||
Notice that in $\mathbb{F}_2$, multiplication is equivalent to AND operation.
|
||
|
||
By simple observations, we know that every 2 columns are linearly independent.
|
||
|
||
Consider the following linear combination:
|
||
|
||
$$
|
||
a_1\begin{pmatrix}
|
||
1\\
|
||
\vdots\\
|
||
\end{pmatrix}+a_2\begin{pmatrix}
|
||
1\\
|
||
\vdots\\
|
||
\end{pmatrix}+a_3\begin{pmatrix}
|
||
1\\
|
||
\vdots\\
|
||
\end{pmatrix}=0
|
||
$$
|
||
|
||
Therefore, every 3 columns are linearly independent since the top row will always be 1. if $a_1,a_2,a_3$ are $1$.
|
||
|
||
Therefore, the minimum distance is 4.
|
||
|
||
</details>
|
||
|
||
For the dimension of this code, we have $k=2^m-m-1$, with total code length $m$.
|
||
|
||
### Augmented Hadamard code
|
||
|
||
Consider what is generated by the parity check matrix of the extended Hamming code.
|
||
|
||
Let $xH_{EXT}$
|
||
|
||
- If $xH_{EXT}=0$, then $(0,x_2,\ldots, x_m)H_{HAM}=(y,0)$, where $y$ is the punctured hadamard codeword.
|
||
- If $xH_{EXT}=1$, then $(1,x_2,\ldots, x_m)H_{HAM}=(1,1,\ldots,1)+(y,0)$, where $y$ is the punctured hadamard codeword.
|
||
|
||
Therefore, the code generated by $H_{EXT}$ is given by:
|
||
|
||
Let $\mathcal{C}$ be the hadamard code.
|
||
|
||
Let $\mathcal{C}+\mathbb{I}$ be its shift by the all 1's vector (flip all bits of all words).
|
||
|
||
– Augmented Hadamard code = $\mathcal{C} \cup \mathcal{C} + \mathbb{I}$.
|
||
|
||
The length of the code is $2^m$.
|
||
|
||
The dimension of the code is $m+1$.
|
||
|
||
Since the code is still linear, the minimum distance is the minimum weight of the codewords.
|
||
|
||
So minimum distance of $\mathcal{C}+\mathbb{I}$ is $2^{m-1}$.
|
||
|
||
### Summary for simple linear codes
|
||
|
||
|The code |Dimension $k$ (effective message length) | Minimum distance $d$ | Dual code | Minimum distance of dual code|
|
||
|---------|--------------|----------------------|-----------|-----------------------------|
|
||
|$\mathbb{F}^n$| $n$ | $1$ | $\{0\}$ | $0$|
|
||
|Parity code| $n-1$ | $2$ | Repetition code | $n$|
|
||
|Hamming code| $2^m-m-1$ (length $2^m-1$) | $3$ | Punctured Hadamard code | $2^{m-1}$|
|
||
|Extended Hamming code| $2^m-m-1$ (length $2^m$) | $4$ | Augmented Hadamard code | $2^{m-1}$|
|
||
|
||
## Boundary of linear codes
|
||
|
||
Natural questions:
|
||
|
||
- Can we extend the table of linear codes infinitely?
|
||
- What set of configuration $(n,k,d)_q$ are impossible?
|
||
- What set of configuration $(n,k,d)_q$ are possible, even if we don't know how to construct them?
|
||
|
||
### Boundary I: Singleton bound
|
||
|
||
> Singleton is a name for the person who discovered this bound.
|
||
|
||
Theorem: For any linear code $\mathcal{C}\subseteq \mathbb{F}^n_q$, we have $d\leq n-k+1$.
|
||
|
||
<details>
|
||
<summary>Proof</summary>
|
||
|
||
Idea: Using the Pigeonhole principle.
|
||
|
||
Assume an code $[n,k,d]_q$ exists.
|
||
|
||
Pigeons: All $q^k$ possible code word of $\mathcal{C}$.
|
||
|
||
Holes: All $q^\ell$ values of the first $\ell$ entries of a codeword (for some $\ell<n$).
|
||
|
||
If $q^\ell<q^k$, then by Pigeonhole principle, there exists two codewords in $\mathcal{C}$ that agrees on the first $\ell$ entries.
|
||
|
||
So $d\leq n-\ell$.
|
||
|
||
So the largest $\ell$ value for which this arguments works is $n-k+1$.
|
||
|
||
</details>
|
||
|
||
#### Definition of Maximum Distance Separable (MDS) code
|
||
|
||
A code $\mathcal{C} = [n,k,d]_q$ with $d = n - k + 1$ is called a Maximum Distance Separable (MDS) code.
|
||
|
||
<details>
|
||
<summary>Examples for singleton bound</summary>
|
||
|
||
$\mathbb{F}^n$: any $n, k = n, d = 1$.
|
||
|
||
- Attains with equality!
|
||
|
||
Parity: any $n, k = n - 1, d = 2$.
|
||
|
||
- Attains with equality!
|
||
|
||
Hamming: $n = 2^m - 1, k = 2^m - m - 1, d = 3$.
|
||
|
||
$n - k + 1 = m + 1 > 3$.
|
||
|
||
This creates some trade-off between $k$ and $d$.
|
||
|
||
</details>
|
||
|
||
### Boundary II: The Sphere Packing Bound
|
||
|
||
Let $r=\lfloor \frac{d-1}{2}\rfloor$, then $\sum_{i=0}^{r}\binom{n}{i}(q-1)^i\leq q^{n-k}$.
|
||
|
||
<details>
|
||
<summary>Proof</summary>
|
||
|
||
Let $c=(c_1,c_2,\ldots,c_n)\in \mathbb{F}^n_q$, and let $B(c,r)=\{y\in \mathbb{F}^n_q: d_H(c,y)\leq r\}$ for some $r\leq n$.
|
||
|
||
Computer $|B(c,r)|$.
|
||
|
||
$|B(c,0)|=1$
|
||
|
||
$|\{y\in \mathbb{F}^n_q: d_H(c,y)=1\}|=n(q-1)$.
|
||
|
||
$|\{y\in \mathbb{F}^n_q: d_H(c,y)=2\}|=\binom{n}{2}(q-1)^2=\frac{n(n-1)}{2}(q-1)^2$.
|
||
|
||
So, $|B(c,r)|=\sum_{i=0}^{r}\binom{n}{i}(q-1)^i$.
|
||
|
||
Recall that $\mathcal{C}$ of minimum distance $d$ if and only if $\forall c_1,c_2\in \mathcal{C}, B(c_1,\lfloor \frac{d-1}{2}\rfloor)\cap B(c_2,\lfloor \frac{d-1}{2}\rfloor)=\emptyset$.
|
||
|
||
Therefore, let $r=\lfloor \frac{d-1}{2}\rfloor$, we have $\sum_{i=0}^{r}\binom{n}{i}(q-1)^i\leq q^{n-k}$.
|
||
|
||
</details>
|
||
|
||
#### Definition for perfect code
|
||
|
||
A code $\mathcal{C}$ is called a perfect code if $|C|=q^{n-k}$.
|
||
|
||
<details>
|
||
<summary>Examples for sphere packing bound</summary>
|
||
|
||
Let $q=2$.
|
||
|
||
$\mathbb{F}_2^n$: any $n, k = n, d = 1$.
|
||
|
||
- $r = \frac{d-1}{2} = 0$.
|
||
- $\Rightarrow \sum_{i=0}^{0}\binom{n}{i}(q-1)^i = 1 \leq q^{n-k} = 2^{n-n} = 1$.
|
||
- Attained with equality!
|
||
|
||
Parity: any $n, k = n - 1, d = 2$.
|
||
|
||
- $r = \frac{d-1}{2} = 0$.
|
||
|
||
- $\Rightarrow \sum_{i=0}^{0}\binom{n}{i}(q-1)^i = 1 \leq q^{n-k} = 2^{n-k} = 2^{n-(n-1)} = 2$.
|
||
|
||
- $q \geq 2 \Rightarrow$ NOT attained with equality.
|
||
|
||
Exercise: Equality is attained for the repetition code (dual of parity) for odd $n$.
|
||
|
||
Hamming: $n = 2^m - 1, k = 2^m - m - 1, d = 3$.
|
||
|
||
- $r = \frac{d-1}{2} = 1$.
|
||
- $\Rightarrow \sum_{i=0}^{1}\binom{n}{i}(q-1)^i = 1 + (2^{m}-1) = 2^{m}$.
|
||
- $\Rightarrow q^{n-k} = 2^{m}$.
|
||
- Attained with equality!
|
||
|
||
• Attained with equality!
|
||
|
||
</details>
|
||
|
||
Fortunately, there are only **4** types of **binary linear perfect codes**:
|
||
|
||
- $\mathbb{F}^n$
|
||
- Repetition code
|
||
- Hamming code
|
||
- $[23,12,7]_2$ Golay code
|
||
|
||
### Boundary III: The Gilbert-Varshamov Bound
|
||
|
||
Let $n,k,d,q$ such that $V_q(n-k, d-2)\leq q^{n-k}$, then there exists an $[n,k,d]_q$ code.
|
||
|
||
> Singleton, sphere-packing provide **necessary** conditions for existence of codes.
|
||
>
|
||
> Are there **sufficient** conditions?
|
||
>
|
||
> Recall:
|
||
>
|
||
> - Lemma: The minimum distance of $\mathcal{C}$ is the maximum integer such that every $d-1$ columns of the parity-check matrix $H$ are linearly independent.
|
||
>
|
||
> Idea:
|
||
>
|
||
> - Construct $H$ column by column, ensuring that no dependencies occur.
|
||
|
||
Idea:
|
||
|
||
Construct $H$ column by column, ensuring that no dependencies occur.
|
||
|
||
Algorithm:
|
||
|
||
- Begin with $(n-k)\times (n-k)$ identity matrix.
|
||
- Assume we choose columns $h_1,h_2,\ldots,h_\ell$ (each $h_i$ is in $\mathbb{F}^n_q$)
|
||
- Then next column $h_{\ell}$ must not be in the space of any previous $d-2$ columns.
|
||
- $h_{\ell}$ cannot be written as $[h_1,h_2,\ldots,h_{\ell-1}]x^\top$ for $x$ of Hamming weight at most $d-2$.
|
||
- So the ineligible candidates for $h_{\ell}$ is:
|
||
- $B_{\ell-1}(0,d-2)=\{x\in \mathbb{F}^{\ell-1}_q: d_H(0,x)\leq d-2\}$.
|
||
- $|B_{\ell-1}(0,d-2)|=\sum_{i=0}^{d-2}\binom{\ell-1}{i}(q-1)^i$, denoted by $V_q(\ell-1, d-2)$.
|
||
- So the candidates for $h_{\ell}$ is:
|
||
- Out of which $V_q(\ell-1, d-2)$ are ineligible.
|
||
- Need $n$ columns overall, so we need $V_q(n-k, d-2)\leq q^{n-k}$.
|