247 lines
6.5 KiB
Markdown
247 lines
6.5 KiB
Markdown
# CSE5313 Coding and information theory for data science (Lecture 6)
|
||
|
||
## Recap
|
||
|
||
### Vector spaces and subspaces over finite fields
|
||
|
||
$\mathbb{F}^n$ is a vector space over $\mathbb{F}$.
|
||
|
||
With point-wise vector addition and scalar multiplication.
|
||
|
||
<details>
|
||
<summary>Example</summary>
|
||
|
||
$\mathbb{F}_2^4$ is a vector space over $\mathbb{F}_2$.
|
||
|
||
Let $v=\begin{pmatrix}
|
||
1 & 1 & 1 & 1
|
||
\end{pmatrix}$
|
||
|
||
Then $v$ is a vector in $\mathbb{F}_2^4$ that's "orthogonal" to itself.
|
||
|
||
$v\cdot v=1+1+1+1=4=0$ in $\mathbb{F}_2$.
|
||
|
||
In general field, the dual space and space may intersect non-trivially.
|
||
|
||
</details>
|
||
|
||
Let $V$ be a subspace of $\mathbb{F}^n$.
|
||
|
||
$V$ is a subgroup of $\mathbb{F}^n$ under vector addition $(\mathbb{F}^n,+)$.
|
||
|
||
- Apply the theorem: If $H$ is finite, non-empty, and closed under the operation of $G$, then $H$ is a subgroup of $G$.
|
||
|
||
<details>
|
||
<summary>Proof</summary>
|
||
Since $H\subseteq G$, $H$ is non-empty and closed under the operation of $G$ and finite, then $H\leq G$.
|
||
|
||
left to show:
|
||
|
||
Associativity: inherited from $G$.
|
||
|
||
Unit element: $0\in H$.
|
||
|
||
Consider $a\in H$, $a,a^2,a^3,\cdots$ are in $H$. Since $H$ is finite, there exists $i,j\in\mathbb{N}$ such that $a^i=a^j$.
|
||
|
||
Then $a^i=a^j\iff a^{i-j}=e\in H$.
|
||
|
||
Inverses: $a^{-1}\in H$.
|
||
|
||
Automatically holds for unit element traversing.
|
||
|
||
</details>
|
||
|
||
> Is every subgroup of $\mathbb{F}^n$ a subspace?
|
||
|
||
<details>
|
||
<summary>Answer</summary>
|
||
No.
|
||
|
||
Consider $F_4=\{0,1,x,x+1\}$ (field extension of $\mathbb{F}_2$ with $p(x)=x$).
|
||
|
||
$F_4^2=\{(a,b):a,b\in F_4\}$, $\{(0,0),(1,1)\}$ is a subgroup of $(F_4^2,+)$.
|
||
|
||
But the span of $F_4\{(1,1)\}$ is $\{(0,0),(1,1),(x,x),(x+1,x+1)\}\neq \{(0,0),(1,1)\}$, which is not a subspace of $F_4^2$.
|
||
|
||
</details>
|
||
|
||
Cosets in this definition are called Affine subspaces.
|
||
|
||
$$
|
||
V+a=\{v+a:v\in V\}\text{ for some }a\in \mathbb{F}^n
|
||
$$
|
||
|
||
## New content
|
||
|
||
### Linear codes
|
||
|
||
A linear code $\mathcal{C}$ is a subspace of $\mathbb{F}^n$ over $\mathbb{F}$.
|
||
|
||
- The dimension of $\mathcal{C}$ is denoted by $k$.
|
||
- The minimum Hamming distance of $\mathcal{C}$ is denoted by $d$.
|
||
- Notation $\mathcal{C}= [n,k,d]_{\mathbb{F}}$.
|
||
|
||
Two equivalent ways to constructing a linear code:
|
||
|
||
- A **generator matrix** $G\in \mathbb{F}^{k\times n}$ with $k$ rows and $n$ columns.
|
||
$$
|
||
\mathcal{C}=\{xG:x\in \mathbb{F}^k\}
|
||
$$
|
||
- The left image of $G$ is $\mathcal{C}$.
|
||
- The rows of $G$ are a basis for $\mathcal{C}$.
|
||
|
||
- A **parity check** matrix $H\in \mathbb{F}^{(n-k)\times n}$ with $(n-k)$ rows and $n$ columns.
|
||
$$
|
||
\mathcal{C}=\{c\in \mathbb{F}^n:Hc^T=0\}
|
||
$$
|
||
- The right kernel of $H$ is $\mathcal{C}$.
|
||
- Multiplying $c^T$ by $H$ "checks" if $c\in \mathcal{C}$.
|
||
|
||
### Encoding of linear codes
|
||
|
||
Reminder:
|
||
|
||
- Encoding is the process of mapping a message $u\in \mathbb{F}^k$ to a codeword $c\in \mathcal{C}\subseteq \mathbb{F}^n$.
|
||
|
||
$E: \mathbb{F}^k\to \mathcal{C}$ is a linear map.
|
||
|
||
Let $\mathcal{C}= [n,k,d]_{\mathbb{F}}$ be a linear code with generator matrix $G\in \mathbb{F}^{k\times n}$.
|
||
|
||
- Encoding is given by $E(x)=xG$.
|
||
- It is injective (1-1). Suppose otherwise, then there exists $x_1,x_2\in \mathbb{F}^k$ such that $x_1G=x_2G$. Then $x_1G-x_2G=0\implies (x_1-x_2)G=0\implies x_1-x_2=0\implies x_1=x_2$. Therefore, $E$ is injective.
|
||
|
||
So linear codes implies linear encoding: $E(x)+E(y)=E(x+y)$.
|
||
|
||
### Systematic codes
|
||
|
||
Fact: Every $G\in \mathbb{F}^{k\times n}$ can be brought to the form $G_{sys}=(I|A)$ by
|
||
|
||
- Row operations.
|
||
- Permutation of columns.
|
||
|
||
Fact $\{xG|x\in \mathbb{F}^k\}$ and $\{xG_{sys}|x\in \mathbb{F}^k\}$ are equivalent.
|
||
|
||
- Same length $n$.
|
||
- Same dimension $k$.
|
||
- Same minimum Hamming distance $d$.
|
||
|
||
Encoding a systematic code:
|
||
|
||
- The input is a part of the output.
|
||
- Efficient encoding
|
||
- Immediate decoding (if no errors).
|
||
|
||
### Codes, cosets, encoding, decoding
|
||
|
||
Linear code $[n,k,d]_{\mathbb{F}}$ is a $k$ dimensional subspace of $\mathbb{F}^n$.
|
||
|
||
Size of the code is $|\mathbb{F}|^k$.
|
||
|
||
Encoding: $x\to xG$.
|
||
|
||
Decoding: $(y+e)\to x$, $y=xG$.
|
||
|
||
Use **syndrome** to identify which coset $\mathcal{C}_i$ that the noisy-code to $\mathcal{C}_i+e$ belongs to.
|
||
|
||
$$
|
||
H(y+e)^T=H(y+e)=Hx+He=He
|
||
$$
|
||
|
||
### Syndrome decoding
|
||
|
||
- Heavily depends onn the linear structure of the code.
|
||
|
||
Linear code $\mathcal{C}= [n,k,d]_{\mathbb{F}}$ is a $k$-dimensional subspace of $(\mathbb{F}^n,+)$.
|
||
|
||
**Shift of** Linear code $[n,k,d]_{\mathbb{F}}$ is a $k$-dimensional affine subspace of $\mathbb{F}^n$.
|
||
|
||
All cosets of the same size
|
||
|
||
If $w_H(e)\leq \lfloor \frac{d-1}{2}\rfloor$, then it is possible to extract $y$ from $y+e$.
|
||
|
||
by syndrome decoding, we can do better than exhaustive search.
|
||
|
||
Idea:
|
||
|
||
Let $y+e$ belogns to the coset $\mathcal{C}+e$.
|
||
|
||
Moreover,$y_1+e$ and $y_2+e$ are in the same coset.
|
||
|
||
#### Standard Array
|
||
|
||
Let $\mathcal{C}= [n,k,d]_{\mathbb{F}}$ and denote $|F|=q$.
|
||
|
||
- Then $|\mathcal{C}|=q^k$.
|
||
- The number of cosets is $q^{n-k}$.
|
||
|
||
Then we arrange all $q^n$ elements of $\mathbb{F}^n$ into a $q^{n-k}\times q^k$ array.
|
||
|
||
- So that every row is a coset (including $\mathcal{C}$ itself)
|
||
- Lightest word in each cosets on the leftmost column
|
||
|
||
<details>
|
||
<summary>Example</summary>
|
||
|
||
Let $\mathbb{F}=\mathbb{Z}_2$ and $C=\{xG|x\in \mathbb{F}_2\}$
|
||
|
||
$$
|
||
G=\begin{pmatrix}
|
||
1 & 0 & 1 & 1 & 0\\
|
||
0 & 1 & 1 & 0 & 1
|
||
\end{pmatrix}
|
||
$$
|
||
|
||
So $\mathcal{C}=\{00000,10110,01011,11101\}$.
|
||
|
||
Then $G=[5,2,3]_2$.
|
||
|
||
The standard array is:
|
||
|
||
First row is $\mathcal{C}$.
|
||
|
||
Second row is $\mathcal{C}+(00001)$,
|
||
|
||
Third row is $\mathcal{C}+(00010)$.
|
||
|
||
Fourth row is $\mathcal{C}+(00100)$.
|
||
|
||
|00000|10110|01011|11101|
|
||
|---|---|---|---|
|
||
|00001|10111|01010|11100|
|
||
|00010|10100|01001|11110|
|
||
|00100|10010|01101|11000|
|
||
|
||
</details>
|
||
|
||
Any two elements in a row are of the form $y_1'=y_1+e$ and $y_2'=y_2+e$ for some $e\in \mathbb{F}^n$.
|
||
|
||
Same syndrome if $H(y_1'+e)^T=H(y_2'+e)^T$.
|
||
|
||
Entries in different rows have different syndrome.
|
||
|
||
<details>
|
||
<summary>Proof</summary>
|
||
|
||
|
||
</details>
|
||
|
||
Choose the lightest word in each coset on the leftmost column.
|
||
|
||
Time complexity: $O(n(n-k))$. Space complexity: $n|F|^n$ space.
|
||
|
||
Compare with exhaustive search: Time: $O(|F|^n)$.
|
||
|
||
#### Syndrome decoding - Intuition
|
||
|
||
Given 𝒚′, we identify the set 𝐶 + 𝒆 to which 𝒚′ belongs by computing the syndrome.
|
||
• We identify 𝒆 as the coset leader (leftmost entry) of the row 𝐶 + 𝒆.
|
||
• We output the codeword in 𝐶 which is closest (𝒄3) by subtracting 𝒆 from 𝒚′.
|
||
|
||
#### Syndrome decoding - Formal
|
||
|
||
Given $y'\in \mathbb{F}^n$, we identify the set $\mathcal{C}+e$ to which $y'$ belongs by computing the syndrome.
|
||
|
||
We identify $e$ as the coset leader (leftmost entry) of the row $\mathcal{C}+e$.
|
||
|
||
We output the codeword in $\mathcal{C}$ which is closest ($c_3$) by subtracting $e$ from $y'$.
|