367 lines
9.5 KiB
Markdown
367 lines
9.5 KiB
Markdown
# CSE5313 Coding and information theory for data science (Lecture 2)
|
|
|
|
## Review on Channel coding
|
|
|
|
Let $F$ be the input alphabet, $\Phi$ be the output alphabet.
|
|
|
|
e.g. $F=\{0,1\},\mathbb{R}$.
|
|
|
|
Introduce noise: $\operatorname{Pr}(c'\text{ received}|c\text{ transmitted})$.
|
|
|
|
We use $u$ to denote the information to be transmitted
|
|
|
|
$c$ to be the codeword.
|
|
|
|
$c'$ is the received codeword. given to the decoder.
|
|
|
|
$u'$ is the decoded information word.
|
|
|
|
Error if $u' \neq u$.
|
|
|
|
Example:
|
|
|
|
**Binary symmetric channel (BSC)**
|
|
|
|
$F=\Phi=\{0,1\}$
|
|
|
|
Every bit of $c$ is flipped with probability $p$.
|
|
|
|
**Binary erasure channel (BEC)**
|
|
|
|
$F=\Phi=\{0,1,*\}$, very common in practice when we are unsure when the bit is transmitted.
|
|
|
|
$c$ is transmitted, $c'$ is received.
|
|
|
|
$c'$ is $c$ with probability $1-p$, $e$ with probability $p$.
|
|
|
|
## Encoding
|
|
|
|
Encoding $E$ is a function from $F^k$ to $F^n$.
|
|
|
|
Where $E(u)=c$ is the codeword.
|
|
|
|
Assume $n\geq k$, we don't compress the information.
|
|
|
|
A code $\mathcal{C}$ is a subset of $F^n$.
|
|
|
|
Encoding is a one to one mapping from $F^k$ to $\mathcal{C}$.
|
|
|
|
In practice, we usually choose $\mathcal{C}\subseteq F^n$ to be the size of $F^k$.
|
|
|
|
## Decoding
|
|
|
|
$D$ is a function from $\Phi^n$ to $\mathcal{C}$.
|
|
|
|
$D(c')=\hat{c}$
|
|
|
|
The decoder then outputs the unique $u'$ such that $E(u')=\hat{c}$.
|
|
|
|
Our aim is to have $u=u'$.
|
|
|
|
Decoding error probability: $\operatorname{P}_{err}=\max_{c\in \mathcal{C}}\operatorname{P}_{err}(c)$.
|
|
|
|
where $\operatorname{P}_{err}(c)=\sum_{y|D(y)\neq c}\operatorname{Pr}(y\text{ received}|c\text{ transmitted})$.
|
|
|
|
Our goal is to construct decoder $D$ such that $\operatorname{P}_{err}$ is bounded.
|
|
|
|
Example:
|
|
|
|
Repetition code in binary symmetric channel:
|
|
|
|
Let $F=\Phi=\{0,1\}$. Every bit of $c$ is flipped with probability $p$.
|
|
|
|
Say $k=1$, $n=3$ and let $\mathcal{C}=\{000,111\}$.
|
|
|
|
Let the encoder be $E(u)=u u u$.
|
|
|
|
The decoder is $D(000)=D(100)=D(010)=D(001)=0$, $D(110)=D(101)=D(011)=D(111)=1$.
|
|
|
|
Exercise: Compute the error probability of the repetition code in binary symmetric channel.
|
|
|
|
<details>
|
|
<summary>Solution</summary>
|
|
|
|
Recall that $P_{err}(c)=\sum_{y|D(y)\neq c}\operatorname{Pr}(y\text{ received}|c\text{ transmitted})$.
|
|
|
|
Use binomial random variable:
|
|
|
|
$$
|
|
\begin{aligned}
|
|
P_{err}(000)&=\sum_{y|D(y)\neq 000}\operatorname{Pr}(y\text{ received}|000\text{ transmitted})\\
|
|
&=\operatorname{Pr}(2\text{ flipes or more})\\
|
|
&=\binom{n}{2}p^2(1-p)+\binom{n}{3}p^3\\
|
|
&=3p^2(1-p)+p^3\\
|
|
\end{aligned}
|
|
$$
|
|
|
|
The computation is identical for $111$.
|
|
|
|
$P_{err}=\max\{P_{err}(000),P_{err}(111)\}=P_{err}(000)=3p^2(1-p)+p^3$.
|
|
|
|
</details>
|
|
|
|
### Maximum likelihood principle
|
|
|
|
For $p\leq 1/2$, the last example is maximum likelihood decoder.
|
|
|
|
Notice that $\operatorname{Pr}(c'=000|c=000)=(1-p)^3$ and $\operatorname{Pr}(c'=000|c=111)=p^3$.
|
|
|
|
- If $p\leq 1/2$, then $(1-p)^3\geq p^3$. $c=000$ is more likely to be transmitted than $c=111$.
|
|
|
|
When $\operatorname{Pr}(c'=001|c=000)=(1-p)^2p$ and $\operatorname{Pr}(c'=001|c=111)=p^2(1-p)$.
|
|
|
|
- If $p\leq 1/2$, then $(1-p)^2p\geq p^2(1-p)$. $c=001$ is more likely to be transmitted than $c=110$.
|
|
|
|
For $p>1/2$, we just negate the above.
|
|
|
|
In general, Maximum likelihood decoder is $D(c')=\arg\max_{c\in \mathcal{C}}\operatorname{Pr}(c'\text{ received}|c\text{ transmitted})$.
|
|
|
|
## Defining a "good" code
|
|
|
|
Two metrics:
|
|
|
|
- How many redundant bits are needed?
|
|
- e.g. repetition code: $k=1$, $n=3$ sends $2$ redundant bits.
|
|
- What is the resulting error probability?
|
|
- Depends on the decoding function.
|
|
- Normally, maximum likelihood decoding is assumed.
|
|
- Should go zero with $n$.
|
|
|
|
### Definition for rate of code is $\frac{k}{n}$.
|
|
|
|
More generally, $\log_{|F|}\frac{|\mathcal{C}|}{n}$.
|
|
|
|
### Definition for information entropy
|
|
|
|
Let $X$ be a random variable over a discrete set $\mathcal{X}$.
|
|
|
|
- That is every $x\in \mathcal{X}$ has a probability $\operatorname{Pr}(X=x)$.
|
|
|
|
The entropy $H(X)$ of a discrete random variable $X$ is defined as:
|
|
|
|
$$
|
|
H(X)=\mathbb{E}_{x\sim X}{\log \frac{1}{\operatorname{Pr}(x)}}=-\sum_{x\in \mathcal{X}}\operatorname{Pr}(x)\log \operatorname{Pr}(x)
|
|
$$
|
|
|
|
when $X=Bernouili(p)$, we denote $H(X)=H(p)=-p\log p-(1-p)\log (1-p)$.
|
|
|
|
A deeper explanation will be given in the later in the course.
|
|
|
|
## Which rate are possible?
|
|
|
|
Claude Shannon '48: Coding theorem of the BSC(binary symmetric channel)
|
|
|
|
Recall $r=\frac{k}{n}$.
|
|
|
|
Let $H(\cdot)$ be the entropy function.
|
|
|
|
For every $0\leq r<1-H(p)$,
|
|
|
|
- There exists $\mathcal{C}_1, \mathcal{C}_2,\ldots$ of rates $r_1,r_2,\ldots$ lengths $n_1,n_2,\ldots$ and $r_i\geq r$.
|
|
- That with Maximum likelihood decoding satisifies $P_{err}\to 0$ as $i\to \infty$.
|
|
|
|
For any $R\geq 1-H(p)$,
|
|
|
|
- Any sequence $\mathcal{C}_1, \mathcal{C}_2,\ldots$ of rates $r_1,r_2,\ldots$ lengths $n_1,n_2,\ldots$ and $r_i\geq R$,
|
|
- Any andy decoding algorithm, $P_{err}\to 1$ as $i\to \infty$.
|
|
|
|
$1-H(p)$ is the capacity of the BSC.
|
|
|
|
- Informally, the capacity is the best possible rate of the code (asymptotically).
|
|
- A special case of a broader theorem (Shannon's coding theorem).
|
|
- We will see later in this course.
|
|
|
|
Polar codes, for explicit construction of codes with rate arbitrarily close to capacity.
|
|
|
|
### BSC capacity - Intuition
|
|
|
|
Capacity of the binary symmetric channel with crossover probability $p=1-H(p)$.
|
|
|
|
A correct decoder $c'\to c$ essentially identifies two objects:
|
|
|
|
- The codeword $c$
|
|
- The error word $e=c'-c$ subtraction $\mod 2$.
|
|
- $c$ and $e$ are independent of each other.
|
|
|
|
A **typical** $e$ has $\approx np$ $1$'s (law of large numbers), say $n(p\pm \delta)$.
|
|
|
|
Exercise:
|
|
|
|
$\operatorname{Pr}(e)=p^{n(p\pm \delta)}(1-p)^{n(1-p\pm \delta)}=2^{-n(H(p)+\epsilon)}$ for some $\epsilon$ goes to zero as $n\to \infty$.
|
|
|
|
<details>
|
|
<summary>Intuition</summary>
|
|
|
|
There exists $\approx 2^{n(H(p)}$ typical error words.
|
|
|
|
To index those typical error words, we need $\log_2 (2^{nH(p)})=nH(p)+O(1)$. bits to identify the error word $e$.
|
|
|
|
To encode the message, we need $\log_2 |\mathcal{C}|$ bits.
|
|
|
|
Since we send $n$ bits, the rate is $k+nH(p)+O(1)\leq n$, so $\frac{k}{n}\leq 1-H(p)$.
|
|
|
|
So the rate cannot exceed $1-H(p)$.
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>Formal proof</summary>
|
|
|
|
$$
|
|
\begin{aligned}
|
|
\operatorname{Pr}(e)&=p^{n(p\pm \delta)}(1-p)^{n(1-p\pm \delta)}\\
|
|
&=p^{np}(1-p)^{n(1-p)}p^{\pm n\delta}(1-p)^{\mp n\delta}\\
|
|
\end{aligned}
|
|
$$
|
|
|
|
And
|
|
|
|
$$
|
|
\begin{aligned}
|
|
2^{-n(H(p)+\epsilon)}&=2^{-n(-p\log p-(1-p)\log (1-p)+\epsilon)}\\
|
|
&=2^{np\log p}2^{n(1-p)\log (1-p)}2^{-n\epsilon}\\
|
|
&=p^{np}(1-p)^{n(1-p)}2^{-n\epsilon}\\
|
|
\end{aligned}
|
|
$$
|
|
|
|
So we need to check there exists $\epsilon>0$ such that
|
|
|
|
$$
|
|
\lim_{n\to \infty}p^{\pm n\delta}(1-p)^{\mp n\delta}\leq 2^{-n\epsilon}
|
|
$$
|
|
|
|
Test
|
|
|
|
$$
|
|
\begin{aligned}
|
|
2^{-n\epsilon}&=p^{np}(1-p)^{n(1-p)}2^{-n\epsilon}\\
|
|
-n\epsilon&=\delta n\log p-\delta n\log (1-p)\\
|
|
\epsilon&=\delta (\log (1-p)-\log p)\\
|
|
\end{aligned}
|
|
$$
|
|
|
|
|
|
</details>
|
|
|
|
## Hamming distance
|
|
|
|
How to quantify the noise in the channel?
|
|
|
|
- Number of flipped bits.
|
|
|
|
Definition of Hamming distance:
|
|
|
|
- Denote $c=(c_1,c_2,\ldots,c_n)$ and $c'=(c'_1,c'_2,\ldots,c'_n)$.
|
|
- $d_H(c,c')=\sum_{i=1}^n [c_i\neq c'_i]$.
|
|
|
|
Minimum hamminng distance:
|
|
|
|
- Let $\mathcal{C}$ be a code.
|
|
- $d_H(\mathcal{C})=\min_{c_1,c_2\in \mathcal{C},c_1\neq c_2}d_H(c_1,c_2)$.
|
|
|
|
Hamming distance is a metric.
|
|
|
|
- $d_H(x,y)\geq 0$ equal iff $x=y$.
|
|
- $d_H(x,y)=d_H(y,x)$
|
|
- Triangle inequality: $d_H(x,y)\leq d_H(x,z)+d_H(z,y)$
|
|
|
|
### Level of error handling
|
|
|
|
- error detection
|
|
- erasure correction
|
|
- error correction
|
|
|
|
Erasure: replacement of an entry by $*\not\in F$.
|
|
|
|
Error: substitution of one entry by a different one.
|
|
|
|
Example: If $d_H(\mathcal{C})=d$.
|
|
|
|
#### Error detection
|
|
|
|
Theorem: If $d_H(\mathcal{C})=d$, then there exists $f:F^n\to \mathcal{C}\cap \{\text{"error detected"}\}$. that detects every patter of $\leq d-1$ errors correctly.
|
|
|
|
- That is, we can identify if the channel introduced at most $d-1$ errors.
|
|
- No decoding is needed.
|
|
|
|
Idea:
|
|
|
|
Since $d_H(\mathcal{C})=d$, one needs $\geq d$ errors to cause "confusion".
|
|
|
|
<details>
|
|
<summary>Proof</summary>
|
|
|
|
The function
|
|
$$
|
|
f(y)=\begin{cases}
|
|
y\text{ if }y\in \mathcal{C}\\
|
|
\text{"error detected"} & \text{otherwise}
|
|
\end{cases}
|
|
$$
|
|
|
|
will only fails if there are $\geq d$ errors.
|
|
|
|
</details>
|
|
|
|
#### Erasure correction
|
|
|
|
Theorem: If $d_H(\mathcal{C})=d$, then there exists $f:\{F^n\cup \{*\}\}\to \mathcal{C}\cap \{\text{"failed"}\}$. that recovers every patter of at most $d-1$ erasures.
|
|
|
|
Idea:
|
|
|
|
Suppose $d=4$.
|
|
|
|
If $4$ erasures occurred, there might be two possible codewords $c,c'\in \mathcal{C}$.
|
|
|
|
If $\leq 3$ erasures occurred, there is only one possible codeword $c\in \mathcal{C}$.
|
|
|
|
#### Error correction
|
|
|
|
Define the Hamming ball of radius $r$ centered at $c$ as:
|
|
|
|
$$
|
|
B_H(c,r)=\{y\in F^n:d_H(c,y)\leq r\}
|
|
$$
|
|
|
|
Theorem: If $d_H(\mathcal{C})\geq d$, then there exists $f:F^n\to \mathcal{C}$ that corrects every pattern of at most $\lfloor \frac{d-1}{2}\rfloor$ errors.
|
|
|
|
Ideas:
|
|
|
|
The ball $\{B_H(c,\lfloor \frac{d-1}{2}\rfloor)|c\in \mathcal{C}\}$ are disjoint.
|
|
|
|
Use closest neighbor decoding, use triangle inequality.
|
|
|
|
## Intro to linear codes
|
|
|
|
Summary: a code of minimum hamming distance $d$ can
|
|
|
|
- detect $\leq d-1$ errors.
|
|
- correct $\leq d-1$ erasures.
|
|
- Correct $\leq \lfloor \frac{d-1}{2}\rfloor$ errors.
|
|
|
|
Problems:
|
|
|
|
- How to construct good codes, $k/n$ and $d$ large?
|
|
- How good can these codes possibly be?
|
|
- How to encode?
|
|
- How to decode with noisy channel
|
|
|
|
Tools
|
|
|
|
- Linear algebra over finite fields.
|
|
|
|
### Linear codes
|
|
|
|
Consider $F^n$ as a vector space, and let $\mathcal{C}\subseteq F^n$ be a subspace.
|
|
|
|
$F,\Phi$ are finites, we use finite fields (algebraic objects that "immitate" $\mathbb{R}^n$, $\mathbb{C}^n$).
|
|
|
|
Formally, satisfy the field axioms.
|
|
|
|
Next Lectures:
|
|
|
|
- Field axioms
|
|
- Prime fields ($\mathbb{F}_p$)
|
|
- Field extensions (e.g. $\mathbb{F}_{p^t}$)
|
|
|