# CSE5313 Coding and information theory for data science (Lecture 15)

## Information theory

Transmission, processing, extraction, and utilization of information.

- Information: "Resolution" of uncertainty.
- Question in the 1940's: How to quantify the complexity of information?
  - Questions:
    - How many bits are required to **describe** an information source?
    - How many bits are required to **transmit** an information source?
    - How much information does one source **reveal** about another?
  - Applications:
    - Data Compression.
    - Channel Coding.
    - Privacy.
- Claude Shannon 1948: Information Entropy.

### Entropy

The "information value" of a message depends on how surprising it is.

- The more unlikely the event, the more informative the message.

The **Shannon information** of an event $E$:

$$
I(E)=-\log_2 \frac{1}{Pr(E)}
$$

Entropy the the expected amount of information in a random trial.

- Rolling a die has more entropy than rolling a coin (6 states vs 2 states).

#### Information entropy

Let $X$ be a random variable with values in some finite set $\mathcal{X}$.

The entropy $H(X)$ of $X$ is defined as:

$$
H(X)=\sum_{x\in \mathcal{X}} \log_2 \frac{1}{Pr(X=x)}=-\mathbb{E}_{x\sim X}[Pr(X=x)\log_2 Pr(X=x)]
$$

Idea: How many bits are required, on average, to describe $X$?

- The more unlikely the event, the more informative the message.
- Use "few" bits for common $x$ (i.e., $Pr(x)$ large).
- Use "many" bits for rare $x$ (i.e., $Pr(x)$ small).

Notes:

- $H(X)=\mathbb{E}_{x\sim X}[I(X=x)]$
- $H(X)\geq 0$
- $H(X)=0$ if and only if $Pr(X=x)=1$ for some $x\in \mathcal{X}$.
- Does not depend on $\mathcal{X}$ but on the probability distribution $Pr(X=x)$.
- Maximum entropy is achieved when the distribution is uniform. $H(Uniform(n))=\log_2 n$. Where $n$ is the number of events. ($a$ bits required to describe each event of size $2^a$).

<details>
<summary>Example</summary>

For uniform distribution $X\sim Uniform\{0,1\}$, we have $H(X)=\log_2 2=1$.

---

For Bernoulli distribution $X\sim Bernoulli(p)$, we have $H(X)=-p\log_2 p-(1-p)\log_2 (1-p)=H(p)$.

</details>

### Motivation

Optimal compression:

Consider the $X$ with distribution given belows:

|Value|Probability| Encoding|
|-----|-----------|---------|
|1    | 1/8       |000      |
|2    | 1/4       |001      |
|3    | 1/4       |01       |
|4    | 1/2       |1        |

The average length of the encoding is $1/8\times 3+1/4\times 3+1/4\times 2+1/2\times 1=7/4$.

And the entropy of $X$ is $H(X)=1/8\times \log_2 8+1/4\times \log_2 4+1/4\times \log_2 2+1/2\times \log_2 1=7/4$.

So the average length of the encoding is equal to the entropy of $X$.

This is the optimal compression.

This is the Huffman coding.

#### Few extra theorems that will not be proved in CS course

- **Theorem**: Avg. # of bits in any prefix-free compression of $X$ is ≥ $H(X)$.
- **Theorem**: Huffman coding is optimal.
  - I.e., Avg. # of bits equals $H(X)$.
- **Disadvantage of Huffman coding**: the distribution must be known.
  - Generally not the case.
- **[Lempel-Ziv 1978]**: Universal lossless data compression.

### Conditional and joint entropy

How does the entropy of different random variables interact?

- As a function of their dependence.
- Needed in order to relate:
  - A variable and its compression.
  - A variable and its transmission over a noisy channel.
  - A variable and its encryption

#### Definition for joint entropy

Let $X$ and $Y$ be discrete random variables.

The joint entropy $H(X,Y)$ of $X$ and $Y$ is defined as:

$$
H(X,Y)=-\sum_{x\in \mathcal{X}, y\in \mathcal{Y}} Pr(X=x, Y=y) \log_2 Pr(X=x, Y=y)
$$

Notes:

- $H(X,Y)\geq 0$
- $H(X,Y)=H(Y,X)$
- $H(X,Y)\geq \max\{H(X),H(Y)\}$
- $H(X,Y)\leq H(X)+H(Y)$ with equality if and only if $X$ and $Y$ are independent.

#### Conditional entropy

> [!NOTE]
>
> Recall that the conditional probability $P(Y|X)$ is defined as: $P(Y|X)=\frac{P(Y,X)}{P(X)}$.

What is the average amount of information revealed by $Y$ given the distribution of $X$?

For each given $x\in \mathcal{X}$, the conditional entropy $H(Y|X=x)$ is defined as:

$$
\begin{aligned} 
H(Y|X=x)&=-\sum_{y\in \mathcal{Y}} \log_2 \frac{1}{Pr(Y=y|X=x)} \\
&=-\sum_{y\in \mathcal{Y}} Pr(Y=y|X=x) \log_2 Pr(Y=y|X=x) \\
\end{aligned}
$$

The conditional entropy $H(Y|X)$ is defined as:

$$
\begin{aligned}
H(Y|X)&=\mathbb{E}_{x\sim X}[H(Y|X=x)] \\
&=-\sum_{x\in \mathcal{X}} Pr(X=x)H(Y|X=x) \\
&=-\sum_{x\in \mathcal{X}, y\in \mathcal{Y}} Pr(X=x, Y=y) \log_2 Pr(Y=y|X=x) \\
&=-\sum_{x\in \mathcal{X}, y\in \mathcal{Y}} Pr(x)\sum_{y\in \mathcal{Y}} Pr(Y=y|X=x) \log_2 Pr(Y=y|X=x) \\
\end{aligned}
$$

Notes:

- $H(X|X)=0$
- $H(Y|X)=0$ when $Y$ is a function of $X$.
- Conditional entropy not necessarily symmetric. $H(X|Y)\neq H(Y|X)$.
- Conditioning will not increase entropy. $H(X|Y)\leq H(X)$.

#### Chain rules of entropy

Joint entropy: $H(X,Y)=\mathbb{E}_{x\sim X, y\sim Y}\log \frac{1}{Pr(X=x, Y=y)}$.

Conditional entropy: $H(Y|X)=\mathbb{E}_{x\sim X}\log \frac{1}{Pr(Y=y|X=x)}$.

Chain rule: $H(X,Y)=H(X)+H(Y|X)$.

<details>
<summary>Proof</summary>

For **statistically independent** $X$ and $Y$, we have $Pr(x,y)=Pr(x)Pr(y)$.

$Pr(y|x)=\frac{Pr(x,y)}{Pr(x)}=\frac{Pr(x)Pr(y)}{Pr(x)}=Pr(y)$.

_Apply the symmetry of the joint distribution_.

</details>

<details>
<summary>Example of computing the conditional entropy</summary>

For the given distribution:

|Y\X|1|2|3|4|Y marginal|
|-----|-----|-----|-----|-----|-----|
|1    |1/8  |1/16  |1/32  |1/32  |1/4  |
|2    |1/16 |1/8   |1/16  |1/8   |1/4  |
|3    |1/16 |1/16  |1/16   |1/16  |1/4  |
|4    |1/4 |0|0  |0   |1/4  |
|X marginal|1/2  |1/4   |1/8   |1/8   |1.0  |

Here $H(X)=H((1/2,1/4,1/8,1/8))=-(1/2\log_2 1/2+1/4\log_2 1/4+1/8\log_2 1/8+1/8\log_2 1/8)=7/4$.

$H(Y)=H((1/4,1/4,1/4,1/4))=-(1/4\log_2 1/4+1/4\log_2 1/4+1/4\log_2 1/4+1/4\log_2 1/4)=2$.

$H(Y|X=1)=H((1/4,1/8,1/8,1/2))=-(1/4\log_2 1/4+1/8\log_2 1/8+1/8\log_2 1/8+1/2\log_2 1/2)=7/4$.

$H(Y|X=2)=H((1/4,1/2,1/4))=-(1/4\log_2 1/4+1/2\log_2 1/2+1/4\log_2 1/4)=1.5$.

$H(Y|X=3)=H((1/4,1/4,1/2,1/4))=-(1/4\log_2 1/4+1/4\log_2 1/4+1/2\log_2 1/2+1/4\log_2 1/4)=1.5$.

$H(Y|X=4)=H((1/4,1/4,1/2))=-(1/4\log_2 1/4+1/4\log_2 1/4+1/2\log_2 1/2)=1.5$.

So $H(Y|X)=\sum_{x\in \mathcal{X}} Pr(x)H(Y|X=x)=1/2\times 7/4+1/4\times 1.5+1/8\times 1.5+1/8\times 1.5=13/8$



</details>

### Mutual information

#### Definition for mutual information

The mutual information $I(X;Y)$ of $X$ and $Y$ is defined as:

$$
I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)
$$

#### Properties of mutual information

- $I(X;Y)\geq 0$
  - Prove via Jensen's inequality.
  - Conditioning will not increase entropy.
- $I(X;Y)=I(Y;X)$ symmetry.
- $I(X;X)=H(X)-H(X|X)=H(X)$
- $I(X;Y)=H(X)+H(Y)-H(X,Y)$

<details>
<summary>Example of computing the mutual information</summary>

For the given distribution:

|Y\X|1|2|3|4|Y marginal|
|-----|-----|-----|-----|-----|-----|
|1    |1/8  |1/16  |1/32  |1/32  |1/4  |
|2    |1/16 |1/8   |1/16  |1/8   |1/4  |
|3    |1/16 |1/16  |1/16   |1/16  |1/4  |
|4    |1/4 |0|0  |0   |1/4  |
|X marginal|1/2  |1/4   |1/8   |1/8   |1.0  |

Recall from the previous example:

$H(Y|X)=\frac{13}{8}$
$H(Y)=2$

then the mutual information is:

$$
I(X;Y)=H(Y)-H(Y|X)=2-\frac{13}{8}=\frac{3}{8}
$$

So the entropy of $Y$ is reduced by $\frac{3}{8}$ bits on average when $X$ is known.

</details>

## Applications

### Channel capacity

Recall a channel is tuple of $(F,\Phi,\operatorname{Pr})$.

#### Definition of discrete channel

A discrete channel is a system consisting of a discrete input alphabet $\mathcal{X}$, a discrete output alphabet $\mathcal{Y}$, and a probability transition $\operatorname{Pr}(y|x)$.

The channel is _memoryless_ if the output at any time depends only on the input at that time and not on the inputs and outputs at other times.

#### Definition of channel capacity

The channel capacity $C$ of a channel is defined as:

$$
C=\max_{x\in \mathcal{X}} I(X;Y)
$$

where $I(X;Y)$ is the mutual information between $X$ and $Y$.

#### Shannon's noisy coding theorem

Recall the rate of the code is $R=\frac{\log_{|F|}|\mathcal{C}|}{n}$.

For a discrete memoryless channel with capacity $C$, every rate $R<C$ is achievable.

- There exists a sequence of codes of length $n$ and size $|F|^{nR}$ with vanishing probability of error as $n\to \infty$.
- In 1 channel use, it is impossible beat the noise.
- In $n$ channel uses as $n\to \infty$,it is possible to beat the **all** the noise.

#### Corollary for the binary symmetric channel

The capacity of the binary symmetric channel with crossover probability $p$ is $1-H(p)$.

<details>
<summary>Proof</summary>

$$
\begin{aligned}
H(Y|X=x)&=-\operatorname{Pr}(y=0|x)\log_2 \operatorname{Pr}(y=0|x)-\operatorname{Pr}(y=1|x)\log_2 \operatorname{Pr}(y=1|x) \\
&=-p\log p-(1-p)\log (1-p)\\
&=H(p)
\end{aligned}
$$

So,

$$
\begin{aligned}
H(Y|X)&=\sum_{x\in \mathcal{X}} \operatorname{Pr}(x)H(Y|X=x)\\
&=H(p)\sum_{x\in \mathcal{X}} \operatorname{Pr}(x) \\
&=H(p)
$$

So,

$$
\begin{aligned}
I(X;Y)&=H(Y)-H(Y|X)\\
&\leq 1-H(p)
$$

When $p=0$ the capacity is $1$.

When $p=\frac{1}{2}$ the capacity is $0$. (completely noisy channel)

</details>