update

content/CSE5313/CSE5313_L3.md

@@ -0,0 +1,307 @@
+# CSE5313 Coding and information theory for data science (Lecture 3)
+
+Finite Fields
+
+## Why finite fields?
+
+Most information systems are discrete.
+
+- Use bits, byte etc.
+
+Use bits/bytes to represent real numbers.
+
+- Problems of overflow, accuracy, etc.
+
+We wish to build "good" codes $\mathcal{C} \subset \mathbb{F}^n$:
+
+- Large $\frac{k}{n}$
+- Lage $d_H(\mathcal{C})\implies$ error detection/correction, erasure correction.
+
+Idea: Use linear algebraic operations to encode/decode.
+
+- $F=\mathbb{F}_q$, a finite field with $q$ elements.
+
+## Finite fields
+
+### Fields and field axioms
+
+A field is a set $\mathbb{F}$ with two operations $+$ and $\cdot$ that satisfy the following axioms:
+
+- Associativity: $(a+b)+c = a+(b+c)$ and $(a\cdot b)\cdot c = a\cdot (b\cdot c)$
+- Commutativity: $a+b = b+a$ and $a\cdot b = b\cdot a$
+- Distributivity: $a\cdot (b+c) = a\cdot b + a\cdot c$
+- Existence of Identity elements: $a+0 = a$ and $a\cdot 1 = a$
+- Existence of Inverse elements: $a+(-a) = 0$ and $a\cdot a^{-1} = 1$
+
+Every set of elements which satisfies these axioms is a field.
+
+We can "do algebra" over it (matrices, vector spaces, etc.).
+
+Are there finite sets which satisfy the field axioms?
+
+What are the possible sizes of such sets?
+
+### Background – Basic number theory
+
+- For $a, b \in \mathbb{N}$,
+  - Greatest Common Denominator: $\gcd(a, b) =$ the largest integer $m$ such that $m|a$ and $m|b$.
+  - Lowest Common Multiplier: $\operatorname{lcm}(a, b) =$ the smallest integer $m$ such that $a|m$ and $b|m$.
+- $a, b$ are coprime if $\gcd(a, b) = 1$.
+- Fact: (Euclid’s lemma) Say $a \geq b$,
+  - There exists a quotient $q \geq 0$ and a remainder $0 \leq r < b$ such that $a = bq + r$.
+- Theorem (Euclid): If $\gcd(a, b) = 1$ then there exist $m, n \in \mathbb{Z}$ such that $am + bn = 1$.
+  - Proof by repeated application of Euclid’s lemma.
+  - Example:
+    - If $a = 3, b = 8$,
+    - then $m = -5, n = 2$,
+    - satisfy $3 \cdot -5 + 8 \cdot 2 = 1$.
+
+### Modular arithmetic
+
+Defined a set with addition $\oplus$ and multiplication $\odot$ that satisfy the field axioms.
+
+$\mathbb{Z}_p$ is a field if $p$ is a prime number.
+
+- Addition and multiplication are defined modulo $p$.
+- $a \oplus b = (a+b) \mod p$
+- $a \odot b = (a\cdot b) \mod p$
+
+- $0$ is the additive identity.
+- $1$ is the multiplicative identity.
+- $a$ has an additive inverse $p-a$.
+- $a$ has a multiplicative inverse $a^{-1}$ such that $a \odot a^{-1} = 1$.
+
+Proof for existence of multiplicative inverse for $a\in \mathbb{Z}_p\setminus \{0\}$:
+
+<details>
+<summary>Proof</summary>
+
+Since $p$ is prime, $\gcd(a, p) = 1$.
+
+By euclid's theorem, there exist $m, n \in \mathbb{Z}$ such that $am + pn = 1$.
+
+Take mod $p$ on both sides:
+
+$$
+a_{\mod p}\odot m_{\mod p} \equiv 1_{\mod p}
+$$
+
+Thus, $m_{\mod p}$ is the multiplicative inverse of $a_{\mod p}$.
+
+</details>
+
+Polynomials over prime fields is also a field.
+
+$(\mathbb{Z}_2,\operatorname{XOR},\operatorname{AND})$ is a field.
+
+### Polynomials over finite fields
+
+A polynomial over a field $\mathbb{Z}_p$ is a expression of the form:
+
+$$
+a(x)=\sum_{i=0}^n a_i x^i
+$$
+
+- Polynomial degree: largest index of a non-zero coefficient.
+- Polynomial addition: $a(x) \oplus b(x) = \sum_{i=0}^n (a_i \oplus b_i) x^i$
+- Polynomial multiplication: $a(x)\odot b(x) = \sum_{i=0}^n \sum_{j=0}^n a_i \odot b_j x^{i+j}$
+- Polynomial equality: $a(x) = b(x)$ if and only if $a_i = b_i$ for all $i$.
+- Polynomial division: suppose $\deg(a(x)) \geq \deg(b(x))$, then there exist unique polynomials $q(x)$ and $r(x)$ such that $a(x) = b(x)q(x) \oplus r(x)$ and $\deg(r(x)) < \deg(b(x))$. (do long division for polynomials)
+
+denoted as $\mathbb{Z}_p[x]$.
+
+<details>
+<summary>Example</summary>
+
+$$
+p(x) = x^2 + 6x+3\in \mathbb{Z}_7[x]
+$$
+
+$p(1) = 1^2 + 6\cdot 1 + 3 = 10 \equiv 3 \mod 7$
+
+$p(2) = 2^2 + 6\cdot 2 + 3 = 4+5+3 = 12 \equiv 5 \mod 7$
+
+</details>
+
+#### Irreducible polynomials
+
+A polynomial $p(x)$ is irreducible if it cannot be factored into two non-constant polynomials.
+
+If $\gcd(a(x),b(x))=1$, then there exist $m(x),n(x)\in \mathbb{Z}_p[x]$ such that $a(x)m(x)\oplus b(x)n(x)=1$.
+
+Proved similar to euclid's theorem.
+
+> [!TIP]
+>
+> If a polynomial $p(x)$ has a root, say $r$, then $p(x) = (x-r)q(x)$ for some $q(x)\in \mathbb{Z}_p[x]$.
+
+Example in $\mathbb{Z}_2[x]$:
+
+$$
+p(x) = x^2 \oplus 1
+$$
+
+is reducible because $p(x) = (x\oplus 1)(x\oplus 1)$.
+
+$$
+p(x) = x^3 \oplus x \oplus 1
+$$
+
+is irreducible.
+
+<details>
+<summary>Proof</summary>
+
+We prove by contradiction.
+
+Suppose $p(x)$ is reducible, then $p(x) = a(x)b(x)$ for some $a(x),b(x)\in \mathbb{Z}_2[x]$.
+
+Then $\deg(p(x)) = \deg(a(x)) + \deg(b(x))$.
+
+Let $\deg b(x)=1$, then $b(x) \in \{x, x\oplus 1\}$.
+
+If $b(x) = x$, then $p(0)=0$ but $p(x)$ is $1$.
+
+If $b(x) = x\oplus 1$, then $p(1)=0$ but $p(x)$ is $1$.
+
+</details>
+
+It is not the case in $\mathbb{Z}_2[x]$, that every polynomial with no root is irreducible. (e.g consider $(x^3\oplus x\oplus 1)^2$ has no root but is reducible.)
+
+#### Polynomial modular arithmetic
+
+There exist quotient $q(x)$ and remainder $r(x)$, $\deg(r(x)) < \deg(b(x))$ such that
+
+$$
+a(x) = b(x)q(x) + r(x)
+$$
+
+$$
+\implies a(x) \mod b(x) = r(x)
+$$
+
+"$\mod b(x)$" is an operation on polynomials in $\mathbb{Z}_p[x]$ that:
+
+- Preserves polynomial addition:
+  - $a(x) \oplus c(x) \mod b(x) = a(x) \mod b(x) \oplus c(x) \mod b(x)$
+- Preserves polynomial multiplication:
+  - $a(x) \odot c(x) \mod b(x) = a(x) \mod b(x) \odot c(x) \mod b(x)$
+
+### Extension fields
+
+Let $p$ be a prime number. then $(\mathbb{Z}_p[x], \oplus, \odot)$ is a field.
+
+Fix a polynomial $f(x)\in \mathbb{Z}_p[x]$ of degree $t$.
+
+Define a set
+
+Elements: polynomials of degree at most $t-1$ in $\mathbb{Z}_p[x]$. (finite set, size is $p^t$.)
+
+Define addition:
+
+$$
+a(x) \oplus_f b(x) = (a(x) \oplus b(x)) \mod f(x)
+$$
+
+Define multiplication:
+
+$$
+a(x) \odot_f b(x) = (a(x) \odot b(x)) \mod f(x)
+$$
+
+Denote this set as $\mathbb{Z}_p[x] \mod f(x)$.
+
+This is not a field because it does not have a multiplicative inverse for every element.
+
+<details>
+<summary>Proof</summary>
+
+We prove by contradiction.
+
+Suppose there exists a polynomial $g(x)\in \mathbb{Z}_p[x] \mod f(x)$ such that $a(x) \odot_f g(x) = 1$.
+
+Let $p=2,f(x)=x^2\oplus 1$.
+
+The polynomials in $\mathbb{Z}_2[x] \mod f(x)$ are $\{0, 1, x, x\oplus 1\}$.
+
+Consider the modular inverse of $(x\oplus 1)$.
+
+- $0\odot_f (x\oplus 1) = 0$
+- $1\odot_f (x\oplus 1) = x\oplus 1$
+- $x\odot_f (x\oplus 1) = (x^2\oplus x)\mod (x^2\oplus 1) = x\oplus 1$
+- $(x\oplus 1)\odot_f (x\oplus 1) = (x^2\oplus 1)\mod (x^2\oplus 1) = 0$
+
+</details>
+
+To make our field extension works, we need to find a polynomial $f(x)$ that is irreducible.
+
+Theorem: If $f(x)$ is irreducible over $\mathbb{Z}_p$, then $\mathbb{Z}_p[x] \mod f(x)$ is a field.
+
+<details>
+<summary>Proof</summary>
+
+Let $a(x)\in \mathbb{Z}_p[x] \mod f(x)$, $a(x)\neq 0$.
+
+Existence of $a(x)^{-1}$ in $\mathbb{Z}_p[x] \mod f(x)$ can be done by Euclid's Theorem.
+
+Since $\gcd(a(x),f(x))=1$, there exist $m(x),n(x)\in \mathbb{Z}_p[x]$ such that $a(x)m(x)\oplus f(x)n(x)=1$.
+
+Take mod $f(x)$ on both sides:
+
+$$
+a(x)m(x) \mod f(x) = 1 \mod f(x)
+$$
+
+Thus, $m(x) \mod f(x)$ is the multiplicative inverse of $a(x) \mod f(x)$.
+
+So $a(x)^{-1} = m(x) \mod f(x)$.
+
+</details>
+
+Corollary:
+
+We can extend a prime field $\mathbb{Z}_p$ with irreducible polynomial
+
+Intuitively, we add to $\mathbb{Z}_p$ a new element $x$ that satisfies $f(x)=0$.
+
+Observation: – We only used the general field properties of $\mathbb{Z}_p$. – ⇒ any “base field” can be used instead of $\mathbb{Z}_p$. – ⇒ Any field can be “extended”.
+
+Say we wish to build a field $F$ with $2^8$ elements.
+
+- Option 1:
+  - Take $\mathbb{Z}_2$ and $f(x)$ irreducible of degree 8.
+  - $F = \mathbb{Z}_2[x] \mod f(x)$.
+
+- Option 2:
+  - Take $\mathbb{Z}_2$, and $g_1(x) \in \mathbb{Z}_2[x]$ irreducible of degree 4,
+  - $F_1 = \mathbb{Z}_2[x] \mod g_1(x)$. Note $|F_1| = 2^4 = 16$.
+  - Take $g_2(x) \in F_1[x]$ irreducible of degree 2.
+  - $F_2 = F_1[x] \mod g_2(x)$.
+
+#### Uniqueness of the finite field
+
+Theorems:
+
+- As long as it is irreducible, the choice of $f(x)$ does not matter.
+  - If $f_1(x), f_2(x)$ are irreducible of the same degree, then $\mathbb{Z}_p[x] \mod f_1(x) \cong \mathbb{Z}_p[x] \mod f_2(x)$.
+- Over every $\mathbb{Z}_p$ (𝑝 prime), there exists an irreducible polynomial of every degree.
+- All finite fields of the same size are isomorphic.
+- All finite fields are of size $p^d$ for prime $p$ and integer $d$.
+
+Corollary: This is effectively the **only** way to construct finite fields!
+
+#### Extension of fields
+
+$\mathbb{R}[x]\mod (x^2+1)$ is a field, $\cong \mathbb{C}$.
+
+|Terms | Finite field extension $F_1\to F_2$ | $\mathbb{R}\to \mathbb{C}$ |
+|---|---|---|
+|Base field| any field $\mathbb{F}_1$ | $\mathbb{R}$ |
+|Irreducible polynomial| $f(x)$ | $x^2+1$ |
+|New elements added| $x$ | $i$ |
+| Add/mul| $\mod f(x)$ | $\mod (x^2+1)$ |
+
+You cannot do algebraic extension of $\mathbb{Q}$ to $\mathbb{R}$.
+
+Transcendental extension:
+

content/CSE5313/_meta.js

@@ -5,4 +5,5 @@ export default {
     },
     CSE5313_L1: "CSE5313 Coding and information theory for data science (Lecture 1)",
     CSE5313_L2: "CSE5313 Coding and information theory for data science (Lecture 2)",
+    CSE5313_L3: "CSE5313 Coding and information theory for data science (Lecture 3)",
 }