# Topic 1: Probability under language of measure theory

## Section 1: Uniform Random Numbers

### Basic Definitions

#### Definition of Random Variables

A random variable is a function $f:[0,1]\to S$, where $[0,1]\subset \mathbb{R}$ and $S$ is a set of potential outcomes of a random phenomenon.

#### Definition of Uniform Distribution

The uniform distribution is defined by the length of function on subsets of $[0,1]$ as a measure of probability ([Lebesgue measure](https://notenextra.trance-0.com/Math4121/Math4121_L30#lebesgue-measure) by default).

Let $X$ be a random number taken from $[0,1]$ and having the uniform distribution. The probability that $X$ should be the probability of the event that $X$ lies in $A$.

$$
\operatorname{Prob}(X\in A) =\lambda(A)=\text{length of }A
$$

#### Definition of Expectation

Let $f:[0,1]\to \mathbb{R}$ be a random variable (with nice properties such that it is integrable). Then the expectation of $f$ is defined as

$$
\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_0^1 f(x)dx
$$

#### Definition of Indicator Function

The indicator function of an event $A$ is defined as

$$
\mathbb{I}_A(x)=\begin{cases}
1 & \text{if } x\in A \\
0 & \text{if } x\notin A
\end{cases}
$$

#### Definition of Law of variable X

The law of a random variable $X$ is the probability distribution of $X$.

Let $Y$ be the outcome of $f(X)$. Then the law of $Y$ is the probability distribution of $Y$.

$$
\mu_Y(A)=\lambda(f^{-1}(A))=\lambda(\{x\in [0,1]: f(x)\in A\})
$$

### 1.1 Mathematical Coin Flip model

A coin flip if a random experiment with two possible outcomes: $S=\{0,1\}$. with probability $p$ for $0$ and $1-p$ for $1$, where $p\in (0,1)\subset \mathbb{R}$.

#### Definition of Independent Events

Two events $A$ and $B$ are independent if

$$
\lambda(A\cap B)=\lambda(A)\lambda(B)
$$

or equivalently,

$$
\operatorname{Prob}(X\in A\cap B)=\operatorname{Prob}(X\in A)\operatorname{Prob}(X\in B)
$$

Generalization to $n$ events:

$$
\lambda(A_1\cap A_2\cap \cdots \cap A_n)=\lambda(A_1)\lambda(A_2)\cdots \lambda(A_n)
$$

#### Definition of Outcome selecting function

Let the set of all possible outcomes represented by a Cartesian product $S=\{0,1\}^{\mathbb{N}}$. $(a_1,a_2,a_3,\cdots)\subset S$ is an infinite (or finite) sequence of coin flips.

$\pi_i:S\to \{0,1\}$ is the $i$-th projection function defined as $\pi_i((a_1,a_2,a_3,\cdots))=a_i$.

> Note, this representation is isomorphic to the dyadic rationals (i.e., numbers that can be written as a fraction whose denominator is a power of 2) in the interval $[0,1]$.

## Section 2: Formal definitions

> Recall, the $\sigma$-algebra (denoted as $\mathcal{A}$ in Math4121) is the collection of all subsets of a set $S$ satisfying the following properties:
>
> 1. $\emptyset\in \mathcal{A}$ (empty set is in the $\sigma$-algebra)
> 2. If $A\in \mathcal{A}$, then $A^c\in \mathcal{A}$ (if a set is in the $\sigma$-algebra, then its complement is in the $\sigma$-algebra)
> 3. If $A_1,A_2,A_3,\cdots\in \mathcal{A}$, then $\bigcup_{i=1}^{\infty}A_i\in \mathcal{A}$ (if a countable sequence of sets is in the $\sigma$-algebra, then their union is in the $\sigma$-algebra)

### Event, probability, and random variable

Let $\Omega$ be a non-empty set.

Let $\mathscr{F}$ be a $\sigma$-algebra on $\Omega$ (Note, $\mathscr{F}$ is a collection of subsets of $\Omega$ that satisfies the properties of a $\sigma$-algebra).

#### Definition of Event

An event is a element of $\mathscr{F}$.

#### Definition of Probability Measure

A probability measure $P$ is a function $P:\mathscr{F}\to [0,1]$ satisfying the following properties:

1. $P(\Omega)=1$
2. If $A_1,A_2,A_3,\cdots\in \mathscr{F}$ are pairwise disjoint ($\forall i\neq j, A_i\cap A_j=\emptyset$), then $P(\bigcup_{i=1}^{\infty}A_i)=\sum_{i=1}^{\infty}P(A_i)$

#### Definition of Probability Space

A probability space is a triple $(\Omega, \mathscr{F}, P)$ defined above.

An event $A$ is said to occur almost surely (a.s.) if $P(A)=1$.

#### Definition of Random Variable

A random variable is a function $X:\Omega\to \mathbb{R}$ that is measurable with respect to the $\sigma$-algebra $\mathscr{F}$.

That is, for any Borel set $B\subset \mathbb{R}$, the preimage $f^{-1}(B)\in \mathscr{F}$.

$$
f^{-1}(B)=\{x\in \Omega: f(x)\in B\}\in \mathscr{F}
$$

#### Definition of sigma-algebra generated by a random variable

Let $\{f_\alpha:\Omega\to \mathbb{R},\alpha\in I\}$ be a family of functions where $I$ is an index set which is not necessarily finite or countable. The $\sigma$-algebra generated by the family of functions $\{f_\alpha:\alpha\in I\}$, denoted as $\sigma\{f_\alpha:\alpha\in I\}$, is the smallest $\sigma$-algebra containing all the subsets of $\Omega$ of the form

$$
f_\alpha^{-1}(B)=\{\omega\in \Omega: f_\alpha(\omega)\in B\}\in \mathscr{F}
$$

for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.

Equivalently,

$$
\sigma\{f_\alpha:\alpha\in I\}=\sigma\left(\bigcup_{\alpha\in I}f_\alpha^{-1}(B)\right)
$$

the sigma-algebra generated by a random variable $X$ is the intersection of all $\sigma$-algebras on $\Omega$ containing the sets $f_\alpha^{-1}(B)$ for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.

#### Definition of distribution of random variable

Let $f:\Omega\to \mathbb{R}$ be a random variable. The distribution of $f$ is the probability measure $P_f$ on $\mathbb{R}$ defined by

$$
P_f(B)=P(f^{-1}(B))=P(\{x\in \Omega: f(x)\in B\})
$$

also noted as $f_*P$.

#### Definition of joint distribution of random variables

Let $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ be random variables. The joint distribution of $f_1,f_2,\cdots,f_n$ is the probability measure $P_{f_1,f_2,\cdots,f_n}$ on $\mathbb{R}^n$ defined by

$$
P_{f_1,f_2,\cdots,f_n}(B)=P(f_1^{-1}(B_1)\cap f_2^{-1}(B_2)\cap \cdots \cap f_n^{-1}(B_n))=P(\omega\in \Omega: (f_1(\omega),f_2(\omega),\cdots,f_n(\omega))\in B)
$$

### Expectation of a random variable

Let $f:\Omega\to \mathbb{R}$ be a random variable. The expectation of $f$ is defined as

$$
\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_\Omega f(x)dP
$$

Note, $P$ is the probability measure on $\Omega$.

#### Definition of variance

The variance of a random variable $f$ is defined as

$$
\operatorname{Var}(f)=\mathbb{E}[(f-\mathbb{E}[f])^2]=\mathbb{E}[f^2]-(\mathbb{E}[f])^2
$$

#### Definition of covariance

The covariance of two random variables $f,g:\Omega\to \mathbb{R}$ is defined as

$$
\operatorname{Cov}(f,g)=\mathbb{E}[(f-\mathbb{E}[f])(g-\mathbb{E}[g])]
$$

### Point measures

#### Definition of Dirac measure

The Dirac measure is a probability measure on $\Omega$ defined as

$$
\delta_\omega(A)=\begin{cases}
1 & \text{if } \omega\in A \\
0 & \text{if } \omega\notin A
\end{cases}
$$

Note that $\int_\Omega f(x)d\delta_\omega(x)=f(\omega)$.

### Infinite sequence of independent coin flips

> Side notes from basic topology:
>
> **Definition of product topology**:
>
> It is a set constructed by the Cartesian product of the sets. Suppose $X_i$ is a set for all $i\in I$. The element of the product set is a tuple $(x_i)_{i\in I}$ where $x_i\in X_i$ for all $i\in I$.
>
> For example, if $X_i=[0,1]$ for all $i\in \mathbb{N}$, then the product set is $[0,1]^{\mathbb{N}}$. An element of such product set is $(1,0.5,0.25,\cdots)$.

The set of outcomes of such infinite sequence of coin flips is the product set of the set of outcomes of each coin flip.

$$
S=\{0,1\}^{\mathbb{N}}
$$

### Conditional probability

#### Definition of conditional probability

The conditional probability of an event $A$ given an event $B$ is defined as

$$
P(A|B)=\frac{P(A\cap B)}{P(B)}
$$

The law of total probability:

$$
P(A)=\sum_{i=1}^{\infty}P(A|B_i)P(B_i)
$$

Bayes' theorem:

$$
P(B_i|A)=\frac{P(A|B_i)P(B_i)}{\sum_{j=1}^{\infty}P(A|B_j)P(B_j)}
$$

#### Definition of independence of random variables

Two random variables $f,g:\Omega\to \mathbb{R}$ are independent if for any Borel sets $A,B\subset \mathscr{B}(\mathbb{R})$ the events

$$
\{\omega\in \Omega: f(\omega)\in A\}\text{ and } \{\omega\in \Omega: g(\omega)\in B\}
$$

are independent.

In general, a finite or infinite family of random variables $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ are independent if every finite collection of random variables from this family are independent.

#### Definition of independence of sigma-algebras

Let $\mathscr{G}$ and $\mathscr{H}$ be two $\sigma$-algebras on $\Omega$. They are independent if for any Borel sets $A\subset \mathscr{B}(\mathbb{R})$ and $B\subset \mathscr{B}(\mathbb{R})$, the finite collection of events are independent.

## Section 3: Further definitions in measure theory and integration

### $L^2$ space

#### Definition of $L^2$ space

Let $(\Omega, \mathscr{F}, P)$ be a measure space. The $L^2$ space is the space of all square integrable, complex-valued measurable functions on $\Omega$.

Denoted by $L^2(\Omega, \mathscr{F}, P)$.

The square integrable functions are the functions $f:\Omega\to \mathbb{C}$ such that

$$
\int_\Omega |f(\omega)|^2 dP(\omega)<\infty
$$

With inner product defined by

$$
\langle f,g\rangle=\int_\Omega \overline{f(\omega)}g(\omega)dP(\omega)
$$

The $L^2(\Omega, \mathscr{F}, P)$ space is a Hilbert space.