NoteNextra-origin/content/Math401/Freiwald_summer/Math401_T1.md

Math401 Topic 1: Probability under language of measure theory

Section 1: Uniform Random Numbers

Basic Definitions

Definition of Random Variables

A random variable is a function f:[0,1]\to S, where [0,1]\subset \mathbb{R} and S is a set of potential outcomes of a random phenomenon.

Definition of Uniform Distribution

The uniform distribution is defined by the length of function on subsets of [0,1] as a measure of probability (Lebesgue measure by default).

Let X be a random number taken from [0,1] and having the uniform distribution. The probability that X should be the probability of the event that X lies in A.

\operatorname{Prob}(X\in A) =\lambda(A)=\text{length of }A

Definition of Expectation

Let f:[0,1]\to \mathbb{R} be a random variable (with nice properties such that it is integrable). Then the expectation of f is defined as

\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_0^1 f(x)dx

Definition of Indicator Function

The indicator function of an event A is defined as

\mathbb{I}_A(x)=\begin{cases}
1 & \text{if } x\in A \\
0 & \text{if } x\notin A
\end{cases}

Definition of Law of variable X

The law of a random variable X is the probability distribution of X.

Let Y be the outcome of f(X). Then the law of Y is the probability distribution of Y.

\mu_Y(A)=\lambda(f^{-1}(A))=\lambda(\{x\in [0,1]: f(x)\in A\})

1.1 Mathematical Coin Flip model

A coin flip if a random experiment with two possible outcomes: S=\{0,1\}. with probability p for 0 and 1-p for 1, where p\in (0,1)\subset \mathbb{R}.

Definition of Independent Events

Two events A and B are independent if

\lambda(A\cap B)=\lambda(A)\lambda(B)

or equivalently,

\operatorname{Prob}(X\in A\cap B)=\operatorname{Prob}(X\in A)\operatorname{Prob}(X\in B)

Generalization to n events:

\lambda(A_1\cap A_2\cap \cdots \cap A_n)=\lambda(A_1)\lambda(A_2)\cdots \lambda(A_n)

Definition of Outcome selecting function

Let the set of all possible outcomes represented by a Cartesian product S=\{0,1\}^{\mathbb{N}}. (a_1,a_2,a_3,\cdots)\subset S is an infinite (or finite) sequence of coin flips.

\pi_i:S\to \{0,1\} is the $i$-th projection function defined as \pi_i((a_1,a_2,a_3,\cdots))=a_i.

Note, this representation is isomorphic to the dyadic rationals (i.e., numbers that can be written as a fraction whose denominator is a power of 2) in the interval [0,1].

Section 2: Formal definitions

Recall, the $\sigma$-algebra (denoted as \mathcal{A} in Math4121) is the collection of all subsets of a set S satisfying the following properties:

1. \emptyset\in \mathcal{A} (empty set is in the $\sigma$-algebra)
2. If A\in \mathcal{A}, then A^c\in \mathcal{A} (if a set is in the $\sigma$-algebra, then its complement is in the $\sigma$-algebra)
3. If A_1,A_2,A_3,\cdots\in \mathcal{A}, then \bigcup_{i=1}^{\infty}A_i\in \mathcal{A} (if a countable sequence of sets is in the $\sigma$-algebra, then their union is in the $\sigma$-algebra)

Event, probability, and random variable

Let \Omega be a non-empty set.

Let \mathscr{F} be a $\sigma$-algebra on \Omega (Note, \mathscr{F} is a collection of subsets of \Omega that satisfies the properties of a $\sigma$-algebra).

Definition of Event

An event is a element of \mathscr{F}.

Definition of Probability Measure

A probability measure P is a function P:\mathscr{F}\to [0,1] satisfying the following properties:

1. P(\Omega)=1
2. If A_1,A_2,A_3,\cdots\in \mathscr{F} are pairwise disjoint (\forall i\neq j, A_i\cap A_j=\emptyset), then P(\bigcup_{i=1}^{\infty}A_i)=\sum_{i=1}^{\infty}P(A_i)

Definition of Probability Space

A probability space is a triple (\Omega, \mathscr{F}, P) defined above.

An event A is said to occur almost surely (a.s.) if P(A)=1.

Definition of Random Variable

A random variable is a function X:\Omega\to \mathbb{R} that is measurable with respect to the $\sigma$-algebra \mathscr{F}.

That is, for any Borel set B\subset \mathbb{R}, the preimage f^{-1}(B)\in \mathscr{F}.

f^{-1}(B)=\{x\in \Omega: f(x)\in B\}\in \mathscr{F}

Definition of sigma-algebra generated by a random variable

Let \{f_\alpha:\Omega\to \mathbb{R},\alpha\in I\} be a family of functions where I is an index set which is not necessarily finite or countable. The $\sigma$-algebra generated by the family of functions \{f_\alpha:\alpha\in I\}, denoted as \sigma\{f_\alpha:\alpha\in I\}, is the smallest $\sigma$-algebra containing all the subsets of \Omega of the form

f_\alpha^{-1}(B)=\{\omega\in \Omega: f_\alpha(\omega)\in B\}\in \mathscr{F}

for all \alpha\in I and B\in \mathscr{B}(\mathbb{R}).

Equivalently,

\sigma\{f_\alpha:\alpha\in I\}=\sigma\left(\bigcup_{\alpha\in I}f_\alpha^{-1}(B)\right)

the sigma-algebra generated by a random variable X is the intersection of all $\sigma$-algebras on \Omega containing the sets f_\alpha^{-1}(B) for all \alpha\in I and B\in \mathscr{B}(\mathbb{R}).

Definition of distribution of random variable

Let f:\Omega\to \mathbb{R} be a random variable. The distribution of f is the probability measure P_f on \mathbb{R} defined by

P_f(B)=P(f^{-1}(B))=P(\{x\in \Omega: f(x)\in B\})

also noted as f_*P.

Definition of joint distribution of random variables

Let f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R} be random variables. The joint distribution of f_1,f_2,\cdots,f_n is the probability measure P_{f_1,f_2,\cdots,f_n} on \mathbb{R}^n defined by

P_{f_1,f_2,\cdots,f_n}(B)=P(f_1^{-1}(B_1)\cap f_2^{-1}(B_2)\cap \cdots \cap f_n^{-1}(B_n))=P(\omega\in \Omega: (f_1(\omega),f_2(\omega),\cdots,f_n(\omega))\in B)

Expectation of a random variable

Let f:\Omega\to \mathbb{R} be a random variable. The expectation of f is defined as

\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_\Omega f(x)dP

Note, P is the probability measure on \Omega.

Definition of variance

The variance of a random variable f is defined as

\operatorname{Var}(f)=\mathbb{E}[(f-\mathbb{E}[f])^2]=\mathbb{E}[f^2]-(\mathbb{E}[f])^2

Definition of covariance

The covariance of two random variables f,g:\Omega\to \mathbb{R} is defined as

\operatorname{Cov}(f,g)=\mathbb{E}[(f-\mathbb{E}[f])(g-\mathbb{E}[g])]

Point measures

Definition of Dirac measure

The Dirac measure is a probability measure on \Omega defined as

\delta_\omega(A)=\begin{cases}
1 & \text{if } \omega\in A \\
0 & \text{if } \omega\notin A
\end{cases}

Note that \int_\Omega f(x)d\delta_\omega(x)=f(\omega).

Infinite sequence of independent coin flips

Side notes from basic topology:

Definition of product topology:

It is a set constructed by the Cartesian product of the sets. Suppose X_i is a set for all i\in I. The element of the product set is a tuple (x_i)_{i\in I} where x_i\in X_i for all i\in I.

For example, if X_i=[0,1] for all i\in \mathbb{N}, then the product set is [0,1]^{\mathbb{N}}. An element of such product set is (1,0.5,0.25,\cdots).

The set of outcomes of such infinite sequence of coin flips is the product set of the set of outcomes of each coin flip.

S=\{0,1\}^{\mathbb{N}}

Conditional probability

Definition of conditional probability

The conditional probability of an event A given an event B is defined as

P(A|B)=\frac{P(A\cap B)}{P(B)}

The law of total probability:

P(A)=\sum_{i=1}^{\infty}P(A|B_i)P(B_i)

Bayes' theorem:

P(B_i|A)=\frac{P(A|B_i)P(B_i)}{\sum_{j=1}^{\infty}P(A|B_j)P(B_j)}

Definition of independence of random variables

Two random variables f,g:\Omega\to \mathbb{R} are independent if for any Borel sets A,B\subset \mathscr{B}(\mathbb{R}) the events

\{\omega\in \Omega: f(\omega)\in A\}\text{ and } \{\omega\in \Omega: g(\omega)\in B\}

are independent.

In general, a finite or infinite family of random variables f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R} are independent if every finite collection of random variables from this family are independent.

Definition of independence of sigma-algebras

Let \mathscr{G} and \mathscr{H} be two $\sigma$-algebras on \Omega. They are independent if for any Borel sets A\subset \mathscr{B}(\mathbb{R}) and B\subset \mathscr{B}(\mathbb{R}), the finite collection of events are independent.

Section 3: Further definitions in measure theory and integration

L^2 space

Definition of L^2 space

Let (\Omega, \mathscr{F}, P) be a measure space. The L^2 space is the space of all square integrable, complex-valued measurable functions on \Omega.

Denoted by L^2(\Omega, \mathscr{F}, P).

The square integrable functions are the functions f:\Omega\to \mathbb{C} such that

\int_\Omega |f(\omega)|^2 dP(\omega)<\infty

With inner product defined by

\langle f,g\rangle=\int_\Omega \overline{f(\omega)}g(\omega)dP(\omega)

The L^2(\Omega, \mathscr{F}, P) space is a Hilbert space.