276 lines
9.0 KiB
Markdown
276 lines
9.0 KiB
Markdown
# Math401 Topic 1: Probability under language of measure theory
|
|
|
|
## Section 1: Uniform Random Numbers
|
|
|
|
### Basic Definitions
|
|
|
|
#### Definition of Random Variables
|
|
|
|
A random variable is a function $f:[0,1]\to S$, where $[0,1]\subset \mathbb{R}$ and $S$ is a set of potential outcomes of a random phenomenon.
|
|
|
|
#### Definition of Uniform Distribution
|
|
|
|
The uniform distribution is defined by the length of function on subsets of $[0,1]$ as a measure of probability ([Lebesgue measure](https://notenextra.trance-0.com/Math4121/Math4121_L30#lebesgue-measure) by default).
|
|
|
|
Let $X$ be a random number taken from $[0,1]$ and having the uniform distribution. The probability that $X$ should be the probability of the event that $X$ lies in $A$.
|
|
|
|
$$
|
|
\operatorname{Prob}(X\in A) =\lambda(A)=\text{length of }A
|
|
$$
|
|
|
|
#### Definition of Expectation
|
|
|
|
Let $f:[0,1]\to \mathbb{R}$ be a random variable (with nice properties such that it is integrable). Then the expectation of $f$ is defined as
|
|
|
|
$$
|
|
\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_0^1 f(x)dx
|
|
$$
|
|
|
|
#### Definition of Indicator Function
|
|
|
|
The indicator function of an event $A$ is defined as
|
|
|
|
$$
|
|
\mathbb{I}_A(x)=\begin{cases}
|
|
1 & \text{if } x\in A \\
|
|
0 & \text{if } x\notin A
|
|
\end{cases}
|
|
$$
|
|
|
|
#### Definition of Law of variable X
|
|
|
|
The law of a random variable $X$ is the probability distribution of $X$.
|
|
|
|
Let $Y$ be the outcome of $f(X)$. Then the law of $Y$ is the probability distribution of $Y$.
|
|
|
|
$$
|
|
\mu_Y(A)=\lambda(f^{-1}(A))=\lambda(\{x\in [0,1]: f(x)\in A\})
|
|
$$
|
|
|
|
### 1.1 Mathematical Coin Flip model
|
|
|
|
A coin flip if a random experiment with two possible outcomes: $S=\{0,1\}$. with probability $p$ for $0$ and $1-p$ for $1$, where $p\in (0,1)\subset \mathbb{R}$.
|
|
|
|
#### Definition of Independent Events
|
|
|
|
Two events $A$ and $B$ are independent if
|
|
|
|
$$
|
|
\lambda(A\cap B)=\lambda(A)\lambda(B)
|
|
$$
|
|
|
|
or equivalently,
|
|
|
|
$$
|
|
\operatorname{Prob}(X\in A\cap B)=\operatorname{Prob}(X\in A)\operatorname{Prob}(X\in B)
|
|
$$
|
|
|
|
Generalization to $n$ events:
|
|
|
|
$$
|
|
\lambda(A_1\cap A_2\cap \cdots \cap A_n)=\lambda(A_1)\lambda(A_2)\cdots \lambda(A_n)
|
|
$$
|
|
|
|
#### Definition of Outcome selecting function
|
|
|
|
Let the set of all possible outcomes represented by a Cartesian product $S=\{0,1\}^{\mathbb{N}}$. $(a_1,a_2,a_3,\cdots)\subset S$ is an infinite (or finite) sequence of coin flips.
|
|
|
|
$\pi_i:S\to \{0,1\}$ is the $i$-th projection function defined as $\pi_i((a_1,a_2,a_3,\cdots))=a_i$.
|
|
|
|
> Note, this representation is isomorphic to the dyadic rationals (i.e., numbers that can be written as a fraction whose denominator is a power of 2) in the interval $[0,1]$.
|
|
|
|
## Section 2: Formal definitions
|
|
|
|
> Recall, the $\sigma$-algebra (denoted as $\mathcal{A}$ in Math4121) is the collection of all subsets of a set $S$ satisfying the following properties:
|
|
>
|
|
> 1. $\emptyset\in \mathcal{A}$ (empty set is in the $\sigma$-algebra)
|
|
> 2. If $A\in \mathcal{A}$, then $A^c\in \mathcal{A}$ (if a set is in the $\sigma$-algebra, then its complement is in the $\sigma$-algebra)
|
|
> 3. If $A_1,A_2,A_3,\cdots\in \mathcal{A}$, then $\bigcup_{i=1}^{\infty}A_i\in \mathcal{A}$ (if a countable sequence of sets is in the $\sigma$-algebra, then their union is in the $\sigma$-algebra)
|
|
|
|
### Event, probability, and random variable
|
|
|
|
Let $\Omega$ be a non-empty set.
|
|
|
|
Let $\mathscr{F}$ be a $\sigma$-algebra on $\Omega$ (Note, $\mathscr{F}$ is a collection of subsets of $\Omega$ that satisfies the properties of a $\sigma$-algebra).
|
|
|
|
#### Definition of Event
|
|
|
|
An event is a element of $\mathscr{F}$.
|
|
|
|
#### Definition of Probability Measure
|
|
|
|
A probability measure $P$ is a function $P:\mathscr{F}\to [0,1]$ satisfying the following properties:
|
|
|
|
1. $P(\Omega)=1$
|
|
2. If $A_1,A_2,A_3,\cdots\in \mathscr{F}$ are pairwise disjoint ($\forall i\neq j, A_i\cap A_j=\emptyset$), then $P(\bigcup_{i=1}^{\infty}A_i)=\sum_{i=1}^{\infty}P(A_i)$
|
|
|
|
#### Definition of Probability Space
|
|
|
|
A probability space is a triple $(\Omega, \mathscr{F}, P)$ defined above.
|
|
|
|
An event $A$ is said to occur almost surely (a.s.) if $P(A)=1$.
|
|
|
|
#### Definition of Random Variable
|
|
|
|
A random variable is a function $X:\Omega\to \mathbb{R}$ that is measurable with respect to the $\sigma$-algebra $\mathscr{F}$.
|
|
|
|
That is, for any Borel set $B\subset \mathbb{R}$, the preimage $f^{-1}(B)\in \mathscr{F}$.
|
|
|
|
$$
|
|
f^{-1}(B)=\{x\in \Omega: f(x)\in B\}\in \mathscr{F}
|
|
$$
|
|
|
|
#### Definition of sigma-algebra generated by a random variable
|
|
|
|
Let $\{f_\alpha:\Omega\to \mathbb{R},\alpha\in I\}$ be a family of functions where $I$ is an index set which is not necessarily finite or countable. The $\sigma$-algebra generated by the family of functions $\{f_\alpha:\alpha\in I\}$, denoted as $\sigma\{f_\alpha:\alpha\in I\}$, is the smallest $\sigma$-algebra containing all the subsets of $\Omega$ of the form
|
|
|
|
$$
|
|
f_\alpha^{-1}(B)=\{\omega\in \Omega: f_\alpha(\omega)\in B\}\in \mathscr{F}
|
|
$$
|
|
|
|
for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.
|
|
|
|
Equivalently,
|
|
|
|
$$
|
|
\sigma\{f_\alpha:\alpha\in I\}=\sigma\left(\bigcup_{\alpha\in I}f_\alpha^{-1}(B)\right)
|
|
$$
|
|
|
|
the sigma-algebra generated by a random variable $X$ is the intersection of all $\sigma$-algebras on $\Omega$ containing the sets $f_\alpha^{-1}(B)$ for all $\alpha\in I$ and $B\in \mathscr{B}(\mathbb{R})$.
|
|
|
|
#### Definition of distribution of random variable
|
|
|
|
Let $f:\Omega\to \mathbb{R}$ be a random variable. The distribution of $f$ is the probability measure $P_f$ on $\mathbb{R}$ defined by
|
|
|
|
$$
|
|
P_f(B)=P(f^{-1}(B))=P(\{x\in \Omega: f(x)\in B\})
|
|
$$
|
|
|
|
also noted as $f_*P$.
|
|
|
|
#### Definition of joint distribution of random variables
|
|
|
|
Let $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ be random variables. The joint distribution of $f_1,f_2,\cdots,f_n$ is the probability measure $P_{f_1,f_2,\cdots,f_n}$ on $\mathbb{R}^n$ defined by
|
|
|
|
$$
|
|
P_{f_1,f_2,\cdots,f_n}(B)=P(f_1^{-1}(B_1)\cap f_2^{-1}(B_2)\cap \cdots \cap f_n^{-1}(B_n))=P(\omega\in \Omega: (f_1(\omega),f_2(\omega),\cdots,f_n(\omega))\in B)
|
|
$$
|
|
|
|
### Expectation of a random variable
|
|
|
|
Let $f:\Omega\to \mathbb{R}$ be a random variable. The expectation of $f$ is defined as
|
|
|
|
$$
|
|
\mathbb{E}[f]=\mathbb{E}[f(X)]=\int_\Omega f(x)dP
|
|
$$
|
|
|
|
Note, $P$ is the probability measure on $\Omega$.
|
|
|
|
#### Definition of variance
|
|
|
|
The variance of a random variable $f$ is defined as
|
|
|
|
$$
|
|
\operatorname{Var}(f)=\mathbb{E}[(f-\mathbb{E}[f])^2]=\mathbb{E}[f^2]-(\mathbb{E}[f])^2
|
|
$$
|
|
|
|
#### Definition of covariance
|
|
|
|
The covariance of two random variables $f,g:\Omega\to \mathbb{R}$ is defined as
|
|
|
|
$$
|
|
\operatorname{Cov}(f,g)=\mathbb{E}[(f-\mathbb{E}[f])(g-\mathbb{E}[g])]
|
|
$$
|
|
|
|
### Point measures
|
|
|
|
#### Definition of Dirac measure
|
|
|
|
The Dirac measure is a probability measure on $\Omega$ defined as
|
|
|
|
$$
|
|
\delta_\omega(A)=\begin{cases}
|
|
1 & \text{if } \omega\in A \\
|
|
0 & \text{if } \omega\notin A
|
|
\end{cases}
|
|
$$
|
|
|
|
Note that $\int_\Omega f(x)d\delta_\omega(x)=f(\omega)$.
|
|
|
|
### Infinite sequence of independent coin flips
|
|
|
|
> Side notes from basic topology:
|
|
>
|
|
> **Definition of product topology**:
|
|
>
|
|
> It is a set constructed by the Cartesian product of the sets. Suppose $X_i$ is a set for all $i\in I$. The element of the product set is a tuple $(x_i)_{i\in I}$ where $x_i\in X_i$ for all $i\in I$.
|
|
>
|
|
> For example, if $X_i=[0,1]$ for all $i\in \mathbb{N}$, then the product set is $[0,1]^{\mathbb{N}}$. An element of such product set is $(1,0.5,0.25,\cdots)$.
|
|
|
|
The set of outcomes of such infinite sequence of coin flips is the product set of the set of outcomes of each coin flip.
|
|
|
|
$$
|
|
S=\{0,1\}^{\mathbb{N}}
|
|
$$
|
|
|
|
### Conditional probability
|
|
|
|
#### Definition of conditional probability
|
|
|
|
The conditional probability of an event $A$ given an event $B$ is defined as
|
|
|
|
$$
|
|
P(A|B)=\frac{P(A\cap B)}{P(B)}
|
|
$$
|
|
|
|
The law of total probability:
|
|
|
|
$$
|
|
P(A)=\sum_{i=1}^{\infty}P(A|B_i)P(B_i)
|
|
$$
|
|
|
|
Bayes' theorem:
|
|
|
|
$$
|
|
P(B_i|A)=\frac{P(A|B_i)P(B_i)}{\sum_{j=1}^{\infty}P(A|B_j)P(B_j)}
|
|
$$
|
|
|
|
#### Definition of independence of random variables
|
|
|
|
Two random variables $f,g:\Omega\to \mathbb{R}$ are independent if for any Borel sets $A,B\subset \mathscr{B}(\mathbb{R})$ the events
|
|
|
|
$$
|
|
\{\omega\in \Omega: f(\omega)\in A\}\text{ and } \{\omega\in \Omega: g(\omega)\in B\}
|
|
$$
|
|
|
|
are independent.
|
|
|
|
In general, a finite or infinite family of random variables $f_1,f_2,\cdots,f_n:\Omega\to \mathbb{R}$ are independent if every finite collection of random variables from this family are independent.
|
|
|
|
#### Definition of independence of sigma-algebras
|
|
|
|
Let $\mathscr{G}$ and $\mathscr{H}$ be two $\sigma$-algebras on $\Omega$. They are independent if for any Borel sets $A\subset \mathscr{B}(\mathbb{R})$ and $B\subset \mathscr{B}(\mathbb{R})$, the finite collection of events are independent.
|
|
|
|
## Section 3: Further definitions in measure theory and integration
|
|
|
|
### $L^2$ space
|
|
|
|
#### Definition of $L^2$ space
|
|
|
|
Let $(\Omega, \mathscr{F}, P)$ be a measure space. The $L^2$ space is the space of all square integrable, complex-valued measurable functions on $\Omega$.
|
|
|
|
Denoted by $L^2(\Omega, \mathscr{F}, P)$.
|
|
|
|
The square integrable functions are the functions $f:\Omega\to \mathbb{C}$ such that
|
|
|
|
$$
|
|
\int_\Omega |f(\omega)|^2 dP(\omega)<\infty
|
|
$$
|
|
|
|
With inner product defined by
|
|
|
|
$$
|
|
\langle f,g\rangle=\int_\Omega \overline{f(\omega)}g(\omega)dP(\omega)
|
|
$$
|
|
|
|
The $L^2(\Omega, \mathscr{F}, P)$ space is a Hilbert space. |