diff --git a/pages/CSE559A/CSE559A_L25.md b/pages/CSE559A/CSE559A_L25.md index e61cb27..bcb6962 100644 --- a/pages/CSE559A/CSE559A_L25.md +++ b/pages/CSE559A/CSE559A_L25.md @@ -135,7 +135,7 @@ $$ ##### Approaches 3: Linear approach -$x_1\simeq P_1X$ and $x_2\simeq P_2X$ +$x_1\cong P_1X$ and $x_2\cong P_2X$ $x_1\times P_1X = 0$ and $x_2\times P_2X = 0$ diff --git a/pages/CSE559A/CSE559A_L26.md b/pages/CSE559A/CSE559A_L26.md index 8b13789..6446035 100644 --- a/pages/CSE559A/CSE559A_L26.md +++ b/pages/CSE559A/CSE559A_L26.md @@ -1 +1,177 @@ +# CSE559A Lecture 26 + +## Continue on Geometry and Multiple Views + +### The Essential and Fundamental Matrices + +#### Math of the epipolar constraint: Calibrated case + +Recall Epipolar Geometry + +![Epipolar Geometry Configuration](https://notenextra.trance-0.com/CSE559A/Epipolar_geometry_setup.png) + +Epipolar constraint: + +If we set the config for the first camera as the world origin and $[I|0]\begin{pmatrix}y\\1\end{pmatrix}=x$, and $[R|t]\begin{pmatrix}y\\1\end{pmatrix}=x'$, then + +Notice that $x'\cdot [t\times (Ry)]=0$ + +$$ +x'^T E x_1 = 0 +$$ + +We denote the constraint defined by the Essential Matrix as $E$. + +$E x$ is the epipolar line associated with $x$ ($l'=Ex$) + +$E^T x'$ is the epipolar line associated with $x'$ ($l=E^T x'$) + +$E e=0$ and $E^T e'=0$ ($x$ and $x'$ don't matter) + +$E$ is singular (rank 2) and have five degrees of freedom. + +#### Epipolar constraint: Uncalibrated case + +If the calibration matrices $K$ and $K'$ are unknown, we can write the epipolar constraint in terms of unknown normalized coordinates: + +$$ +x'^T_{norm} E x_{norm} = 0 +$$ + +where $x_{norm}=K^{-1} x$, $x'_{norm}=K'^{-1} x'$ + +$$ +x'^T_{norm} E x_{norm} = 0\implies x'^T_{norm} Fx=0 +$$ + +where $F=K'^{-1}EK^{-1}$ is the **Fundamental Matrix**. + +$$ +(x',y',1)\begin{bmatrix} +f_{11} & f_{12} & f_{13} \\ +f_{21} & f_{22} & f_{23} \\ +f_{31} & f_{32} & f_{33} +\end{bmatrix}\begin{pmatrix} +x\\y\\1 +\end{pmatrix}=0 +$$ + +Properties of $F$: + +$F x$ is the epipolar line associated with $x$ ($l'=F x$) + +$F^T x'$ is the epipolar line associated with $x'$ ($l=F^T x'$) + +$F e=0$ and $F^T e'=0$ + +$F$ is singular (rank two) and has seven degrees of freedom + +#### Estimating the fundamental matrix + +Given: correspondences $x=(x,y,1)^T$ and $x'=(x',y',1)^T$ + +Constraint: $x'^T F x=0$ + +$$ +(x',y',1)\begin{bmatrix} +f_{11} & f_{12} & f_{13} \\ +f_{21} & f_{22} & f_{23} \\ +f_{31} & f_{32} & f_{33} +\end{bmatrix}\begin{pmatrix} +x\\y\\1 +\end{pmatrix}=0 +$$ + +**Each pair of correspondences gives one equation (one constraint)** + +At least 8 pairs of correspondences are needed to solve for the 9 elements of $F$ (The eight point algorithm) + +We know $F$ needs to be singular/rank 2. How do we force it to be singular? + +Solution: take SVD of the initial estimate and throw out the smallest singular value + +$$ +F=U\begin{bmatrix} +\sigma_1 & 0 \\ +0 & \sigma_2 \\ +0 & 0 +\end{bmatrix}V^T +$$ + +## Structure from Motion + +Not always uniquely solvable. + +If we scale the entire scene by some factor $k$ and, at the same time, scale the camera matrices by the factor of $1/k$, the projections of the scene points remain exactly the same: +$x\cong PX =(1/k P)(kX)$ + +Without a reference measurement, it is impossible to recover the absolute scale of the scene! + +In general, if we transform the scene using a transformation $Q$ and apply the inverse transformation to the camera matrices, then the image observations do not change: + +$x\cong PX =(P Q^{-1})(QX)$ + +### Types of Ambiguities + +![Ambiguities in projection](https://notenextra.trance-0.com/CSE559A/Ambiguities_in_projection.png) + +### Affine projection : more general than orthographic + +A general affine projection is a 3D-to-2D linear mapping plus translation: + +$$ +P=\begin{bmatrix} +a_{11} & a_{12} & a_{13} & t_1 \\ +a_{21} & a_{22} & a_{23} & t_2 \\ +0 & 0 & 0 & 1 +\end{bmatrix}=\begin{bmatrix} +A & t \\ +0^T & 1 +\end{bmatrix} +$$ + +In non-homogeneous coordinates: + +$$ +\begin{pmatrix} +x\\y\\1 +\end{pmatrix}=\begin{bmatrix} +a_{11} & a_{12} & a_{13} \\ +a_{21} & a_{22} & a_{23} +\end{bmatrix}\begin{pmatrix} +X\\Y\\Z +\end{pmatrix}+\begin{pmatrix} +t_1\\t_2 +\end{pmatrix}=AX+t +$$ + +### Affine Structure from Motion + +Given: 𝑚 images of 𝑛 fixed 3D points such that + +$$ +x_{ij}=A_iX_j+t_i, \quad i=1,\dots,m, \quad j=1,\dots,n +$$ + +Problem: use the 𝑚𝑛 correspondences $x_{ij}$ to estimate 𝑚 projection matrices $A_i$ and translation vectors $t_i$, and 𝑛 points $X_j$ + +The reconstruction is defined up to an arbitrary affine transformation $Q$ (12 degrees of freedom): + +$$ +\begin{bmatrix} +A & t \\ +0^T & 1 +\end{bmatrix}\rightarrow\begin{bmatrix} +A & t \\ +0^T & 1 +\end{bmatrix}Q^{-1}, \quad \begin{pmatrix}X_j\\1\end{pmatrix}\rightarrow Q\begin{pmatrix}X_j\\1\end{pmatrix} +$$ + +How many constraints and unknowns for $m$ images and $n$ points? + +$2mn$ constraints and $8m + 3n$ unknowns + +To be able to solve this problem, we must have $2mn \geq 8m+3n-12$ (affine ambiguity takes away 12 dof) + +E.g., for two views, we need four point correspondences diff --git a/public/CSE559A/Ambiguities_in_projection.png b/public/CSE559A/Ambiguities_in_projection.png new file mode 100644 index 0000000..87de743 Binary files /dev/null and b/public/CSE559A/Ambiguities_in_projection.png differ