From 2d76afaf555755e82c5edbf33106ccbd9ca13e04 Mon Sep 17 00:00:00 2001 From: Trance-0 <60459821+Trance-0@users.noreply.github.com> Date: Sat, 25 Jan 2025 16:26:48 -0600 Subject: [PATCH] update --- pages/CSE559A/CSE559A_L4.md | 196 +++++++++++++++++++++++++++++++ public/CSE559A/8x8_DCT_basis.png | Bin 0 -> 7085 bytes 2 files changed, 196 insertions(+) create mode 100644 public/CSE559A/8x8_DCT_basis.png diff --git a/pages/CSE559A/CSE559A_L4.md b/pages/CSE559A/CSE559A_L4.md index e69de29..69567c4 100644 --- a/pages/CSE559A/CSE559A_L4.md +++ b/pages/CSE559A/CSE559A_L4.md @@ -0,0 +1,196 @@ +# Lecture 4 + +## Practical issues with filtering + +$$ +h[m,n]=\sum_{k=0}^{m-i}\sum_{l=0}^{n-i}g[k,l]f[m+k,n+l] +$$ + +Loss of information on edges of image + +- The filter window falls off the edge of the image +- Need to extrapolate +- Methods: + - clip filter + - wrap around (extend the image periodically) + - copy edge (extend the image by copying the edge pixels) + - reflect across edge (extend the image by reflecting the edge pixels) + +## Convolution vs Correlation + +- Convolution: + - The filter is flipped and convolved with the image + +$$ +h[m,n]=\sum_{k=i}^{m}\sum_{l=i}^{n}g[k,l]f[m-k,n-l] +$$ + +- Correlation: + - The filter is not flipped and convolved with the image + +$$ +h[m,n]=\sum_{k=0}^{m-i}\sum_{l=0}^{n-i}g[k,l]f[m+k,n+l] +$$ + +does not matter for deep learning + +```python +scipy.signal.convolve2d(image, kernel, mode='same') +scipy.signal.correlate2d(image, kernel, mode='same') +``` + +but pytorch uses correlation for convolution, the convolution in pytorch is actually a correlation in scipy. + +## Frequency domain representation of linear image filters + +TL;DR: It can be helpful to think about linear spatial filters in terms fro their frequency domain representation + +- Fourier transform and frequency domain +- The convolution theorem + +Hybrid image: More in homework 2 + +Human eye is sensitive to low frequencies in far field, high frequencies in near field + +### Change of basis from an image perspective + +For vectors: + +- Vector -> Invertible matrix multiplication -> New vector +- Normally we think of the standard/natural basis, with unit vectors in the direction of the axes + +For images: + +- Image -> Vector -> Invertible matrix multiplication -> New vector -> New image +- Standard basis is just a collection of one-hot images + +Use `im.flatten()` to convert an image to a vector + +$$ +Image(M^{-1}GMVec(I)) +$$ + +- M is the change of basis matrix, $M^{-1}M=I$ +- G is the operation we want to perform +- Vec(I) is the vectorized image + +#### Lossy image compression (JPEG) + +- JPEG is a lossy compression algorithm +- It uses the DCT (Discrete Cosine Transform) to transform the image to the frequency domain +- The DCT is a linear operation, so it can be represented as a matrix multiplication +- The JPEG algorithm then quantizes the coefficients and entropy codes them (use Huffman coding) + +## Thinking in frequency domain + +### Fourier transform + +Any univariate function can be represented as a weighted sum of sine and cosine functions + +$$ +X[k]=\sum_{n=N-1}^{0}x[n]e^{-2\pi ikn/N}=\sum_{n=0}^{N-1}x[n]\left[\sin\left(\frac{2\pi}{N}kn\right)+i\cos\left(\frac{2\pi}{N}kn\right)\right] +$$ + +- $X[k]$ is the Fourier transform of $x[n]$ +- $e^{-2\pi ikn/N}$ is the basis function +- $x[n]$ is the original function + +Real part: + +$$ +\text{Re}(X[k])=\sum_{n=0}^{N-1}x[n]\cos\left(\frac{2\pi}{N}kn\right) +$$ + +Imaginary part: + +$$ +\text{Im}(X[k])=\sum_{n=0}^{N-1}x[n]\sin\left(\frac{2\pi}{N}kn\right) +$$ + +Fourier transform stores the magnitude and phase of the sine and cosine function at each frequency + +- Amplitude: encodes how much signal there is at a particular frequency +- Phase: encodes the spacial information (indirectly) +- For mathematical convenience, this is often written as a complex number + +Amplitude: $A=\pm\sqrt{\text{Re}(\omega)^2+\text{Im}(\omega)^2}$ + +Phase: $\phi=\tan^{-1}\left(\frac{\text{Im}(\omega)}{\text{Re}(\omega)}\right)$ + +So use $A\sin(\omega+\phi)$ to represent the signal + +Example: + +$g(t)=\sin(2\pi ft)+\frac{1}{3}\sin(2\pi (3f)t)$ + +### Fourier analysis of images + +Intensity image and Fourier image + +Signals can be composed. + +![jpeg basis](https://static.notenextra.trance-0.com/CSE559A/8x8_DCT_basis.png) + +Note: frequency domain is often visualized using a log of the absolute value of the Fourier transform + +Blurring the image is to delete the high frequency components (removing the center of the frequency domain) + +## Convolution theorem + +The Fourier transform of the convolution of two functions is the product of their Fourier transforms + +$$ +F[f*g]=F[f]F[g] +$$ + +- $F$ is the Fourier transform +- $*$ is the convolution + +Convolution in spatial domain is equivalent to multiplication in frequency domain + +$$ +g*h=F^{-1}[F[g]F[h]] +$$ + +- $F^{-1}$ is the inverse Fourier transform + +### Is convolution invertible? + +- Redo the convolution in the image domain is division in the frequency domain + +$$ +g*h=F^{-1}\left[\frac{F[g]}{F[h]}\right] +$$ + +- This is not always possible, because $F[h]$ may be zero and we may not know the filter + +Small perturbations in the frequency domain can cause large perturbations in the spatial domain and vice versa + +Deconvolution is hard and a active area of research + +- Even if you know the filter, it is not always possible to invert the convolution, requires strong regularization +- If you don't know the filter, it is even harder + +## 2D image transformations + +### Array slicing and image wrapping + +Fast operation for extracting a subimage + +- cropped image `image[10:20, 10:20]` +- flipped image `image[::-1, ::-1]` + +Image wrapping allows more flexible operations + +#### Upsampling an image + +- Upsampling an image is the process of increasing the resolution of the image + +Bilinear interpolation: + +- Use the average of the 4 nearest pixels to determine the value of the new pixel + +Other interpolation methods: + +- Bicubic interpolation: Use the average of the 16 nearest pixels to determine the value of the new pixel +- Nearest neighbor interpolation: Use the value of the nearest pixel to determine the value of the new pixel diff --git a/public/CSE559A/8x8_DCT_basis.png b/public/CSE559A/8x8_DCT_basis.png new file mode 100644 index 0000000000000000000000000000000000000000..d50ea424a1821d1a6e6fc61999b554346015efd7 GIT binary patch literal 7085 zcmdT}dsvd^->>;@?pJx-!JH*oGe?%#EYC`%*2LwhgPMt%D>F4CGes2E+7i=p70N8Z ztRA)URGtlF6=`;WLc>CXg@-33KlwxS zwe;<$o}ID%;pY%0dpL5X&HY7+TZqcO!hrLHsgwy#eMFyPoT~3>1iKqeYQqtdcNP|0 z&pfg!xr6r{Ah{OYR9)kN7kYN-HcOTQsItKAc`1@B8Nvh%NS%u6-Y01RSm~pWT@$wJ z?(;@ZvG0t^o~{0{wXPBu&n6m2HWc6UvDxC{rQt~pIUrv^VN6D z|DSweYxCDS8&wliXqLWHmn886BFY1I=r#~cyCnob(*}M4c$Ec?OuS+qjgkm~^Sm~| zu3U)iE|63Jx69quFgsPk=oyggX<8%0*U_N3xV&_Un0F#fg9^B$V(_kDGTjoec?SuB z*Pf8FC4m674HUKFGFQK9IvOG+ZAG%AhUgW~W;?>~jh(5NEEmMTq6G2$E@Yn)wX4%}idq{{<(%{)9HxMR`kT)Yl3KGs&_I zthNS`Ujx&G<=@fi4E0s&9A; zXj*$#(`f(9W1ZfgIP0wZ=D%K;vf15I@rig*rQLAdQ7Mm<4*hXwhW-x24RUf+9K2}f zU3j12*Qy{7FoyX)O{ee4Cl%E_naQ%peZIeyW`B#4j>})7{?HtupN1xJb>{J`n0P;> zN1$|1!DDWBELv*DnsrjYr#r|!R@9k-7$?0gaAuVceXeYcN58@QGtt&A%d6g$9mfSG zbf+K{*2Xj0nL7ldJ^>LmtPd}I^9HLb&KHh) zvCHZbRlB5HmH?p0oxAftL#FS~6LKET^bCcFHMbB!*@T_&Gb6j3JlOeutttHbx}3Dp zcOTEj3VV5@W)2G!W$U*RXa}ZAWRt2JU0J+qK%mE8Jsr;?CN7p&bch@_6IfpLeZY&V z2+Q{)Mlc0HyB*-UCbD7?wIVyu)>4&Jb89V0WYwEw+`8*__T_y$pZ(*Oga2NeD~QM? zdmcqkZ+Q^(T6kotL0oC0Chcr9x(Ynh2ffj1&{H0Rxt5}Ai=Ivtu7*M-x1u?rw7Bjp zq&kxrr7MJy{?5~s{Q>E(8ACY0g&IvN)NL>?H(pXX0E1F(fh10*Re73106i7^luo5Y zC%)0~%rMjru&Qj@w1+qc)*UYZ$H}9Ig>@nX7r%Ux8)+(3ZZ-rcs_Zb=MBk63TxZsb zm<>>mr+g|ySWTGHeB*4%HooQc%JrW7{{S94a{~H^h|=GYPicBZ!#d_uK}D4!&Rm*I zgl{9jZe`1}Q=f=CfsMXFr_la~~uhW_P#KVDU zNm300sI-a~2>MTmO*p)yG%R8bv!3Fez>Sz*_@p2X3gq9(uRo8EL+Sg8^I19d zMs6&$Y$7Ee=HlO~vTqXk=5vsus@qfg$Qc+clVo2mx*b_Mah%T~{EJ){k{D9E25lUM&-a>G-;@JTyy5R$^Hf+m~o~6 zKueR2@fdA@0sZ?Nv=Gp%fBfmBTtyfy(nOY4`1(M?++}nR_%-W^*4f0xg?WC1&c2r^ zgwg(D2|nP%i%!beIk9~`88oommbw?+ls=l@bM((woo;?dIxIKXXBh7TBRLVfEex}b z!ScV?w+~e6@vX2)N_P1q+cm>8;&p=#&R zFsyBuKx?qd+b(+xY^ zcjav)pgYv&lh!s{#0Gn5Qe}jfOnoX>cwL3SuwM37yIHAxeS&25;3>X*oHryOh*P;8 zn!XO4XNEFT^Qm`XpNNIU7>cv@kwiNAu7`O4$snaB3&YwmX*ZjdB8N6*GO9;7KfafW z6_K^MSb}KO?-8b}=~Ui*LI-gqEyT#V*FZB=$*D8l&&D44C*(yU@$1>gUm&J``GX7# zNO;cx5=fz*%Lz7GWLGz#pb&>bbz(wygk)a>g2ZB|uJVn0{ZMP z7WNOq+^XzNOOo}`(>b0fU}`PiYs{S4_mLoX5Vw-`3yV;%CbX$XkTRa_#mKdsCLeGO8u;Mc^D8P)#cWr9bC? zJ~_`u^g7UeI^4T3WaM?*jB&!#GW%!mDmHVcJ0+zx9;wcqa&tx$=Jpz947L2yJpZ>= z_$|$K0ak4-=rNZlD)=K0MLi;vhGi6}K%-Vfxs^C3T{OPptR~pe3Y0gx7F<@niJn3v z@18EPL_uLl@X@SeY`az~UTqPaU_UwYE2un#R4fs17;JMo^k0J+Q<^8}0bLZD%qXYD z`*B%r1>J3*nUY?yX^xpt&kKQGf6cz#10h}JcF(thN_ZU?!!>^|tOm%nLBwegec3=u znJ#<7{lFSVbUSX24KROTaDA)KSmIZ)Gs)%!vAT_t+h!04mB15fe4n~+-*Q3i&KU;A zoNY|M4bx2mijGs_G0^>#NmK70(%HmR3<$I6Q!TOptyhW1cJr@JPJXv7QzziHHK6e^ zV{y3f5*CR?(R1eb`L68pPr5rP?-nSE@G`u7h7pRyEt_0!^=hG2vgTn{c(F3BGV8L? z0&ISe<8zfl5EPl+M0+q@h;=mO{*h#Ez+0x}dd;nts+cxh9!*L_`Io$MjbIeBKzg1o zi)FaQ=it}>d$xHjBLB4>d^6zMr?3BP^4lgiC>{wm*IiWZ-Zn)OGwiYtVpln{3iB(r z3^Q(1&Q)zPJ}n2h<;Gj5gNU~HO@j(2s|kM1;Kh}sPkRRFg@SAP@0qDn#VJMz(~l7Q ze~AxGp&U6=oHsD`FI;%~H;AIFJucL98aBRzfU%=G#$$*)957~fvve%XzNmbbK(zPB z>)a~)JJ5C}QR%t0XgGa`uAlb=QxvcAjzFh89}R{KpF^fDO6Mn<+gXQ7yuVOdM>tsq z=cm-*s;ZAoBrmPBN~S#~#31!}EPSv;l%mQJtT-qy6bG|M$y;t(^7{;x1NtaQPsWnj z9q*i>E!VAnFe?f1y31V~`%fD+cdVwn~&hQ=7)C1NUG>-X&GPr`!@42VGQH1XN+lp2$c}4`wpo%TN`Gz%GtQKmIB$IYWeYR3QPZwc^qT7GTTB7Zn;gEUt5i zR6lXFU_bi~kXdF+ihW5ZygDHEKO8G50K&o$2cf!!Tj^HX@7M`nQjVbg@LqmpwlOce zDEW?)%YvFl7l{sVlVz1AF~c7BIX@I@3Enva|Mt3HBSN6G$@r?5y0dcjJFxYbID)jA z`Chjfu5i?wE7ArKZ&tre#LzA9tCRxePcR=zD&bV)&t8A2uZWW&WAE^e>Fv7Fhq>0L z%tdMr*jNeb;p^!xO*?qQ>cWOx;w{Vf(=5o5-S~6aM;>LG{v@Q4|a0u zaAJV^H+14dd_7%$*Eq8NpE3!XNX-!Nc4%9J=y-#O!RHuCxW2InWs4i1tD9pR12^u? ze{iwd(pq6Z`FPPQeMsN}P4TNCdg0`Tdyz|1C%1prw5;^a3{Mu;U;46*;(xI?tjnte zsz^eFg)!S}!&%e~a`k2o==W%cT7(I3Ga-*fD2#1wYJP``#>6k`1fINRKbf0=9 z7V4a36~xMW%T2#PPHrC0g%MAhY1&xxY%<($oU6J=jFyvgEm#^|^h_~ZBnZfOl@37jUuE(9Li zw{=&Z_;@a372)hN-#8NZVL}^Fb|_WG>}frFV7B6{#;!yY=Q3zh6KJI&;--pqe(W^w z%(et6SL-geV~2j(uGuhh!n?0(SWj97IC7r>BX*l#T}r`pp`gStbkswmwb${{VPzzo z0J_#p5lkAM=@3*WeT-kt$;KU;x((Qs*=o8kJt6HxDw+qviZY*7K@gSahtZ^R`^s#G zmea>(JFau&(9*8rzMe;=5O2UybwuA2LSTiYfpomsN|XWVx=@RKy*5{#z&+R^ z0Mw(qpTj8?WB3`WkoYviBW!4<=J6%LQMb!J_>n&(`pD z?g;6n@O9F+*_U5Nh&|ZH-!X&Kq#A*~+S}%@lRy2u=#$w+B{wtwnx>cZv!?e^-#?}O z@6fg!PHpGjTXBQ!bixCr6Y9-9o)t#4^v#emA(vI7HlK1Bb^LZ}X>ve`@|dI;2+~lY zHYE6XaiRa>7d5a8w;C6vC-~MfMGV!S&huVC%HzM3XKW)5d)jcLt-*927_(CQU)O|1 ze6xP`kTyn-`--d#X2+0I(q9Ql)f6}=f4tn1-F0m#rfADy?1IijnldV){TL?|7kEnl zv?^1U(@GrdcY^Db!{G}K9fYMPf?{j8*N85+1-n?P33$k*|0dM=9sNPtC)#EL%lYAk zLQ@~9j4tQ}8m+TibJFK(xpXfN|CkY0XWQln&>Qctng$l<>p-NA&kNo8Qxi*wdID~rDL+P~E z2+rrgYNQQ?i-+EV11Y2j#mgI8J5_%~Pah?VDr54tBNYhoy4rTo5$x)2?JLTyp?@|i zjKvsI|E%1Lv-0O?T$Xsjl{@N^rL}g$pxVgVfS<8u(E0wnv--BEseLka16G*!Gq&Y;mUEID~2`5-(m{G5FELYOW#!c@(Rn3 z$QP?`l)8-@z8bY<`7%@}U@&I(K8!uaQ3QJYK4J$gqfLZl~zPlavxSN)cp_d(}MVgFySh4#aWzZf!!h6oNh`9IV z&ZHt}Qu6`@(>8Nv#Kd{`f%~OwDkNLPNrX8#nFw G>AwLj_9JZo literal 0 HcmV?d00001