Tweet
Tweet

Slide 1

Slide 1 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#: 461&33&40-65*0/ '30."4*/(-&*."(& %BOJFM(MBTOFS 4IBJ#BHPO .JDIBM*SBOJ5IF8FJ[NBOO*OTUJUVUFPG4DJFODF *OUFSOBUJPOBM$POGFSFODFPO$PNQVUFS7JTJPO *$$7 QQ  4IBP$IVOH$IFO 7JEFP*NBHF1SPDFTTJOH-BC /5/6 (SPVQ.FFUJOH .BZ   #JDVCJD*OUFSQPMBUJPO 6OJpFE4JOHMF*NBHF43 @ʍA º º

Slide 2

Slide 2 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJUIUUQXXXQIEDPNJDTDPNDPNJDTBSDIJWFQIQ DPNJDJE *'574$*&/$&8"4.03&-*,&3&"-4$*&/$& Super-Resolution

Slide 3

Slide 3 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJUIUUQXXXQIEDPNJDTDPNDPNJDTBSDIJWFQIQ DPNJDJE DMBTTJDBMNVMUJJNBHF FYBNQMFCBTFE XJUIEBUBCBTF *'574$*&/$&8"4.03&-*,&3&"-4$*&/$& Super-Resolution

Slide 4

Slide 4 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical Multi-Image

Slide 5

Slide 5 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene Classical Multi-Image

Slide 6

Slide 6 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments Classical Multi-Image

Slide 7

Slide 7 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints Classical Multi-Image

Slide 8

Slide 8 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values Classical Multi-Image

Slide 9

Slide 9 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available Classical Multi-Image

Slide 10

Slide 10 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved Classical Multi-Image

Slide 11

Slide 11 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved •Limited only to little increase in resolution Classical Multi-Image

Slide 12

Slide 12 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved •Limited only to little increase in resolution •By factors smaller than 2 Classical Multi-Image

Slide 13

Slide 13 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based

Slide 14

Slide 14 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” Example-based

Slide 15

Slide 15 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images Example-based

Slide 16

Slide 16 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs Example-based

Slide 17

Slide 17 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 Example-based

Slide 18

Slide 18 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image Example-based

Slide 19

Slide 19 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version Example-based

Slide 20

Slide 20 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version •Exceeds the limits of classical SR Example-based

Slide 21

Slide 21 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low & high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version •Exceeds the limits of classical SR •But not guaranteed to provide the true (unknown) details Example-based

Slide 22

Slide 22 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR

Slide 23

Slide 23 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods

Slide 24

Slide 24 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

Slide 25

Slide 25 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness

Slide 26

Slide 26 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution

Slide 27

Slide 27 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details

Slide 28

Slide 28 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image

Slide 29

Slide 29 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency

Slide 30

Slide 30 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image)

Slide 31

Slide 31 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form)

Slide 32

Slide 32 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form) •Example-based

Slide 33

Slide 33 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form) •Example-based •High-resolution information are available in patches database, and learned from low-res/high-res pairs

Slide 34

Slide 34 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  This Paper

Slide 35

Slide 35 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  proposed a VOJpFEGSBNFXPSL This Paper

Slide 36

Slide 36 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  proposed a VOJpFEGSBNFXPSL for combining those UXP methods This Paper

Slide 37

Slide 37 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  proposed a VOJpFEGSBNFXPSL for combining those UXP methods to obtain super resolution from a TJOHMF image This Paper

Slide 38

Slide 38 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  proposed a VOJpFEGSBNFXPSL for combining those UXP methods with OPEBUBCBTFPSQSJPSFYBNQMFT to obtain super resolution from a TJOHMF image This Paper

Slide 39

Slide 39 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation

Slide 40

Slide 40 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image

Slide 41

Slide 41 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image

Slide 42

Slide 42 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale

Slide 43

Slide 43 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale •Classical SR constraints

Slide 44

Slide 44 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser)

Slide 45

Slide 45 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based

Slide 46

Slide 46 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based •Image Completion, Image Re-targeting, Image Denoising

Slide 47

Slide 47 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based •Image Completion, Image Re-targeting, Image Denoising •Statistically proved later

Slide 48

Slide 48 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz Patch Recurrence

Slide 49

Slide 49 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches

Slide 50

Slide 50 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels

Slide 51

Slide 51 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure

Slide 52

Slide 52 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc

Slide 53

Slide 53 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS)

Slide 54

Slide 54 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale

Slide 55

Slide 55 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s

Slide 56

Slide 56 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I

Slide 57

Slide 57 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is}

Slide 58

Slide 58 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale)

Slide 59

Slide 59 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD

Slide 60

Slide 60 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors

Slide 61

Slide 61 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors •“Good Distance”: SSD with misaligned (0.5px) duplicate

Slide 62

Slide 62 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors •“Good Distance”: SSD with misaligned (0.5px) duplicate •Below “Good Distance” means similar

Slide 63

Slide 63 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 64

Slide 64 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25% source patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 65

Slide 65 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25% source patches •with the highest intensity variance *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 66

Slide 66 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25% source patches •with the highest intensity variance •Excludes the uniform and low-frequency patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 67

Slide 67 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25% source patches •with the highest intensity variance •Excludes the uniform and low-frequency patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 68

Slide 68 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part

Slide 69

Slide 69 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given

Slide 70

Slide 70 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments)

Slide 71

Slide 71 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj

Slide 72

Slide 72 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate

Slide 73

Slide 73 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF)

Slide 74

Slide 74 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q)

Slide 75

Slide 75 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •on the unknown high-res intensity values {H(Qi} within the neighborhood around its corresponding high-res pixel q ∈ H

Slide 76

Slide 76 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •on the unknown high-res intensity values {H(Qi} within the neighborhood around its corresponding high-res pixel q ∈ H •size determined by the support of blur kernel B

Slide 77

Slide 77 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.)

Slide 78

Slide 78 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S

Slide 79

Slide 79 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined

Slide 80

Slide 80 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H

Slide 81

Slide 81 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5)

Slide 82

Slide 82 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts)

Slide 83

Slide 83 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images

Slide 84

Slide 84 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course

Slide 85

Slide 85 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P

Slide 86

Slide 86 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P •for increased numerical stability

Slide 87

Slide 87 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P •for increased numerical stability •patches of high similarity to P have stronger influence

Slide 88

Slide 88 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 89

Slide 89 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 90

Slide 90 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 91

Slide 91 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 92

Slide 92 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 93

Slide 93 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 94

Slide 94 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 95

Slide 95 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 96

Slide 96 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H •scale each equation by its reliability (patch similarity) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 97

Slide 97 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀ pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H •scale each equation by its reliability (patch similarity) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 98

Slide 98 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 99

Slide 99 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part

Slide 100

Slide 100 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S

Slide 101

Slide 101 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H

Slide 102

Slide 102 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn

Slide 103

Slide 103 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function

Slide 104

Slide 104 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj

Slide 105

Slide 105 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known

Slide 106

Slide 106 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj )

Slide 107

Slide 107 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj ) •scale factor sj = αj, the constraint Ij = (H ∗ Bn-1)↓Sn-1 still hold for all j

Slide 108

Slide 108 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H, and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj ) •scale factor sj = αj, the constraint Ij = (H ∗ Bn-1)↓Sn-1 still hold for all j •uniform scale factor guarantees that Ij and Ij+m are related by the same Bm , regardless of j

Slide 109

Slide 109 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.)

Slide 110

Slide 110 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m

Slide 111

Slide 111 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L)

Slide 112

Slide 112 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p

Slide 113

Slide 113 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0

Slide 114

Slide 114 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j

Slide 115

Slide 115 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q]

Slide 116

Slide 116 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ)

Slide 117

Slide 117 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ)

Slide 118

Slide 118 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ) P0(p) find NN P-j(ṕ) parent Q0(sj· ṕ) copy Qj(sj· p)

Slide 119

Slide 119 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 120

Slide 120 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 121

Slide 121 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.)

Slide 122

Slide 122 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L

Slide 123

Slide 123 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H

Slide 124

Slide 124 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR

Slide 125

Slide 125 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q)

Slide 126

Slide 126 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF

Slide 127

Slide 127 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine

Slide 128

Slide 128 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j)

Slide 129

Slide 129 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j) •when solving equations for image Ij+1

Slide 130

Slide 130 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j) •when solving equations for image Ij+1 •employ newly recovered high-res images so far (I0, ..., Ij )

Slide 131

Slide 131 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.)

Slide 132

Slide 132 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images

Slide 133

Slide 133 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from RGB to YIQ

Slide 134

Slide 134 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from RGB to YIQ •Apply SR algorithm to Y (intensity) channel only

Slide 135

Slide 135 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from RGB to YIQ •Apply SR algorithm to Y (intensity) channel only •The I and Q (chromatic) are interpolated (bi-cubic)

Slide 136

Slide 136 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from RGB to YIQ •Apply SR algorithm to Y (intensity) channel only •The I and Q (chromatic) are interpolated (bi-cubic) •Combined to form such SR results

Slide 137

Slide 137 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Efficiency

Slide 138

Slide 138 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Efficiency Takes 6 seconds with 2.1GHz CPU on a PC to up-sample from 1282 to 5122 pixels.

Slide 139

Slide 139 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Efficiency Takes 6 seconds with 2.1GHz CPU on a PC to up-sample from 1282 to 5122 pixels. Fattal et al., “Image Upsampling via Imposed Edges Statistics”, Proc. SIGGRAPH 2007

Slide 140

Slide 140 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Results *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz

Slide 141

Slide 141 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  More on goo.gl/7vXav IUUQXXXXJTEPNXFJ[NBOOBDJMdWJTJPO4JOHMF*NBHF43IUNM

Slide 142

Slide 142 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  #0/64 @ʍA •Comparison with other up-scaling approaches •Applications •Space-Time SR •JPEG Removal •Video Streaming for Mobile Devices

Slide 143

Slide 143 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  A Performance Evaluation of Image Interpolation and Superresolution Algorithms Xin Ye, Xiqun Lu; Zhejiang University International Conference on Multimedia Technology (ICMT), 2011

Slide 144

Slide 144 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU9*O:F 9JRVO-V l"1FSGPSNBODF&WBMVBUJPOPG*NBHF*OUFSQPMBUJPOBOE4VQFSSFTPMVUJPO"MHPSJUINTz

Slide 145

Slide 145 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU9*O:F 9JRVO-V l"1FSGPSNBODF&WBMVBUJPOPG*NBHF*OUFSQPMBUJPOBOE4VQFSSFTPMVUJPO"MHPSJUINTz • TRI: H. S. Hou and H. C. Andrews, “Cubic Splines for Image Interpolation and Digital Filtering,” IEEE Trans. On Acoustics, Speech, and Signal Processing, 1981 • NEDI: X. Li and M. T. Orchard, “New Edge-Directed Interpolation,” IEEE Trans. On Image Processing, 2001 • GPP: J. Sun, J. Sun, Z. B. Xun and H. Y. Shum, “Image Super-resoluton using Gradient Profile Prior,” CVPR, 2008 • IUES: R. Fattal, “Image upsampling via imposed edge statistics,” Proceeding of ACM SIGGRAPH, 2007 • SRSI: D. Glasner, S. Bagon, M. Irani, “Super-Resolution from a Single Image,” ICCV, 2009

Slide 146

Slide 146 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Use PSNR & MSSIM to evaluate reconstructed image quality •PSNR & MSSIM of bilinear & cubic convolution are larger than new-edge directed interpolation & unified SR •Visual evaluation •primary metric •PSNR & MSSIM don’t provide accurate measure of visual quality in such case

Slide 147

Slide 147 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Space-Time Super-Resolution from a Single Video Oded Shahar, Alon Faktor, Michal Irani; The Weizmann Institute of Science Conference of Computer Vision and Pattern Recognition (CVPR), 2011

Slide 148

Slide 148 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU4IBIBSFUBM l4QBDF5JNF4VQFS3FTPMVUJPOGSPNB4JOHMF7JEFPz

Slide 149

Slide 149 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU4IBIBSFUBM l4QBDF5JNF4VQFS3FTPMVUJPOGSPNB4JOHMF7JEFPz

Slide 150

Slide 150 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU4IBIBSFUBM l4QBDF5JNF4VQFS3FTPMVUJPOGSPNB4JOHMF7JEFPz

Slide 151

Slide 151 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Demo on goo.gl/X9JD7 IUUQXXXXJTEPNXFJ[NBOOBDJMdWJTJPO4JOHMF7JEFP43IUNM

Slide 152

Slide 152 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Learning for Single-Image Super-Resolution and JPEG Artifact Removal Kwang In Kim and Younghee Kwon; Max Planck Institute for Biological Cybernetics DAGM Symposium for Pattern Recognition, 2008

Slide 153

Slide 153 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU,*,JNFUBM l&YBNQMFCBTFE-FBSOJOHGPS4JOHMF*NBHF4VQFS3FTPMVUJPOBOE+1&("SUJGBDUT3FNPWBMz

Slide 154

Slide 154 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  MobiUP: An Upsampling-Based System Architecture for High-Quality Video Streaming on Mobile Devices Hong-Han Shuai, De-Nian Yang, Wen-Huang Cheng, Ming-Syan Chen Academia Sinica, National Taiwan University IEEE Transactions on Multimedia (TMM), 2011

Slide 155

Slide 155 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU))4IVBJFUBM l.PCJ61"O6QTBNQMJOH#BTFE4ZTUFN"SDIJUFDUVSFGPS)JHI2VBMJUZ7JEFP4USFBNJOHPO.PCJMF%FWJDFTz 

Slide 156

Slide 156 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU))4IVBJFUBM l.PCJ61"O6QTBNQMJOH#BTFE4ZTUFN"SDIJUFDUVSFGPS)JHI2VBMJUZ7JEFP4USFBNJOHPO.PCJMF%FWJDFTz

Slide 157

Slide 157 text

#Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  '*/5)"/,4