Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Super-Resolution from a Single Image

Super-Resolution from a Single Image

Daniel Glasner, Shai Bagon, Michal Irani, "Super-Resolution from a Single Image", ICCV, 2009.

May 11, 2012, Group Meeting, VIPLab, NTNU.

CC BY-NC 3.0

Dan Chen

May 11, 2012
Tweet

More Decks by Dan Chen

Other Decks in Research

Transcript

  1. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints Classical Multi-Image
  2. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values Classical Multi-Image
  3. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available Classical Multi-Image
  4. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved Classical Multi-Image
  5. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved •Limited only to little increase in resolution Classical Multi-Image
  6. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Set of low-resolution images of same scene •At

    subpixel misalignments •Each imposes a set of linear constraints •On the unknown high-resolution intensity values •Enough low-resolution images available •Set of equations becomes determined & can be solved •Limited only to little increase in resolution •By factors smaller than 2 Classical Multi-Image
  7. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs Example-based
  8. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 Example-based
  9. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image Example-based
  10. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version Example-based
  11. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version •Exceeds the limits of classical SR Example-based
  12. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •A.k.a. “Image Hallucination” •Corresponding patches between low &

    high resolution images •Learned from database of low-res/high-res image pairs •Usually with a relative scale factor of 2 •Applied to a new low-resolution image •Recover its most likely high-resolution version •Exceeds the limits of classical SR •But not guaranteed to provide the true (unknown) details Example-based
  13. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution
  14. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details
  15. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image
  16. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency
  17. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image)
  18. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form)
  19. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form) •Example-based
  20. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Sophisticated vs. SR •Sophisticated methods •Learning edge models

    •Magnify (up-scale) while maintaining the edge sharpness •Super-Resolution •Recover new missing high-resolution details •Not explicitly found in any individual low-resolution image •Details beyond the Nyquist frequency •Classical (multi-image) •High-frequency information are split across multiple low- resolution images (in subpixel aliased form) •Example-based •High-resolution information are available in patches database, and learned from low-res/high-res pairs
  21. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  proposed a VOJpFEGSBNFXPSL for combining those UXP methods

    with OPEBUBCBTFPSQSJPSFYBNQMFT to obtain super resolution from a TJOHMF image This Paper
  22. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale
  23. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale •Classical SR constraints
  24. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser)
  25. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based
  26. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based •Image Completion, Image Re-targeting, Image Denoising
  27. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Observation •Patches in a single natural image •Tend

    to redundantly recur many times inside the image •within the same scale •Classical SR constraints •and across different scales (coarser) •Provides low-res/high-res pairs for Exampled-based •Image Completion, Image Re-targeting, Image Denoising •Statistically proved later
  28. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc
  29. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS)
  30. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale
  31. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s
  32. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I
  33. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is}
  34. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale)
  35. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD
  36. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors
  37. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors •“Good Distance”: SSD with misaligned (0.5px) duplicate
  38. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches •5 x 5 pixels •Cannot visually

    perceive any obvious repetitive structure •vary small patches contain only an edge, a corner, etc •Tested on Berkeley Segmentation Database (BSDS) •Each image I in the database converted to grayscale •Generate down-scaled {Is} from I, as scale factor = 1.25s •s = 0, -1, ..., -6 (I0 = I), smallest one is 1.25-6 = 0.26 of input I •Each patches in I are compared against all patches in {Is} •without their DC (average grayscale) •Distance function: gaussian-weighted SSD •Textured patches have much larger SSD errors •“Good Distance”: SSD with misaligned (0.5px) duplicate •Below “Good Distance” means similar
  39. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25%

    source patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  40. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25%

    source patches •with the highest intensity variance *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  41. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25%

    source patches •with the highest intensity variance •Excludes the uniform and low-frequency patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  42. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Recurring Patches (cont.) •Repeated the experiment using 25%

    source patches •with the highest intensity variance •Excludes the uniform and low-frequency patches *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  43. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments)
  44. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj
  45. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate
  46. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF)
  47. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q)
  48. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •on the unknown high-res intensity values {H(Qi} within the neighborhood around its corresponding high-res pixel q ∈ H
  49. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part •In Classical SR, a set

    of low-res images {L1, ..., Ln} given •of the same scene (at subpixel misalignments) •Lj = (H ∗ Bj)↓Sj •↓ denotes a subsampling operation, Sj is subsampling rate •Bj(q) is the corresponding Point Spread Function (PSF) •Each low-res pixel p = (x, y) in Lj induces the linear constraint Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •on the unknown high-res intensity values {H(Qi} within the neighborhood around its corresponding high-res pixel q ∈ H •size determined by the support of blur kernel B
  50. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H
  51. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5)
  52. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts)
  53. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images
  54. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course
  55. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P
  56. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P •for increased numerical stability
  57. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Only a single low-resolution

    image L = (H ∗ B)↓S •Recovering H becomes under-determined •constraints induced by L are fewer than the unknowns in H •Let p be a pixel in L, and P be its surrounding patch (5 x 5) •Multiple similar patches P1, ..., Pk in L (at subpixel shifts) •Treated as if taken from k different low-res images •of the “same scene”, of course •Each equation induced by Pi is scaled by the degree of similarity of Pi to its patch source P •for increased numerical stability •patches of high similarity to P have stronger influence
  58. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  59. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  60. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  61. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  62. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  63. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  64. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H •scale each equation by its reliability (patch similarity) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  65. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Classical SR Part (cont.) •Summarized as following... •∀

    pixel in L, find its k nearest patch neighbors in same L •eg. Approximate Nearest Neighbor with k = 9 •compute their subpixel alignment (at 1/s pixel shifts) •(where s is the scale factor) •assuming sufficient neighbors are found •determined set of linear constraints on the intensity in H •scale each equation by its reliability (patch similarity) *NBHF$SFEJU(MBTOFSFUBM l4VQFS3FTPMVUJPOGSPNB4JOHMF*NBHFz
  66. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H
  67. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn
  68. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function
  69. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj
  70. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known
  71. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj )
  72. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj ) •scale factor sj = αj, the constraint Ij = (H ∗ Bn-1)↓Sn-1 still hold for all j
  73. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part •Low-res image L, high-res image H,

    and L = (H ∗ B)↓S •I0, I1, ..., In denote a cascade of unknown images of increasing resolutions (scales) ranging from L to H •I0 = L, In = H, with corresponding blur functions B0, ..., Bn •Bn = B is the PSF relating H to L; B0 is the δ function •Ij satisfies L = (Ij ∗Bj)↓Sj •Although {I0, ..., In} are unknown, the cascade blur kernels {B0, ..., Bn} can be assumed to be known •PSF B can be approximated with a gaussian; Bj = B(sj) are a cascade of gaussians (variances determined by sj ) •scale factor sj = αj, the constraint Ij = (H ∗ Bn-1)↓Sn-1 still hold for all j •uniform scale factor guarantees that Ij and Ij+m are related by the same Bm , regardless of j
  74. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m
  75. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L)
  76. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p
  77. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0
  78. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j
  79. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q]
  80. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ)
  81. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ)
  82. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Part (cont.) •I0, I-1, ..., I-m denote

    a cascade of unknown images of decreasing resolutions (scales) ranging L, using I-j = (L ∗Bj)↓Sj where j = 0, ..., m •Unlike high-res cascade, low-res {I-j} are known (computed from L) •Let Pj(p) denote a patch in Ij at pixel location p •For any pixel p ∈ L (L = I0 ) and its surround patch P0(p), we search k nearest similar patches within {I-j} where j > 0 •Let P-j(ṕ) denote such matching patch in I-j , it’s high-res “parent” patch Q0(sj· ṕ) can be extracted from any level between L and I-j •low-res/high-res pair [P, Q] •high-res parent of low-res P0(p) in L is Qj(sj· ṕ) in unknown Ij ; it can be copied from extracted Q0(sj· ṕ) P0(p) find NN P-j(ṕ) parent Q0(sj· ṕ) copy Qj(sj· p)
  83. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR
  84. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q)
  85. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF
  86. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine
  87. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j)
  88. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j) •when solving equations for image Ij+1
  89. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Repeat the Example-based process for all

    pixel in L •large collection of high-res patch {Qj} between L and H •such “learned” high-res patch Qj induces linear constraints on the unknown target resolution H in the form of Classical SR •Lj(p) = (H ∗ Bj) (q) = ∑qi ∈ Support(Bj) H(qi) Bj(qi - q) •with more compactly supported blur kernel than B = PSF •Solving Coarse-to-Fine •use constant factor α = 1.25 (namely, sj = 1.25j) •when solving equations for image Ij+1 •employ newly recovered high-res images so far (I0, ..., Ij )
  90. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from

    RGB to YIQ •Apply SR algorithm to Y (intensity) channel only •The I and Q (chromatic) are interpolated (bi-cubic)
  91. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Combining! (cont.) •Working with color images •Transform from

    RGB to YIQ •Apply SR algorithm to Y (intensity) channel only •The I and Q (chromatic) are interpolated (bi-cubic) •Combined to form such SR results
  92. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Efficiency Takes 6 seconds with 2.1GHz CPU on

    a PC to up-sample from 1282 to 5122 pixels. Fattal et al., “Image Upsampling via Imposed Edges Statistics”, Proc. SIGGRAPH 2007
  93. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  A Performance Evaluation of Image Interpolation and Superresolution

    Algorithms Xin Ye, Xiqun Lu; Zhejiang University International Conference on Multimedia Technology (ICMT), 2011
  94. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  *NBHF$SFEJU9*O:F 9JRVO-V l"1FSGPSNBODF&WBMVBUJPOPG*NBHF*OUFSQPMBUJPOBOE4VQFSSFTPMVUJPO"MHPSJUINTz • TRI: H. S. Hou

    and H. C. Andrews, “Cubic Splines for Image Interpolation and Digital Filtering,” IEEE Trans. On Acoustics, Speech, and Signal Processing, 1981 • NEDI: X. Li and M. T. Orchard, “New Edge-Directed Interpolation,” IEEE Trans. On Image Processing, 2001 • GPP: J. Sun, J. Sun, Z. B. Xun and H. Y. Shum, “Image Super-resoluton using Gradient Profile Prior,” CVPR, 2008 • IUES: R. Fattal, “Image upsampling via imposed edge statistics,” Proceeding of ACM SIGGRAPH, 2007 • SRSI: D. Glasner, S. Bagon, M. Irani, “Super-Resolution from a Single Image,” ICCV, 2009
  95. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  •Use PSNR & MSSIM to evaluate reconstructed image

    quality •PSNR & MSSIM of bilinear & cubic convolution are larger than new-edge directed interpolation & unified SR •Visual evaluation •primary metric •PSNR & MSSIM don’t provide accurate measure of visual quality in such case
  96. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Space-Time Super-Resolution from a Single Video Oded Shahar,

    Alon Faktor, Michal Irani; The Weizmann Institute of Science Conference of Computer Vision and Pattern Recognition (CVPR), 2011
  97. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  Example-based Learning for Single-Image Super-Resolution and JPEG Artifact

    Removal Kwang In Kim and Younghee Kwon; Max Planck Institute for Biological Cybernetics DAGM Symposium for Pattern Recognition, 2008
  98. #Z4IBP$IVOH$IFO-JDFOTFEVOEFS$$/$#:  MobiUP: An Upsampling-Based System Architecture for High-Quality Video

    Streaming on Mobile Devices Hong-Han Shuai, De-Nian Yang, Wen-Huang Cheng, Ming-Syan Chen Academia Sinica, National Taiwan University IEEE Transactions on Multimedia (TMM), 2011