Manipulating Scale-Dependent Perception of Images

The purpose of most images is to effectively convey information. Implicit in this assumption is the fact that the recipient of that information is a human observer, with a visual system responsible for converting raw sensory inputs into the perceived appearance. The appearance of an image not only depends on the image itself, but the conditions under which it is viewed as well as the response of human visual system to those inputs. This thesis examines the scale-dependent nature of image appearance, where the same stimulus can appear different when viewed at varying scales, that arises from the mechanisms responsible for processing spatial vision in the brain. In particular, this work investigates changes in the perception of blur and contrast resulting from the image being represented by different portions of the viewer's visual system due to changes in image scale. These methods take inspiration from the fundamental organization of spatial image perception into multiple parallel channels for processing visual information and employ models of human spatial vision to more accurately control the appearance of images under changing viewing conditions. The result is a series of methods for understanding the blur and contrast present in images and manipulating the appearance of those qualities in a perceptually-meaningful way.

Transcript

1. Manipulating Scale-Dependent Perception of Images Matthew Trentacoste PhD Dissertation Defense University of British Columbia
2. An example 2
3. An example 2
4. An example 2
5. Implicit assumption 3
6. Implicit assumption 3
7. Implicit assumption 3
8. Spatial vision 4 Contrast sensitivity function (CSF)
9. Spatial vision 4 Contrast sensitivity function (CSF)
10. Spatial vision 4 Contrast sensitivity function (CSF)
11. Spatial vision 4 Contrast sensitivity function (CSF)
12. Spatial vision 4 Contrast sensitivity function (CSF)
13. Spatial vision 4 Contrast sensitivity function (CSF)
14. Spatial vision 4 Contrast sensitivity function (CSF)
15. Spatial vision 4 Contrast sensitivity function (CSF)
16. Spatial vision 4 Contrast sensitivity function (CSF)
17. Spatial vision • HVS processes information with multiple, parallel, band-limited channels • Attributes of each channel differ • Angular resolution of an image feature determines which channel • Operations that move content from one channel to another can alter image perception 5
18. Hybrid images 6 [Oliva et al. 2006]
19. Hybrid images 6 [Oliva et al. 2006]
20. Overview 7 Spatially-variant blur estimation
21. Overview 7 Spatially-variant blur estimation Synthetic DOF for mobile devices
22. Overview 7 Spatially-variant blur estimation Synthetic DOF for mobile devices Blur-aware image downsampling
23. Overview 7 Spatially-variant blur estimation Synthetic DOF for mobile devices Blur-aware image downsampling Scale-dependent perception of countershading
24. Overview 7 Spatially-variant blur estimation Synthetic DOF for mobile devices Blur-aware image downsampling Scale-dependent perception of countershading Defocus dynamic range expansion
25. Spatially-Variant Blur Estimation
26. Blur estimation 9
27. Blur estimation Blur estimation 0px blur 15px blur 9
28. Blur estimation Blur estimation 0px blur 15px blur 9 • Calibrate method of Samadani et al. to provide estimate of blur in absolute units • Relation between width of a Gaussian profile and the peak value of its derivative
29. Blur estimation g (x, σ) = 1 √ 2πσ2 e − x2 √ 2σ2 Edge Gradient magnitude width: σ 10
30. Blur estimation g (x, σ) = 1 √ 2πσ2 e − x2 √ 2σ2 Edge Gradient magnitude width: σ 10
31. Blur estimation g (x, σ) = 1 √ 2πσ2 e − x2 √ 2σ2 Edge Gradient magnitude width: σ Downsampled scale space 10
32. Blur estimation g (x, σ) = 1 √ 2πσ2 e − x2 √ 2σ2 Edge Gradient magnitude width: σ Downsampled scale space 10
33. Blur estimation g (x, σ) = 1 √ 2πσ2 e − x2 √ 2σ2 Edge Gradient magnitude width: σ Downsampled scale space 10
34. 11 Noise sensitivity Noise free image + gradients Noisy image + gradients Actual vs estimated blur by noise level Image
35. 11 Noise sensitivity Noise free image + gradients Noisy image + gradients Actual vs estimated blur by noise level Minimum reliable scale map
36. 12 Original Blur estimate Results
37. 13 Original Blur estimate Results
38. 14 Original Blur estimate Results
39. Synthetic Depth-of-Field for Mobile Devices
40. 16 Nikon D70 Digital SLR iPhone 3GS Depth-of-field
41. 16 Nikon D70 Digital SLR iPhone 3GS Depth-of-field @ f/2.8 @ f/2.8
42. 17 Original Blur estimate Synthetic depth-of-field
43. 17 Original Narrower depth-of-field Synthetic depth-of-field
44. Blur synthesis • Obtain map of estimated blur in image • Modify to represent pattern of desired f-number • Solve for blur to add given blur present 18 Narrower DOF Original Blur estimate Lens blur model
45. Blur synthesis 19 Blur map Final result Scalespace + =
46. 20 Original Synthesized Results
47. 21 Original Synthesized Results
48. 22 Original Synthesized Results
49. Benefits and limitations 23 • Narrower DOF images than optics alone • Reasonable quality, efficient enough to be implemented on mobile devices • Use of Gaussian blur in synthesizing new DOF for efficiency
50. Limitations 24 Image Estimated Original Estimated Synthesized Motion blur Noise vs texture
51. Blur-Aware Image Downsampling
52. Motivation • Sensors higher resolution than displays • Image display implies image downsizing • Conventional downsizing doesn’t accurately represent image appearance and perception of the image changes 26
53. Motivation • Sensors higher resolution than displays • Image display implies image downsizing • Conventional downsizing doesn’t accurately represent image appearance and perception of the image changes 2 Mp 26
54. Motivation • Sensors higher resolution than displays • Image display implies image downsizing • Conventional downsizing doesn’t accurately represent image appearance and perception of the image changes 2 Mp 3-22 Mp 26
55. Blur-aware image downsampling 27
56. Blur-aware image downsampling 27
57. Blur-aware image downsampling 27
58. Perceptual study • Blur-matching experiment • Given large image with reference amount of blur present • Need to adjust blur in smaller images to match appearance of large • Repeated for between 0 and .26 visual degrees and downsamples of 2x 4x 8x &r ςr 28
59. Perceptual study • Blur-matching experiment • Given large image with reference amount of blur present • Need to adjust blur in smaller images to match appearance of large • Repeated for between 0 and .26 visual degrees and downsamples of 2x 4x 8x &r ςr 28
60. Matching results • Matching blur larger than reference blur, smaller images appear sharper • Curves level off with larger blur, downsample -- blur sufficient to covey appearance • Viewing setup had Nyquist limit of 30 cpd - results not due to limited resolution in terms of pixels, but visual angle Full-size image blur radius ( ) [vis deg] &r 29
61. Matching results • Matching blur larger than reference blur, smaller images appear sharper • Curves level off with larger blur, downsample -- blur sufficient to covey appearance • Viewing setup had Nyquist limit of 30 cpd - results not due to limited resolution in terms of pixels, but visual angle Full-size image blur radius ( ) [vis deg] &r 29 • Viewing setup had Nyquist limit of 30 cpd - results not due to limited resolution in terms of pixels, but visual angle
62. Blur synthesis 30 Blur map Final result Scalespace Perceived blur
63. Evaluation Naive Blur-Aware 31
64. Evaluation Naive Blur-Aware 32
65. Evaluation 33 original Original 2x normal 2x blur-aware 2x naive 2x blur-aware 4x blur-aware 4x normal 4x naive 4x aware
66. Evaluation 34 Original 4x naive 4x aware 2x naive 2x blur-aware 2x blur-aware 2x normal 4x blur-aware 4x blur-aware 4x normal
67. Benefits and limitations 35 • Fully automatic image resizing operator that uses a perceptual metric to preserve image appearance • Effect due to HVS: The same metric can account for changes in appearance due to viewing distance • Same limitations of blur estimate apply
68. Scale-Dependent Perception of Countershading
69. Countershading 37
70. Countershading effects 38
71. Countershading effects 38
72. Countershading effects 38
73. 39 Perceptual experiment
74. 39 Perceptual experiment
75. 40 −2 −1 0 1 0.2 0.4 0.6 0.8 1 1.2 Profile width m [log 10 deg] Profile magnitude h Edge − high Edge − med Edge − low Scallop threshold −2 −1 0 1 0.2 0.4 0.6 0.8 1 1.2 Profile width m [log 10 deg] Profile magnitude h Coast Palm beach Building Model fit Scallop threshold Results • U-shaped curve • Subjects tolerated significantly more contrast for narrow/ wide countershading • Trough in the middle where almost any countershading deemed objectionable • Does not correspond to any known aspect of visual perception
76. 41 Contrast comparison
77. 41 Contrast comparison
78. 42 Countershading magnitude Spatial frequency Indistinguishable countershading Objectionable countershading (halos) Acceptable countershading
79. 43 Resizing, different displays
80. 43 Resizing, different displays
81. 44 Tonemapping Original [Durand and Dorsey 2002] Adjust to approximate image scale σs
82. 45 Scale-aware displays • Determine distance of viewer using head-tracking • Present images for specific viewing conditions • Need headset Only works for one viewer
83. 45 Scale-aware displays • Determine distance of viewer using head-tracking • Present images for specific viewing conditions • Need headset Only works for one viewer
84. 46 Countershading estimation
85. Benefits and limitations 47 • Model of the perceived of countershading • Several applications for introducing countershading displaying content at different sizes • Limited by the small set of study conditions • Findings not explainable by existing perceptual models -- best used as a heuristic until a larger study is conducted −2 −1 0 1 0.2 0.4 0.6 0.8 1 1.2 Profile width m [log 10 deg] Profile magnitude h Coast Palm beach Building Model fit Scallop threshold Countershading magnitude Spatial frequency Indistinguishable countershading Objectionable countershading (halos)
86. Defocus Dynamic Range Expansion
87. Sensor dynamic range 49 Over-/under-exposed regions
88. Sensor dynamic range 49 Over-/under-exposed regions Exposure bracketing
89. Sensor dynamic range 49 HDR sensors Over-/under-exposed regions Exposure bracketing
90. Local contrast reduction 50 HDR scene
91. Local contrast reduction 50 HDR scene
92. Local contrast reduction 50 HDR scene Blur kernel
93. Local contrast reduction 50 HDR scene Blur kernel = = Captured image
94. Local contrast reduction 50 HDR scene Blur kernel = = Captured image ⊗−1 Result
95. Local contrast reduction 50 HDR scene Blur kernel = = Captured image ⊗−1 Result
96. Aperture filters, priors 51 Sparse gradient priors
97. Aperture filters, priors 51 Sparse gradient priors
98. Experiment • Filters: • Normal aperture • Gaussian • Veeraraghavan • Levn • Zhou • Deconvolution: • Wiener filtering • Richardson-Lucy • Bando • Levin 52 Morning Night
99. Results, no noise 53 Deconv algo (with Zhou) Aperture filter (with Levin) Morning Night
100. Results, filter size 54 filter = Zhou noise = 0 radius = 1 Weiner Richardson-Lucy Bando Levin
101. Results, filter size 55 filter = Zhou noise = 0 radius = 5 Weiner Richardson-Lucy Bando Levin
102. Results, filter size 56 filter = Zhou noise = 0 radius = 16 Weiner Richardson-Lucy Bando Levin
103. Evaluation 57 • Effectiveness scene-dependent Can work for small bright regions, but not large • More complex algorithms work better, at higher computational cost • No filter+deconv combination was able to reduce the dynamic range without degrading image quality • 15px limit to blur radius
104. Conclusions and future work
105. Contributions • Framework of scale-dependent image perception • Mapping image features to visual channels via display size, resolution and viewing conditions • Applications of this perspective to manipulation of blur and contrast in images 59
106. Future work • Improved separation of texture/noise • Improved blur estimation using other sensors in mobile devices • Improved blur synthesis using autofocus sweep 60
107. Future work • Disparity between weighting of deconvolution algorithm and visual perception • Non-linear forms of deconvolution that distribute error closer to luminance quantization of HVS 61
108. Future work 62 • Preserve appearance of other attributes when resizing • More comprehensive study of objectionable countershading, accounting for more conditions, contrast masking • Better means of displaying of scale-dependent content
109. Future work 63 • Comprehensive model of related perception of blur and contrast • Explore lateral inhibition between visual channels • Develop framework similar to color management for addressing perception of images at different sizes
110. Image appearance pipeline 64 Scene Perceived
111. Image appearance pipeline 64 Capture Scene Perceived
112. Image appearance pipeline 64 Capture Process Scene Perceived
113. Image appearance pipeline 64 Capture Process Display Scene Perceived
114. Image appearance pipeline 64 Capture Process Display Viewing conditions Scene Perceived
115. Image appearance pipeline 64 Capture Process Display Viewing conditions Visual system Scene Perceived
118. Thank you.