Learning to Remove Soft Shadows

This presentation explains details of our SIGGRAPH 2015 paper in a little more accessible format. I have recently presented this at the Berlin Machine Learning Meetup.

Transcript

1. 1 Maciej Gryka RainforestQA Michael Terry University of Waterloo Gabriel Brostow University College London Learning to Remove Soft Shadows
9. Now we can remove and edit shadows. Exciting! 9
10. Previous Work 10 Intrinsic images - [Barrow & Tenenbaum 1978] - [Tappen et al. 2005] - [Bosseau et al. 2009] Shadow rendering - [Parker et al. 1998] - [Chan and Durand 2003] Hard shadow removal - [Finlayson et al. 2009] illumination-invariant images - [Shor and Lischinski 2008] Soft shadow removal - [Mohan et al. 2007] - [Arbel and Hel-Or 2011] - [Wu et al. 2007], [Guo et al. 2012] matting methods
11. Soft shadows: what do penumbrae look like? 11
12. Soft shadows: what do penumbrae look like? 12
13. Soft shadows: what do penumbrae look like? 13
14. Soft shadows: what do penumbrae look like? 14
15. Soft shadows: what do penumbrae look like? 15
16. Soft shadows: what do penumbrae look like? 16
17. Soft shadows: what do penumbrae look like? 17
18. Soft shadows: what do penumbrae look like? 18
19. Instead, we looked at many examples.
20. 20 Instead, we looked at many examples.
21. The algorithm taught itself how penumbrae behave, while ignoring texture variation.
22. System Overview 22 compute a feature vector f( ) Random Forest for each masked intensity patch use a trained Random Forest to obtain several matte suggestions RGB image and a binary mask divide and align patches re-align and regularize to get a single patch per site optimize for color inpaint
23. System Overview 23 compu feature v f( ) for each m RGB image and a binary mask divide and align patches inpaint
24. System Overview 24 compute a feature vector f( ) Random Forest for each masked intensity patch use a trained Random Forest to obtain several matte suggestions and align ches re-align and r to get a s patch pe paint
25. System Overview 25 Random in several estions re-align and regularize to get a single patch per site optimize for color
26. System Overview 26 for e inpaint
27. Guided inpainting 27 input image guided inpainting result
28. 28 sim ilar pixels Guided inpainting input image guided inpainting result
29. Off-the-shelf inpainting 29 Barnes et al., PatchMatch: a Randomized Correspondence Algorithm for Structural Image Editing, SIGGRAPH 2009
30. Preprocessing: patch alignment 30
31. Preprocessing: patch alignment 31
32. Preprocessing: patch alignment 32 f( )
33. Preprocessing: patch alignment 33 f( )
34. Preprocessing: patch alignment 34 - minimize the amount of training data needed
 - only learn things, what we cannot parameterize: penumbra fall-off
 - we find a Euclidean transform for each patch to bring it as close as possible to the template patch
35. Learning: customized Regression Random Forests 35 nearest-neighbor search in non-Euclidean space guided by training data
36. Learning 36
37. Learning 37 false true true false true false
38. Learning 38 false true true false true false
39. 39 false true true false true false
40. 40 false true true false true false
41. 41 false true true false true false
42. 42 [Criminisi et al. 2013] false true true false true false
43. Feature vector 43 Our feature vector contains: - normalized pixel intensity values (shifted in the intensity domain so that their mean falls at 0.5),
44. Feature vector 44 Our feature vector contains: - normalized pixel intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences),
45. Feature vector 45 Our feature vector contains: - normalized pixel intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences), - normalized distance from the edge of the user-masked region,
46. Feature vector 46 Our feature vector contains: - normalized pixel intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences), - normalized distance from the edge of the user-masked region, - predicted matte for this patch (initial guess).
47. Feature frequency 47 0 200 400 600 800 1000 5000 10000 distance from the edge initial guess x-gradient y-gradient normalized intensity
48. Regularization 48
49. 49 Kolmogorov, V., Convergent Tree-reweighted Message Passing for Energy Minimization, PAMI 2006 Regularization
50. Post-processing 50 - Before “putting the patches down” on the graph, we re-align them to their original orientation. - After regularization we have a single-channel shadow matte. - Final optimization recovers color.
51. Post-processing 51 naive solution
52. Post-processing: color optimization 52 0.0 0.5 1.0 1.0
53. Post-processing 53 naive solution our solution
54. Results! 54
57. 57 ours our matte Arbel & Hel-Or 2011 Guo et al. 2012 input
58. 58 ours Arbel & Hel-Or 2011 Guo et al. 2012 input our matte
59. 59 input
60. 60 input initial guess
61. 61 input initial guess output
62. 62 input input mask output
63. Summary of results 63 - The first perceptual user study of soft shadow removal methods. - 2x more soft shadow images than previously available. - Our results were chosen as the most convincing overall. - Our method also wins in synthetic measures (distance to ground truth), but this is not a good success criterion.
64. Two-phase perceptual study 64 Ranking
65. Two-phase perceptual study 65 Ranking Scoring
66. 66 Probability of winning ranking Kruschke, J., Doing Bayesian Data Analysis, 2011 probability probability density
67. 67 Kruschke, J., Doing Bayesian Data Analysis, 2011 probability probability probability density
68. Limitations 68
69. Limitations 69 input image unshadowed result
70. Limitations 70 initial guess (inpainting) unshadowed result
71. Conclusions 71 Our method learns about physical phenomena from synthetic data. It uses the fruits of graphics research together with machine learning to create perceptually superior results. http://visual.cs.ucl.ac.uk/pubs/softshadows/ See our website for: - paper + video + results - data generation / rendering scripts - user study code - algorithm code
72. Thanks!