Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Learning to Remove Soft Shadows

Maciek
September 07, 2015

Learning to Remove Soft Shadows

This presentation explains details of our SIGGRAPH 2015 paper in a little more accessible format. I have recently presented this at the Berlin Machine Learning Meetup.

Maciek

September 07, 2015
Tweet

Other Decks in Research

Transcript

  1. 1 Maciej Gryka RainforestQA Michael Terry University of Waterloo Gabriel

    Brostow University College London Learning to Remove Soft Shadows
  2. 2

  3. 3

  4. 4

  5. 5

  6. 6

  7. 7

  8. 8

  9. Previous Work 10 Intrinsic images - [Barrow & Tenenbaum 1978]

    - [Tappen et al. 2005] - [Bosseau et al. 2009] Shadow rendering - [Parker et al. 1998] - [Chan and Durand 2003] Hard shadow removal - [Finlayson et al. 2009] illumination-invariant images - [Shor and Lischinski 2008] Soft shadow removal - [Mohan et al. 2007] - [Arbel and Hel-Or 2011] - [Wu et al. 2007], [Guo et al. 2012] matting methods
  10. System Overview 22 compute a feature vector f( ) Random

    Forest for each masked intensity patch use a trained Random Forest to obtain several matte suggestions RGB image and a binary mask divide and align patches re-align and regularize to get a single patch per site optimize for color inpaint
  11. System Overview 23 compu feature v f( ) for each

    m RGB image and a binary mask divide and align patches inpaint
  12. System Overview 24 compute a feature vector f( ) Random

    Forest for each masked intensity patch use a trained Random Forest to obtain several matte suggestions and align ches re-align and r to get a s patch pe paint
  13. System Overview 25 Random in several estions re-align and regularize

    to get a single patch per site optimize for color
  14. Off-the-shelf inpainting 29 Barnes et al., PatchMatch: a Randomized Correspondence

    Algorithm for Structural Image Editing, SIGGRAPH 2009
  15. Preprocessing: patch alignment 34 - minimize the amount of training

    data needed
 - only learn things, what we cannot parameterize: penumbra fall-off
 - we find a Euclidean transform for each patch to bring it as close as possible to the template patch
  16. Feature vector 43 Our feature vector contains: - normalized pixel

    intensity values (shifted in the intensity domain so that their mean falls at 0.5),
  17. Feature vector 44 Our feature vector contains: - normalized pixel

    intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences),
  18. Feature vector 45 Our feature vector contains: - normalized pixel

    intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences), - normalized distance from the edge of the user-masked region,
  19. Feature vector 46 Our feature vector contains: - normalized pixel

    intensity values (shifted in the intensity domain so that their mean falls at 0.5), - x- and y-gradients (finite differences), - normalized distance from the edge of the user-masked region, - predicted matte for this patch (initial guess).
  20. Feature frequency 47 0 200 400 600 800 1000 5000

    10000 distance from the edge initial guess x-gradient y-gradient normalized intensity
  21. Post-processing 50 - Before “putting the patches down” on the

    graph, we re-align them to their original orientation. - After regularization we have a single-channel shadow matte. - Final optimization recovers color.
  22. 55

  23. 56

  24. Summary of results 63 - The first perceptual user study

    of soft shadow removal methods. - 2x more soft shadow images than previously available. - Our results were chosen as the most convincing overall. - Our method also wins in synthetic measures (distance to ground truth), but this is not a good success criterion.
  25. 66 Probability of winning ranking Kruschke, J., Doing Bayesian Data

    Analysis, 2011 probability probability density
  26. Conclusions 71 Our method learns about physical phenomena from synthetic

    data. It uses the fruits of graphics research together with machine learning to create perceptually superior results. http://visual.cs.ucl.ac.uk/pubs/softshadows/ See our website for: - paper + video + results - data generation / rendering scripts - user study code - algorithm code