Understanding Faces

Powered by ever more sophisticated sensors, the machines of the future will interact with humans far more naturally than is possible today. Most of us find face-to-face interaction the most comfortable and for good reason - the brain does an outstanding job of interpreting the identity and emotional state of another human - all from just looking at their face.

With this in mind we seek to develop algorithmic approaches to understanding the human face. I’ll explain how powerful generative models of the face can be constructed, and what we can learn from them. I’ll then demonstrate the usefulness of such models in identity and emotion recognition, and highlight how our collaboration with Great Ormond Street Hospital is helping advance craniofacial surgery techniques. Finally, I’ll talk about where we can take statistical facial modelling in the future, and discuss some of the challenges that we most overcome in order to advance.

Transcript

1. Understanding Faces James Booth
2. >>> ./facedetector done. running..
3. >>> ./facedetector done. running..
5. >>> ./emotiondetection runnning…
6. >>> ./emotiondetection runnning… ERROR: EmotionError - No hard edges found. Is the image of sufficient resolution? ah! we need to enhance obviously.
7. >>> enhance —csi=‘miami’ :-) arbitrarily enhancing… done.
8. >>> ./emotiondetection detecting emotion… done. FACE IS HAPPY :-)
9. Understanding Faces • What does iBUG do? • What key techniques do we use? • What are the applications of our work? • What challenges lie in the future?
10. About me • 2008-11 BSc Physics, Imperial • 2011-12 MSc Computing Science, Imperial • 2012-13 RA, iBUG, Imperial • 2013-present PhD candidiate under Stefanos Zafeiriou, iBUG, Imperial
11. iBUG • Intelligent Behaviour Understanding Group • anything to do with machines understanding humans • machine learning • Group roughly split in two • behavioural • computer vision (faces)
12. iBUG - Computer Vision • vision problems fall into two categories REAL TIME OFFLINE
13. Morphable Models video OFFLINE
14. 3D Morphable Models • Built from a statistical model of the face • Learning a detailed 3D face from a 2D face. • Generally not real time, quality is key, not speed • Used in games, CGI effects in movies • Local optimisation method - needs to be pre aligned (probably by a tracker) • My main focus! OFFLINE
15. Facial Tracker demo REAL TIME
16. Facial tracker • Built with a statistical model of the face • Expected to be real time • Robust to occlusions (stuff getting in the way) • May be able to deal with multiple persons • Often required to bootstrap of more vision algorithms • Consumer applications already here (Kinect) REAL TIME
17. PCA PCA 3D scans images Morphable Model Shape Model Lucas! Kanade Cascade of! regressors … … Morphable Model Face Tracker
18. Principal Component Analysis • The bedrock of the CV community • Basic approach extremely simple! • Many exotic derivatives for incremental improvements. • Going to explain the basics
19. Principal Components Analysis x y 100 80
20. Principal Components Analysis x y μ 100 80
21. Principal Components Analysis x y μ 100 80
22. Principal Components Analysis x y μ PC1 100 80 variance maximised this is the First " Principal Component
23. Principal Components Analysis x y μ PC1 PC2 100 80 variance maximised this is the First " Principal Component
24. Principal Component Analysis • Technique to find linear trends in data • Principal Components are orthogonal to each other and are ordered in terms of the variance they capture • Can be applied in N dimensions
25. Principal Components as a basis x y 100 80
26. Principal Components as a basis x y 100 80 = 71x + 62y 71 62
27. Principal Components as a basis x y μ PC1 PC2 100 80 = 71x + 62y 71 62
28. Principal Components as a basis x y μ PC1 PC2 100 80 = 71x + 62y = μ + 0.8PC1 - 0.9PC2 0.8 -0.9
29. Principal Components as a basis x y μ PC1 PC2 100 80 = 71x + 62y = μ + 0.8PC1 - 0.9PC2 = 91x + 32y = μ + 0.8PC1 - 3.0PC2
30. Principal Components as a basis x y μ PC1 PC2 100 80 = 71x + 62y = μ + 0.8PC1 - 0.9PC2 = 91x + 32y ! = μ + 0.8PC1 - 3.0PC2 > 1.0 == outlier
31. Shape Model demo
32. What is shape? • What remains after affects of scale, rotation and translation have been removed
33. What is shape? • What remains after affects of scale, rotation and translation have been removed
34. What is shape? • What remains after affects of scale, rotation and translation have been removed
35. What is shape? • What remains after affects of scale, rotation and translation have been removed
36. Vectorizing shapes • Just assign each coordinate to a new vector axis • As long as we are consistent in the mapping, can always go from vector back to shape • This is why landmarks were numbered • Can be done for arbitrary data Face landmarks (68 points of 2D) 136D vector space landmark 1:x axis 1 landmark 1:y axis 2 landmark 2:x axis 3 … … landmark 68:y axis 136 Take PCA on this!
37. Statistical Shape Models • Summary - Each component captures key traits in face • Gives an ideal for how ‘normal’ a face is from the component weightings • Works well because faces have a very similar structure! • Same idea applied to pixels yields texture models
38. Generating novel faces • By creating linear combinations of components we generate novel faces • Very powerful. Normally first 30 components are sufficient to generate a close approximation to anyones face • We just need an algorithm to find the weightings..
39. Fitting algorithms • Complex beasts! Outside the scope of today • Generally find component weightings that minimise an error function • Lots of active research in this area
40. Great Ormond Street • Collaboration with craniofacial surgeons there • Current surgical approach is an ‘art’ • Want to see if our models can help improve surgery techniques • Requires answers to difficult questions - what is ‘normal’? What is ‘beautiful’?
41. My PhD goal • Emotive Morphable Models • Current techniques can show full range of emotion • Needs lots of data and new techniques • iBUG solving data problem..
42. iBUG4D • 4D (3D at 60fps) capture setup • Capturing data from a range of subjects right now! • Small financial reward, only takes 20mins • Get a cool 3D scan of yourself • Email Teresa if you fancy it t.ng@imperial.ac.uk
43. Thoughts on PhD • Founded a team project in iBUG - PyBug - a Python toolkit for statistical modelling and analysis • Working in a team transforms your PhD - seek collaboration where you can! • Important to raise engineering standards in Computing research, keen to play a role • Imperial Python Users Group…?
44. ? t.ng@imperial.ac.uk For capture experiment: