>>> ./emotiondetection
detecting emotion… done.
FACE IS HAPPY
:-)
Slide 9
Slide 9 text
Understanding Faces
• What does iBUG do?
• What key techniques do we use?
• What are the applications of our work?
• What challenges lie in the future?
Slide 10
Slide 10 text
About me
• 2008-11 BSc Physics, Imperial
• 2011-12 MSc Computing Science, Imperial
• 2012-13 RA, iBUG, Imperial
• 2013-present PhD candidiate under Stefanos
Zafeiriou, iBUG, Imperial
Slide 11
Slide 11 text
iBUG
• Intelligent Behaviour Understanding Group
• anything to do with machines understanding
humans
• machine learning
• Group roughly split in two
• behavioural
• computer vision (faces)
Slide 12
Slide 12 text
iBUG - Computer Vision
• vision problems fall into two categories
REAL
TIME
OFFLINE
Slide 13
Slide 13 text
Morphable Models video
OFFLINE
Slide 14
Slide 14 text
3D Morphable Models
• Built from a statistical model of the face
• Learning a detailed 3D face from a 2D face.
• Generally not real time, quality is key, not speed
• Used in games, CGI effects in movies
• Local optimisation method - needs to be pre aligned
(probably by a tracker)
• My main focus!
OFFLINE
Slide 15
Slide 15 text
Facial Tracker demo
REAL TIME
Slide 16
Slide 16 text
Facial tracker
• Built with a statistical model of the face
• Expected to be real time
• Robust to occlusions (stuff getting in the way)
• May be able to deal with multiple persons
• Often required to bootstrap of more vision algorithms
• Consumer applications already here (Kinect)
REAL TIME
Slide 17
Slide 17 text
PCA PCA
3D scans images
Morphable
Model
Shape
Model
Lucas!
Kanade
Cascade of!
regressors
… …
Morphable Model Face Tracker
Slide 18
Slide 18 text
Principal Component
Analysis
• The bedrock of the CV community
• Basic approach extremely simple!
• Many exotic derivatives for incremental
improvements.
• Going to explain the basics
Slide 19
Slide 19 text
Principal Components
Analysis
x
y
100
80
Slide 20
Slide 20 text
Principal Components
Analysis
x
y
μ
100
80
Slide 21
Slide 21 text
Principal Components
Analysis
x
y
μ
100
80
Slide 22
Slide 22 text
Principal Components
Analysis
x
y
μ
PC1
100
80
variance maximised
this is the First "
Principal Component
Slide 23
Slide 23 text
Principal Components
Analysis
x
y
μ
PC1
PC2
100
80
variance maximised
this is the First "
Principal Component
Slide 24
Slide 24 text
Principal Component
Analysis
• Technique to find linear trends in data
• Principal Components are orthogonal to each other
and are ordered in terms of the variance they
capture
• Can be applied in N dimensions
Slide 25
Slide 25 text
Principal Components as a
basis
x
y
100
80
Slide 26
Slide 26 text
Principal Components as a
basis
x
y
100
80
= 71x + 62y
71
62
Slide 27
Slide 27 text
Principal Components as a
basis
x
y
μ
PC1
PC2
100
80
= 71x + 62y
71
62
Slide 28
Slide 28 text
Principal Components as a
basis
x
y
μ
PC1
PC2
100
80
= 71x + 62y
= μ + 0.8PC1 - 0.9PC2
0.8
-0.9
Slide 29
Slide 29 text
Principal Components as a
basis
x
y
μ
PC1
PC2
100
80
= 71x + 62y
= μ + 0.8PC1 - 0.9PC2
= 91x + 32y
= μ + 0.8PC1 - 3.0PC2
Slide 30
Slide 30 text
Principal Components as a
basis
x
y
μ
PC1
PC2
100
80
= 71x + 62y
= μ + 0.8PC1 - 0.9PC2
= 91x + 32y
!
= μ + 0.8PC1 - 3.0PC2
> 1.0 == outlier
Slide 31
Slide 31 text
Shape Model demo
Slide 32
Slide 32 text
What is shape?
• What remains after affects of scale, rotation and
translation have been removed
Slide 33
Slide 33 text
What is shape?
• What remains after affects of scale, rotation and
translation have been removed
Slide 34
Slide 34 text
What is shape?
• What remains after affects of scale, rotation and
translation have been removed
Slide 35
Slide 35 text
What is shape?
• What remains after affects of scale, rotation and
translation have been removed
Slide 36
Slide 36 text
Vectorizing shapes
• Just assign each
coordinate to a new
vector axis
• As long as we are
consistent in the
mapping, can always go
from vector back to shape
• This is why landmarks
were numbered
• Can be done for arbitrary
data
Face landmarks
(68 points of 2D)
136D vector
space
landmark 1:x axis 1
landmark 1:y axis 2
landmark 2:x axis 3
… …
landmark 68:y axis 136
Take PCA on this!
Slide 37
Slide 37 text
Statistical Shape Models
• Summary - Each component captures key traits in
face
• Gives an ideal for how ‘normal’ a face is from the
component weightings
• Works well because faces have a very similar
structure!
• Same idea applied to pixels yields texture models
Slide 38
Slide 38 text
Generating novel faces
• By creating linear combinations of components we
generate novel faces
• Very powerful. Normally first 30 components are
sufficient to generate a close approximation to
anyones face
• We just need an algorithm to find the weightings..
Slide 39
Slide 39 text
Fitting algorithms
• Complex beasts! Outside the scope of today
• Generally find component weightings that minimise
an error function
• Lots of active research in this area
Slide 40
Slide 40 text
Great Ormond Street
• Collaboration with craniofacial surgeons there
• Current surgical approach is an ‘art’
• Want to see if our models can help improve surgery
techniques
• Requires answers to difficult questions - what is
‘normal’? What is ‘beautiful’?
Slide 41
Slide 41 text
My PhD goal
• Emotive Morphable Models
• Current techniques can show full range of emotion
• Needs lots of data and new techniques
• iBUG solving data problem..
Slide 42
Slide 42 text
iBUG4D
• 4D (3D at 60fps) capture
setup
• Capturing data from a range
of subjects right now!
• Small financial reward, only
takes 20mins
• Get a cool 3D scan of yourself
• Email Teresa if you fancy it
[email protected]
Slide 43
Slide 43 text
Thoughts on PhD
• Founded a team project in iBUG - PyBug - a Python
toolkit for statistical modelling and analysis
• Working in a team transforms your PhD - seek
collaboration where you can!
• Important to raise engineering standards in
Computing research, keen to play a role
• Imperial Python Users Group…?