Slides for the presentation given at Semana da Ciência e Tecnologia da Universidade de Évora, April 17th 2012.
Skeltrack - Open Source Skeleton Tracking
Joaquim Rocha, Igalia
Semana da Ciência e Tecnologia Univ. de Évora, April 2012
Microsoft's Kinect was the ﬁrst camera
with a price aﬀordable to the public
The USB connection is open and thus hackable
This originated Open Source projects like the libfreenect,
a library to control the Kinect device and get its information
We created a GLib wrapper for libfreenect called GFreenect
GFreenect oﬀers asynchronous functions (and some synchronous as
well) and makes it easy to use with other GNOME technologies
Kinect has a time-of-ﬂight (ToF) camera which gives depth information
But that's raw information... values from 0-2048
libfreenect/GFreenect recently can give those values in mm
It does NOT tell you there is a person in the picture
Or a monkey
Or a cow
Let alone a skeleton and where its joints are
For this you need a skeleton tracking solution
Three proprietary/closed solutions exist:
Microsoft Kinect SDK: non-commercial only
OpenNI: commercial compatible
Kinect for Windows: commercial use allowed
but incompatible with the XBox's Kinect
Conclusion: There were no Free solutions to
perform skeleton tracking... :(
So Igalia built one!
What we wanted:
✩ A shared library, no fancy SDK
✩ Device independent
✩ No pattern matching, no databases
✩ Easy to use (everybody wants that!)
Not as easy as it sounds!
After some investigation we found Andreas Baak's
paper "A Data-Driven Approach for Real-Time Full
Body Pose Reconstruction from a Depth Camera"
However this paper uses a database of
poses to get what the user is doing
So we based our work on it until
the part of getting the extremas
How does it work?
First we need to ﬁnd the extremas
Make a graph whose nodes are the depth pixels
Connect two nodes if the distance is less than a
Connect the diﬀerent graph's components by using
Choose a starting point and calculate Dijkstra to
each point of the graph, choose the furthest point:
there you got your extrema!
Then create an edge between the starting point
and the current extrema point with 0 cost and
repeat the same process now using the current
extrema as a starting point.
This comes from Baak's paper and the diﬀerence
starts here: choosing the starting point
Baak chooses a centroid as the starting point
We choose the bottom-most point starting from the
centroid (this showed better results for the upper
So we got ourselves some extremas!
What to do with them?
What extrema is a hand, a head, a shoulder?
For that we use educated guesses...
We calculate 3 extremas
Then we check each other hoping they are the head
For each extrema we look for the points in places
where the shoulders should be, checking their distances
between the extrema and between each other.
If they obey those rules then we assume they are
the head'n'shoulders (tm)
With the remaining 2 extremas, we will try to see if
they are elbows or hands
How to do it?
Calculate Dijkstra from the shoulders to each extrema
The closest extrema to any of the shoulders is either a
hand of an elbow of that shoulder
How to check if it's a hand or elbow?
If the distance between the extrema and the shoulder is
less than a predeﬁned value, then it is an elbow. Otherwise
it is a hand.
If it is a hand, we ﬁnd the elbow by choosing the point in
the middle of the path we created with Dijkstra before
There is still some things missing...
Hands from elbows: If one of the extremas is an elbow, we
need to infer where the hand is
Smoothing: Smooth the jittering of the joints
Robustness: Use restrictions to ignore objects that are not
And of course, get the rest of the joints: hips, knees, etc.
How to use it?
SkeltrackSkeleton *skeleton = SKELTRACK_SKELETON (skeltrack_skeleton_new ());
ID: HEAD, LEFT_ELBOW, RIGHT_HAND, ...
x: X coordinate in real world (in mm)
y: Y coordinate in real world (in mm)
screen_x: X coordinate in the screen (in pixels)
screen_y: Y coordinate in the screen (in pixels)
Creative Commons pictures from ﬂickr:
Kid Playing: Rob Welsh
Skeleton: Dark Botxy