Skeltrack: Open Source Skeleton Tracking

Skeltrack is the first Open Source skeleton tracking library.
These slides are from the presentation about Skeltrack on GUADEC 2012 at A Coruña, Spain.

Transcript

1. Skeltrack - Open Source Skeleton Tracking Joaquim Rocha, Igalia GUADEC 2012 - A Coruña
2. Igalia's Interactivity Team was on to investigate immersive applications using FOSS
3. These applications represent new/different ways of interaction for users
4. Those can go from entertainment to more serious applications
6. The Kinect
7. Microsoft's Kinect was the first camera with a price affordable to the public
8. The USB connection is open and thus hackable
9. This originated Open Source projects like the libfreenect, a library to control the Kinect device and get its information
10. We created a GLib wrapper for libfreenect called GFreenect
11. GFreenect offers asynchronous functions (and some synchronous as well) and makes it easy to use with other GNOME technologies
12. GObject Introspection opens these technologies for users of Python, Javascript, Vala, etc.
13. The Kinect has a structured light camera which gives depth information
14. But that's raw information... values from 0-2048
15. libfreenect/GFreenect can give those values in mm
17. Still...
18. It does NOT tell you there is a person in the picture
19. Or a cow
20. Or a monkey
21. Let alone a skeleton and where its joints are!
22. For this you need a skeleton tracking solution
23. Skeleton Tracking
24. Three proprietary/closed solutions exist:
25. Microsoft Kinect SDK: non-commercial only
26. OpenNI: commercial compatible
27. Kinect for Windows: commercial use allowed but incompatible with the XBox's Kinect
29. Conclusion: There were no Free solutions to perform skeleton tracking... :(
30. So Igalia built one!
31. Enter Skeltrack
32. What we wanted: ✩ A shared library, no fancy SDK ✩ Modular (do one thing) ✩ Device independent ✩ No pattern matching, no databases ✩ Easy to use (everybody wants that!)
33. Not as easy as it sounds!
34. The current version tracks up to 7 joints: ✩ Head; ✩ Shoulders; ✩ Elbows; ✩ Hands.
35. Smoothing is implemented using Holt's Double Exponential formulas
36. Smoothing and heuristics used can be tweaked using GObject properties
37. There are still some things missing...
38. Future work
39. Hands from elbows: If one of the extremas is an elbow, we need to infer where the hand is
40. Robustness: Use restrictions to ignore objects that are not the user
41. Multi-user: Track more than one person at a time
42. And of course, get the rest of the joints: hips, knees, etc.
43. How to use it?
44. First reduce the depth buffer's dimension Reducing the buffer's dimension will make things faster
45. While at it, it's wise to also apply a depth threshold
46. Asynchronous API
47. SkeltrackSkeleton *skeleton = SKELTRACK_SKELETON (skeltrack_skeleton_new ()); skeltrack_skeleton_track_joints (skeleton, depth_buffer, buffer_width, buffer_height, NULL, on_track_joints, NULL);
49. Synchronous API
50. SkeltrackJointList list; list = skeltrack_skeleton_track_joints_sync (skeleton, depth_buffer, buffer_width, buffer_height, NULL, NULL);
51. Skeleton Joint: ID: HEAD, LEFT_ELBOW, RIGHT_HAND, ... x: X coordinate in real world (in mm) y: Y coordinate in real world (in mm) screen_x: X coordinate in the screen (in pixels) screen_y: Y coordinate in the screen (in pixels)
52. Join us! https://github.com/joaquimrocha/Skeltrack
53. Nifty Tools for Development: GFreenect: https://github.com/elima/GFreenect GFreenect Utils: https://github.com/joaquimrocha/gfreenect-utils
54. GFreenect Python Example
55. Tool: depth-file-viewer
56. Tool: record-depth-file
57. Demo? Warning: Might go wrong!
58. Questions?
59. Creative Commons pictures from flickr: Kinect: Auxo.co.kr Monkey: nothingtosay Kid Playing: Rob Welsh Skeleton: Dark Botxy Other non CC images: Doctor: http://research.microsoft.com/en-us/projects/touchless/ Minority Report: http://www.ovelf.com/2010/04/14/oblong-tan-g-speak-arttirilmis-gerceklikte-hareketlerin-3-boyutta-yorumlanmasi/