Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Skeltrack: Open Source Skeleton Tracking

Skeltrack: Open Source Skeleton Tracking

Skeltrack is the first Open Source skeleton tracking library.
These slides are from the presentation about Skeltrack on GUADEC 2012 at A Coruña, Spain.

A0a1e3a9ca85502ca53f11819d236764?s=128

Joaquim Rocha

July 29, 2012
Tweet

More Decks by Joaquim Rocha

Other Decks in Programming

Transcript

  1. Skeltrack - Open Source Skeleton Tracking Joaquim Rocha, Igalia GUADEC

    2012 - A Coruña
  2. Igalia's Interactivity Team was on to investigate immersive applications using

    FOSS
  3. These applications represent new/different ways of interaction for users

  4. Those can go from entertainment to more serious applications

  5. None
  6. The Kinect

  7. Microsoft's Kinect was the first camera with a price affordable

    to the public
  8. The USB connection is open and thus hackable

  9. This originated Open Source projects like the libfreenect, a library

    to control the Kinect device and get its information
  10. We created a GLib wrapper for libfreenect called GFreenect

  11. GFreenect offers asynchronous functions (and some synchronous as well) and

    makes it easy to use with other GNOME technologies
  12. GObject Introspection opens these technologies for users of Python, Javascript,

    Vala, etc.
  13. The Kinect has a structured light camera which gives depth

    information
  14. But that's raw information... values from 0-2048

  15. libfreenect/GFreenect can give those values in mm

  16. None
  17. Still...

  18. It does NOT tell you there is a person in

    the picture
  19. Or a cow

  20. Or a monkey

  21. Let alone a skeleton and where its joints are!

  22. For this you need a skeleton tracking solution

  23. Skeleton Tracking

  24. Three proprietary/closed solutions exist:

  25. Microsoft Kinect SDK: non-commercial only

  26. OpenNI: commercial compatible

  27. Kinect for Windows: commercial use allowed but incompatible with the

    XBox's Kinect
  28. None
  29. Conclusion: There were no Free solutions to perform skeleton tracking...

    :(
  30. So Igalia built one!

  31. Enter Skeltrack

  32. What we wanted: ✩ A shared library, no fancy SDK

    ✩ Modular (do one thing) ✩ Device independent ✩ No pattern matching, no databases ✩ Easy to use (everybody wants that!)
  33. Not as easy as it sounds!

  34. The current version tracks up to 7 joints: ✩ Head;

    ✩ Shoulders; ✩ Elbows; ✩ Hands.
  35. Smoothing is implemented using Holt's Double Exponential formulas

  36. Smoothing and heuristics used can be tweaked using GObject properties

  37. There are still some things missing...

  38. Future work

  39. Hands from elbows: If one of the extremas is an

    elbow, we need to infer where the hand is
  40. Robustness: Use restrictions to ignore objects that are not the

    user
  41. Multi-user: Track more than one person at a time

  42. And of course, get the rest of the joints: hips,

    knees, etc.
  43. How to use it?

  44. First reduce the depth buffer's dimension Reducing the buffer's dimension

    will make things faster
  45. While at it, it's wise to also apply a depth

    threshold
  46. Asynchronous API

  47. SkeltrackSkeleton *skeleton = SKELTRACK_SKELETON (skeltrack_skeleton_new ()); skeltrack_skeleton_track_joints (skeleton, depth_buffer, buffer_width,

    buffer_height, NULL, on_track_joints, NULL);
  48. None
  49. Synchronous API

  50. SkeltrackJointList list; list = skeltrack_skeleton_track_joints_sync (skeleton, depth_buffer, buffer_width, buffer_height, NULL,

    NULL);
  51. Skeleton Joint: ID: HEAD, LEFT_ELBOW, RIGHT_HAND, ... x: X coordinate

    in real world (in mm) y: Y coordinate in real world (in mm) screen_x: X coordinate in the screen (in pixels) screen_y: Y coordinate in the screen (in pixels)
  52. Join us! https://github.com/joaquimrocha/Skeltrack

  53. Nifty Tools for Development: GFreenect: https://github.com/elima/GFreenect GFreenect Utils: https://github.com/joaquimrocha/gfreenect-utils

  54. GFreenect Python Example

  55. Tool: depth-file-viewer

  56. Tool: record-depth-file

  57. Demo? Warning: Might go wrong!

  58. Questions?

  59. Creative Commons pictures from flickr: Kinect: Auxo.co.kr Monkey: nothingtosay Kid

    Playing: Rob Welsh Skeleton: Dark Botxy Other non CC images: Doctor: http://research.microsoft.com/en-us/projects/touchless/ Minority Report: http://www.ovelf.com/2010/04/14/oblong-tan-g-speak-arttirilmis-gerceklikte-hareketlerin-3-boyutta-yorumlanmasi/