Robocalypse - Full Stack Toronto 2015

Robocalypse: Controlling Nodebots with Kinect Gestures. A #fstoconf15 talk by Joel Lord

Transcript

1. confidential not for distribution
   A Full Stack 2015 Toronto presentation by Joel Lord
   Robocalypse: Controlling
   Nodebots with Kinect gestures
   

   View Slide

2. Overview
   About Me
   Why
   The Goals
   What Exists
   What I wanted
   Gestures 101
   Demo
   

   View Slide

3. About Me
   

   View Slide

4. About Me
   Joel Lord
   Development Manager
   Javascript junkie, technology
   enthusiast, IoT hobbyist.
   Macadamian
   http://www.macadamian.com
   Twitter: @joel__lord
   Github: github.com/joellord
   

   View Slide

5. We’re hiring !
   http://www.macadamian.com/careers
   

   View Slide

6. Houston…
   We’ve got a problem.
   

   View Slide

7. Why ?
   

   View Slide

8. Why
   •  Worked with Nodebots for a while now
   

   View Slide

9. Why
   •  Worked with Nodebots for a while now
   •  Wanted to explore other possibilities of
   NodeJs
   

   View Slide

10. Why
    •  Worked with Nodebots for a while now
    •  Wanted to explore other possibilities of
    NodeJs
    •  Because controlling a robot with a Kinect is
    pretty cool
    

    View Slide

11. The Goals
    

    View Slide

12. The Goals
    •  Needed a way to use events to tell
    the robot to perform an action
    •  Gestures seemed the best way to
    achieve this
    

    View Slide

13. The Goals
    •  Needed a way to use events to tell
    the robot to perform an action
    •  Gestures seemed the best way to
    achieve this
    

    View Slide

14. What currently exists
    

    View Slide

15. What current exists
    Basic Nodebot Stack
    •  NodeJs (https://nodejs.org/)
    

    View Slide

16. What current exists
    Basic Nodebot Stack
    •  NodeJs (https://nodejs.org/)
    •  Socket.io (http://socket.io/)
    

    View Slide

17. What current exists
    Basic Nodebot Stack
    •  NodeJs (https://nodejs.org/)
    •  Socket.io (http://socket.io/)
    •  Johnny-Five (http://johnny-five.io/)
    

    View Slide

18. What current exists
    node-openni
    •  Connects the Kinect to NodeJs using OpenNI
    

    View Slide

19. What current exists
    node-openni
    •  Connects the Kinect to NodeJs using OpenNI
    •  Very basic event for almost everything
    

    View Slide

20. What current exists
    node-openni
    •  Connects the Kinect to NodeJs using OpenNI
    •  Very basic event for almost everything
    •  No gestures !
    

    View Slide

21. What I needed
    

    View Slide

22. What I Needed
    •  Support for gestures
    

    View Slide

23. What I Needed
    •  Support for gestures
    •  Support for a full “skeleton”
    

    View Slide

24. Gestures 101
    

    View Slide

25. Gestures 101
    The Basics
    •  Is the initial condition met?
    

    View Slide

26. Gestures 101
    The Basics
    •  Is the initial condition met?
    •  Is the condition still met?
    

    View Slide

27. Gestures 101
    The Basics
    •  Is the initial condition met?
    •  Is the condition still met?
    •  Is the final condition met?
    

    View Slide

28. Gestures 101
    What do I need for gestures?
    •  A skeleton with an event when it changes
    

    View Slide

29. Gestures 101
    What do I need for gestures?
    •  A skeleton with an event when it changes
    •  Know the position of the COM
    

    View Slide

30. Gestures 101
    What do I need for gestures?
    •  A skeleton with an event when it changes
    •  Know the position of the COM
    •  Have a “base unit”
    

    View Slide

31. Gestures 101
    Introducing kinect-gestures
    •  Gesture detection using a base class
    

    View Slide

32. Gestures 101
    Introducing kinect-gestures
    •  Gesture detection using a base class
    •  Tracks the skeleton for an initial condition
    

    View Slide

33. Gestures 101
    Introducing kinect-gestures
    •  Gesture detection using a base class
    •  Tracks the skeleton for an initial condition
    •  Checks every 100ms to see if the condition is still met
    

    View Slide

34. Gestures 101
    Introducing kinect-gestures
    •  Gesture detection using a base class
    •  Tracks the skeleton for an initial condition
    •  Checks every 100ms to see if the condition is still met
    •  If the final condition is met, trigger an event
    

    View Slide

35. Gestures 101
    Introducing kinect-gestures
    •  Gesture detection using a base class
    •  Tracks the skeleton for an initial condition
    •  Checks every 100ms to see if the condition is still met
    •  If the final condition is met, trigger an event
    •  Other classes are used to define the actual gestures
    

    View Slide

36. Robocalypse
    Demo
    Let’s look at some code !
    

    View Slide

37. Next few steps
    •  Return a more friendly coordinate system for skeleton
    •  Add more information to the returned
    •  Add more gestures (jump, wave, kick)
    

    View Slide

38. Questions?
    

    View Slide

39. Thank you.
    @joel__lord
    github.com/joellord
    

    View Slide

40. macadamian.com
    

    View Slide