Intro to Gestural Interfaces with the Kinect

The slides from my O'Reilly webcast in 4/2012.

Transcript

1. Greg Borenstein @atduskgreg http://gregborenstein.com Intro to the Kinect: Gestural Interfaces Tuesday, April 17, 12
2. NYU ITP Resident Making Things See Special Effects About Me http://shop.oreilly.com/product/0636920020684.do Tuesday, April 17, 12
3. The Plan • What is the Kinect? • A silly example of a gestural interface • The ingredients: joint position and vector math • What makes a good gestural interface? • A better example of a gestural interface Tuesday, April 17, 12
4. https://vimeo.com/31698679 Scan and Print with Makerbot Control an Arduino robot Tuesday, April 17, 12
5. github.com/atduskgreg/making-things-see-examples Tuesday, April 17, 12
6. Tuesday, April 17, 12
7. Tuesday, April 17, 12
8. Tuesday, April 17, 12
9. Tuesday, April 17, 12
10. IR Projector Tuesday, April 17, 12
11. Image courtesy of futurepicture.org Tuesday, April 17, 12
12. IR Projector Tuesday, April 17, 12
13. IR Projector IR Camera Tuesday, April 17, 12
14. IR Projector IR Camera RGB Camera Tuesday, April 17, 12
15. Depth Image RGB Image Tuesday, April 17, 12
16. Skeleton Data Tuesday, April 17, 12
17. $2000 Tuesday, April 17, 12
18. $2000 $3000 Tuesday, April 17, 12
19. Nov. 4 - Nov. 10 Tuesday, April 17, 12
20. openkinect.org Tuesday, April 17, 12
21. Processing.org Tuesday, April 17, 12
22. Dan Shiffman photo by Jeff Howard NYU ITP Tuesday, April 17, 12
23. Tuesday, April 17, 12
24. openni.org Tuesday, April 17, 12
25. Simple OpenNI code.google.com/p/simple-openni Tuesday, April 17, 12
26. Something Silly Tuesday, April 17, 12
27. Something Silly http://www.youtube.com/watch?v=RPbocvQ3Y9s Tuesday, April 17, 12
28. How does this work? Define rules of a Pose Measure relative position of joints Trigger mp3 Tuesday, April 17, 12
29. Get Position of Joints kinect.getJointPositionSkeleton() joint1.x > joint2.x etc... Tuesday, April 17, 12
30. kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, rightHand); kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW, rightElbow); kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, rightShoulder); if(rightElbow.y > rightShoulder.y && rightElbow.x > rightShoulder.x) { stroke(255); // <2> } else { stroke(255,0,0); // <3> } Tuesday, April 17, 12
31. The Basic Ingredients Tuesday, April 17, 12
32. Calibration Tuesday, April 17, 12
33. Calibration Tuesday, April 17, 12
34. void onNewUser(int userId) { println("start pose detection"); kinect.startPoseDetection("Psi", userId); } void onEndCalibration(int userId, boolean successful) { if (successful) { println(" User calibrated !!!"); kinect.startTrackingSkeleton(userId); } else { println(" Failed to calibrate user !!!"); kinect.startPoseDetection("Psi", userId); } } void onStartPose(String pose, int userId) { println("Started pose for user"); kinect.stopPoseDetection(userId); kinect.requestCalibrationSkeleton(userId, true); } Tuesday, April 17, 12
35. void onNewUser(int userId) { println("start pose detection"); kinect.startPoseDetection("Psi", userId); } void onEndCalibration(int userId, boolean successful) { if (successful) { println(" User calibrated !!!"); kinect.startTrackingSkeleton(userId); } else { println(" Failed to calibrate user !!!"); kinect.startPoseDetection("Psi", userId); } } void onStartPose(String pose, int userId) { println("Started pose for user"); kinect.stopPoseDetection(userId); kinect.requestCalibrationSkeleton(userId, true); } Tuesday, April 17, 12
36. void draw() { kinect.update(); PImage depth = kinect.depthImage(); image(depth, 0, 0); // make a vector of ints to store the list of users IntVector userList = new IntVector(); // write the list of detected users // into our vector kinect.getUsers(userList); // if we found any users if (userList.size() > 0) { // get the first user int userId = userList.get(0); // if we're successfully calibrated if ( kinect.isTrackingSkeleton(userId)) { ... Tuesday, April 17, 12
37. ... // if we're successfully calibrated if ( kinect.isTrackingSkeleton(userId)) { // make a vector to store the left hand PVector rightHand = new PVector(); // put the position of the left hand into that vector kinect.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_LEFT_HAND, rightHand); ellipse(rightHand.x, rightHand.y, 10, 10); Tuesday, April 17, 12
38. Tuesday, April 17, 12
39. Tuesday, April 17, 12
40. Basic Vector Math rightHand.dist(leftHand) This is the source of the simplest interaction Tuesday, April 17, 12
41. Tuesday, April 17, 12
42. What makes a good gestural interface? Tuesday, April 17, 12
43. Kinect Abnormal Motion Assessment System Involuntary Motion Disorders like Chorea van Sydenham, Tardive Dyskinesia, and Parkinson’s motionassessment.com Tuesday, April 17, 12
44. Kinect Abnormal Motion Assessment System Involuntary Motion Disorders like Chorea van Sydenham, Tardive Dyskinesia, and Parkinson’s motionassessment.com Tuesday, April 17, 12
45. Kinect Abnormal Motion Assessment System Won the National Health 2.0 Developer Challenge. Maybe improve treatment. motionassessment.com Tuesday, April 17, 12
46. What do you want to make? Tuesday, April 17, 12
47. http://shop.oreilly.com/product/0636920020684.do https://github.com/atduskgreg/making-things-see-examples Tuesday, April 17, 12
48. Greg Borenstein @atduskgreg http://gregborenstein.com Thanks! Tuesday, April 17, 12