How to use a Kinect to play Portal 2

Women Who Code Lightning Talks at Parisoma. Tuesday Oct. 25, 2011. @judytuna showed us how to use Kinect + FAAST + OpenNI to map body movements to key presses!

Transcript

1. How to use a Kinect to play Portal2 Judy Tuan @judytuna
2. What is the Kinect? • Microsoft peripheral for xbox360 • Depth sensor! • Released in NA on Nov. 4, 2010 • Immediately “hacked” by Nov. 10, 2010 – http://www.ladyada.net/learn/diykinect/ – Reverse-engineered the USB communication so we can use the signals that the Kinect puts out – Not really “hacked” but everyone still uses the word
3. How does the Kinect work? • Image source: https://secure.wikimedia.org/wikipedia/en/wiki/Kinect
4. How does the Kinect work? • Microphone – Dedicated hardware that filters out noise • Video camera – 640×480 pixels @ 30 Hz (RGB camera) • IR projector – Fires a lot of IR dots • IR sensor – Calculates the distance between dots to find out how close an object is
5. Photo using an infrared camera by audreypenven on flickr: https://secure.flickr.com/photos/audreypenven/5197993474/
6. Why “hack” it? • Natural User Interface – Interact with a computer with your body and voice, instead of mouse/keyboard • Medicine... – Allows a doctor to look through documents without touching anything • Fun – Play existing games! (Minecraft/WoW!) – Lightsaber! – Piano on any surface... and many more
7. Connecting to a computer • A Kinect alone is $150, which some people call “low cost” for its technology • IF you get it bundled with a new xbox360, you'll need a USB dongle+power supply – http://www.microsoftstore.com/store/msst ore/en_US/pd/productID.221244000 • Your computer needs to know what to do with the data streams...
8. Some Frameworks • Microsoft's official SDK – https://research.microsoft.com/en- us/um/redmond/projects/kinectsdk/ – Windows-only • OpenKinect project! – http://openkinect.org/ + mailing list – Windows, Mac, and Linux • FAAST: Flexible Action and Articulated Skeleton Toolkit – http://projects.ict.usc.edu/mxr/faast/ – Windows-only – Uses config file. No coding required!
9. FAAST! • Map right hand to mouse [Mouse] mouse_enabled 1 mouse_control 1 mouse_body_part 1 mouse_origin 1 mouse_left_bound 12 mouse_right_bound 12 mouse_bottom_bound 12 mouse_top_bound 12 mouse_forward_threshold 12 mouse_relative_speed 25 mouse_movement_threshold 2 mouse_multiple_monitors 0
10. FAAST! • Map body movements to keyboard [Actions] # mappings from input events to output events # format: event_name threshold output_type event lean_left 15 key_hold a lean_right 15 key_hold d lean_forwards 15 key_hold w lean_backwards 15 key_hold s left_arm_up 15 key_press space left_arm_forwards 15 key_hold e left_arm_out 15 mouse_click left_button left_arm_across 15 mouse_click right_button left_foot_sideways 24 faast stop
11. What I'd love to do • Learn to write Chinese by tracing characters with my hand • Become a blue alien • What awesome apps can you think of? =D