Sonic Gesture

Sonic Gesture is a Computer Vision project where learned hand poses are translated into corresponding commands with the aim of generating sound. In other words this software makes it possible to create and manipulating sound without physical interaction with a computer.

Transcript

1. SONIC GESTURE Gijs Molenaar Universiteit van Amsterdam Supervised by Ivo Everts and Theo Gevers
2. FLUTTER
4. $40 million ! How I not became a millionaire ;) ! Sonic Gesture
5. WHAT IS IT? ! ! ! Translate hand poses into music Using computer vision
6. DEMO
7. WHY? Computer Vision Music Translate Sign language into sound
8. SIGN LANGUAGE No existing mapping Ambiguous Segmentation difficult
9. HAND POSES Static hand poses, no gestures Movement still usable
10. WHAT HAND POSES? Curwen hand poses Do Re Mi Fa Sol La Ti Di Ri Fi Si Li
11. SOLFEGE
12. CLOSE ENCOUNTER
13. THE DATASET 2 times all Curwen hand poses plus 4 easy to detect hand poses
14. GOAL Real time hand pose detection Normal computer hardware (RGB camera) Non-intrusive No calibration
15. ASSUMPTIONS One person in image Facing camera ‘good enough lightning’ Subject is caucasian Long sleeves
16. SKIN
17. FINDING HANDS Difficult Hands change all the time Skin does not change Use skin color to find hands How to find skin color?
18. FACE DETECTION Face is easy to detect Does not change (very much) Assumes only one person in image
19. PROBLEM Light conditions ! Lighting not uniform ! Shadows
20. COLOR SPACE
22. HISTOGRAM Count Color pixels in face Build a histogram
23. BACKPROJECTION + =
24. TRESHOLD →
25. WHAT IS WHAT?
26. LABELING
27. POSE RECOGNITION
28. POSE RECOGNITION We know where is the hand But what is it?
29. TRAIN DATA ! Compare to previously recorded hands ‘train data’ Gathered 2072 images from 20 people
32. HOW TO COMPARE? A lot of data Extract features Features represent image in compact way
33. HISTOGRAM OF ORIENTED GRADIENTS ! ! ! ! ! result is a list of numbers
34. DIMENSIONS List of numbers +/- 3000 in our case Coordinate in higher dimensional space 3000 dimensional space
35. N-D 2D 3D 3000D ?
36. CLUSTERS Similar hands are close in feature space Use train data to classify new data
37. CLASSIFIER K nearest neighbours Too slow with huge dataset
38. CLASSIFIER Support Vector Machines ! Calculates decision Boundary ! Move calculation to training phase
39. RESULTS
40. DATASET Measure classification performance n-fold tests ! Dataset is diverse, 55% smooth 45% complex background
41. PERFORMANCE overall best results HOG features SVM classifier
 28 hands 15 hands train with end- user 87,92% 91,58% train without end-user 71,83% 80,07% without end- user, noisy background 63,81% 74,71%
42. CONCLUSIONS It works Fast, usable in real-time setting Curwen hand poses not suited Method not person independent
43. SONIC GESTURE Open Source Recognises 28 hand poses for each hand Size and Position of hands Open Sound Control (OSC) User defined mapping Google for ‘Sonic Gesture’
44. OTHER APPLICATIONS
47. DEMO
48. QUESTIONS