Data Miming: Inferring Spatial Object Descriptions from Human Gesture

Speakers often use hand gestures when talking about or describing physical objects. Such gesture is particularly useful when the speaker is conveying distinctions of shape that are difficult to describe verbally. We present data miming—an approach to making sense of gestures as they are used to describe concrete physical objects. We first observe participants as they use gestures to describe real-world objects to another person. From these observations, we derive the data miming approach, which is based on a voxel representation of the space traced by the speaker’s hands over the duration of the gesture. In a final proof-of-concept study, we demonstrate a prototype implementation of matching the input voxel representation to select among a database of known physical objects.

More information on http://www.christianholz.net/data_miming.html

Transcript

1. Data Miming Inferring Spatial Object Descriptions From Human Gesture Christian Holz Hasso Plattner Institute Andrew D. Wilson Microsoft Research
5. so we built a system that does that
9. also with more
10. 3 2 1 which gestures do people use? implementation using kinect evaluation
11. observing gestures
13. we had 4 general insights
14. 1 abstraction towards primitives
15. 1 abstraction towards primitives
16. 1 abstraction towards primitives
17. 2 large & defining parts first
18. 2 large & defining parts first, top-down approach
19. 2 large & defining parts first, top-down approach
20. 2 large & defining parts first
21. 3 then small parts, if at all
22. transitioning 4 hard to tell meaningful from random actively describing
23. pauses? motions? 4 hard to tell meaningful from random
24. redundant ! probably meaningful slow ! probably meaningful 4 hard to tell meaningful from random
25. implementation using kinect
26. summary
27. summary Data Miming doesn’t recognize gestures.
28. summary Data Miming doesn’t recognize gestures, Data Miming matches as a whole.
30. virtual representation voxel space := memory of the system
33. volume from occluded parts extrude down large surface = little occlusion occlusion
34. occlusion small surface = much occlusion
35. enclosed regions
36. enclosed regions [Wilson ’06]
37. enclosed regions [Wilson ’06]
38. enclosed regions
39. depth to voxel space
40. voxel space
41. matching of course, our models need to be abstract, too.
45. iterative closest points [Zhang ’94] matching
46. matching
47. matching
48. evaluation
49. participant experimenter
50. participant experimenter projection
52. task describe to the experimenter...
53. ...office furniture
54. ...chairs
55. ...solids
56. ...parallelepiped
57. design 2 walk-up use + 4 instructed use × 4 categories = 24 trials / participant 15 participants
58. “ stand on the X. you will see an object, please describe it to the experimenter through gestures. the experimenter will guess the object from your description, so please describe it carefully. walk-up session
59. !""# 48% results
60. !""# top match closest 3 82% results 48%
61. “ describe using deliberate and careful gestures. don’t bend your head over, keep your hands in the capture volume. use flat hands to indicate surfaces, use fists or pinches to describe struts if necessary instructed session
62. top match closest 3 !""# 76% 99% results
63. top match !""# 61% 85% 66% 85%
64. !""# closest 3 95% 93%
65. conclusions
66. results support the Data Miming approach beyond retrieval configuring objects conclusions
67. gesture recognizers? gesture input vocabularies! but, maybe out of the box? Data Miming is a deliberate step away less need for training conclusions
68. Hrvoje Benko Ken Hinckley Johnny Lee Steven Feiner Ricardo Jota, David Holman Patrick Baudisch all participants in the study thanks to
69. Data Miming christian holz | andrew d. wilson http://www.christianholz.net http://research.microsoft.com/~awilson