Pro Yearly is on sale from $80 to $50! »

Data Miming: Inferring Spatial Object Descriptions from Human Gesture

Data Miming: Inferring Spatial Object Descriptions from Human Gesture

Speakers often use hand gestures when talking about or describing physical objects. Such gesture is particularly useful when the speaker is conveying distinctions of shape that are difficult to describe verbally. We present data miming—an approach to making sense of gestures as they are used to describe concrete physical objects. We first observe participants as they use gestures to describe real-world objects to another person. From these observations, we derive the data miming approach, which is based on a voxel representation of the space traced by the speaker’s hands over the duration of the gesture. In a final proof-of-concept study, we demonstrate a prototype implementation of matching the input voxel representation to select among a database of known physical objects.

More information on http://www.christianholz.net/data_miming.html

1b1d5420dfff84bf061b2cc53e9b839b?s=128

Christian Holz

May 09, 2011
Tweet

Transcript

  1. Data Miming Inferring Spatial Object Descriptions From Human Gesture Christian

    Holz Hasso Plattner Institute Andrew D. Wilson Microsoft Research
  2. None
  3. None
  4. None
  5. so we built a system that does that

  6. None
  7. None
  8. None
  9. also with more

  10. 3 2 1 which gestures do people use? implementation using

    kinect evaluation
  11. observing gestures

  12. None
  13. we had 4 general insights

  14. 1 abstraction towards primitives

  15. 1 abstraction towards primitives

  16. 1 abstraction towards primitives

  17. 2 large & defining parts first

  18. 2 large & defining parts first, top-down approach

  19. 2 large & defining parts first, top-down approach

  20. 2 large & defining parts first

  21. 3 then small parts, if at all

  22. transitioning 4 hard to tell meaningful from random actively describing

  23. pauses? motions? 4 hard to tell meaningful from random

  24. redundant ! probably meaningful slow ! probably meaningful 4 hard

    to tell meaningful from random
  25. implementation using kinect

  26. summary

  27. summary Data Miming doesn’t recognize gestures.

  28. summary Data Miming doesn’t recognize gestures, Data Miming matches as

    a whole.
  29. None
  30. virtual representation voxel space := memory of the system

  31. None
  32. None
  33. volume from occluded parts extrude down large surface = little

    occlusion occlusion
  34. occlusion small surface = much occlusion

  35. enclosed regions

  36. enclosed regions [Wilson ’06]

  37. enclosed regions [Wilson ’06]

  38. enclosed regions

  39. depth to voxel space

  40. voxel space

  41. matching of course, our models need to be abstract, too.

  42. None
  43. None
  44. None
  45. iterative closest points [Zhang ’94] matching

  46. matching

  47. matching

  48. evaluation

  49. participant experimenter

  50. participant experimenter projection

  51. None
  52. task describe to the experimenter...

  53. ...office furniture

  54. ...chairs

  55. ...solids

  56. ...parallelepiped

  57. design 2 walk-up use + 4 instructed use × 4

    categories = 24 trials / participant 15 participants
  58. “ stand on the X. you will see an object,

    please describe it to the experimenter through gestures. the experimenter will guess the object from your description, so please describe it carefully. walk-up session
  59. !""# 48% results

  60. !""# top match closest 3 82% results 48%

  61. “ describe using deliberate and careful gestures. don’t bend your

    head over, keep your hands in the capture volume. use flat hands to indicate surfaces, use fists or pinches to describe struts if necessary instructed session
  62. top match closest 3 !""# 76% 99% results

  63. top match !""# 61% 85% 66% 85%

  64. !""# closest 3 95% 93%

  65. conclusions

  66. results support the Data Miming approach beyond retrieval configuring objects

    conclusions
  67. gesture recognizers? gesture input vocabularies! but, maybe out of the

    box? Data Miming is a deliberate step away less need for training conclusions
  68. Hrvoje Benko Ken Hinckley Johnny Lee Steven Feiner Ricardo Jota,

    David Holman Patrick Baudisch all participants in the study thanks to
  69. Data Miming christian holz | andrew d. wilson http://www.christianholz.net http://research.microsoft.com/~awilson