Fitness Motion Recognition with Android Wear

Fitness Motion Recognition with Android Wear

Counting is what computers do best and we should let them do it whenever possible. That's true for spreadsheets and it's true for fitness. Wearables exist to count your steps, measure the distance you run, and track how your pulse races after a workout. So why are we still counting pushups, situps, and burpees like they did in the Stone Age?

In this presentation, I will talk about the steps necessary to implement this kind of motion recognition on Android Wear:

- Measuring the motion being recorded by the device
- Deriving a pattern that represents the motion you want to recognize
- Implementing the pattern recognition in the most battery-efficient manner possible

730df0227780e818df8ce1e19c9a6c48?s=128

Edward Dale

June 05, 2015
Tweet

Transcript

  1. Fitness Motion Recognition with Android Wear Edward Dale Freeletics ©

    Edward Dale, 2015 1
  2. http://www.someecards.com/usercards/viewcard/MjAxMy1hMjIwMWUzMTc4NDgyOTA1 © Edward Dale, 2015 2

  3. Agenda • Define scope • Sensors • Algorithms • Battery

    Efficiency © Edward Dale, 2015 3
  4. Defining the problem scope • Segmenting exercise from non-exercise •

    Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 4
  5. Defining the problem scope • Segmenting exercise from non-exercise •

    Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 5
  6. Defining the problem scope • Segmenting exercise from non-exercise •

    Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 6
  7. Defining the problem scope • Segmenting exercise from non-exercise •

    Recognizing which exercise is being performed • Counting repetitions ! • Online © Edward Dale, 2015 7
  8. Defining the problem scope • Segmenting exercise from non-exercise •

    Recognizing which exercise is being performed • Counting repetitions ! • Online ! © Edward Dale, 2015 8
  9. Which "Fitness Motion"? • Running • Swimming • Cycling ©

    Edward Dale, 2015 9
  10. This "Fitness Motion" • Pushups • Jumping Jacks • Burpees

    © Edward Dale, 2015 10
  11. Pushups © Edward Dale, 2015 11

  12. © Edward Dale, 2015 12

  13. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) © Edward Dale, 2015 13

  14. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE,

    Sensor.TYPE_ROTATION_VECTOR) © Edward Dale, 2015 14
  15. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE,

    Sensor.TYPE_ROTATION_VECTOR) • Acceleration (Sensor.TYPE_ACCELEROMETER, Sensor.TYPE_LINEAR_ACCELERATION, Sensor.TYPE_GRAVITY) © Edward Dale, 2015 15
  16. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE,

    Sensor.TYPE_ROTATION_VECTOR) • Acceleration (Sensor.TYPE_ACCELEROMETER, Sensor.TYPE_LINEAR_ACCELERATION, Sensor.TYPE_GRAVITY) ! © Edward Dale, 2015 16
  17. Acceleration Sensors TYPE_ACCELEROMETER uses the accelerometer and only the accelerometer.

    It returns raw accelerometer events, with minimal or no processing at all. TYPE_LINEAR_ACCELERATION and TYPE_GRAVITY ... are "fused" sensors — Mathias Agopian on android-developers Always returns 3 components of acceleration vector © Edward Dale, 2015 17
  18. Acceleration Vector What to do with acceleration direction? Pushup acceleration

    happens in primary one direction Ignore acceleration direction and just use magnitude © Edward Dale, 2015 18
  19. © Edward Dale, 2015 19

  20. © Edward Dale, 2015 20

  21. Not so fast Still have to count But there are

    well-known algorithms for that Google: Online peak detection algorithm © Edward Dale, 2015 21
  22. Peakdet A point is considered a maximum peak if it

    has the maximal value, and was preceded (to the left) by a value lower by DELTA. -- http://www.billauer.co.il/peakdet.html © Edward Dale, 2015 22
  23. © Edward Dale, 2015 23

  24. © Edward Dale, 2015 24

  25. Peakdet • Online ! • Efficient ! • Sensitive to

    DELTA parameter " © Edward Dale, 2015 25
  26. Battery Efficiency • Analyze fewer samples • Do less analysis

    per sample • Analyze sample on the phone • Choose less power-hungry sensors • Watch the Power Optimization for Android talk from day 1 © Edward Dale, 2015 26
  27. Battery Efficiency Analyze fewer samples • Register for sensor updates

    with lowest sampling frequency necessary • SENSOR_DELAY_NORMAL (5Hz) • SENSOR_DELAY_UI (15Hz) • SENSOR_DELAY_GAME (50Hz) • SENSOR_DELAY_FASTEST (~∞Hz) © Edward Dale, 2015 27
  28. Battery Efficiency Analyze fewer samples • Register for sensor updates

    with lowest sampling frequency necessary • Also possible to suggest your own sampling frequency • Just a suggestion to the device © Edward Dale, 2015 28
  29. Battery Efficiency Do less analysis per sample • Choose an

    efficient algorithm • Peakdet is relatively efficient • More efficient than algorithms using derivates © Edward Dale, 2015 29
  30. Battery Efficiency Analyze samples on the phone Just use the

    watch as a wearable sensor that sends data to be analyzed on the phone. © Edward Dale, 2015 30
  31. Battery Efficiency Choose Less Power-Hungry Sensors • Sensor power drain

    will differ on different hardware • Ask the sensor how much power the sensor uses Sensor.getPower() © Edward Dale, 2015 31
  32. Thanks! Edward Dale (@scompt) Freeletics (We're hiring) © Edward Dale,

    2015 32
  33. Links • Walk Detection and Step Counting on Unconstrained Smartphones

    • RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises • Sample Project © Edward Dale, 2015 33