Fitness Motion Recognition with Android Wear

Counting is what computers do best and we should let them do it whenever possible. That's true for spreadsheets and it's true for fitness. Wearables exist to count your steps, measure the distance you run, and track how your pulse races after a workout. So why are we still counting pushups, situps, and burpees like they did in the Stone Age?

In this presentation, I will talk about the steps necessary to implement this kind of motion recognition on Android Wear:

- Measuring the motion being recorded by the device
- Deriving a pattern that represents the motion you want to recognize
- Implementing the pattern recognition in the most battery-efficient manner possible

Transcript

1. Fitness Motion Recognition with Android Wear Edward Dale Freeletics © Edward Dale, 2015 1
2. http://www.someecards.com/usercards/viewcard/MjAxMy1hMjIwMWUzMTc4NDgyOTA1 © Edward Dale, 2015 2
3. Agenda • Define scope • Sensors • Algorithms • Battery Efficiency © Edward Dale, 2015 3
4. Defining the problem scope • Segmenting exercise from non-exercise • Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 4
5. Defining the problem scope • Segmenting exercise from non-exercise • Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 5
6. Defining the problem scope • Segmenting exercise from non-exercise • Recognizing which exercise is being performed • Counting repetitions • Online © Edward Dale, 2015 6
7. Defining the problem scope • Segmenting exercise from non-exercise • Recognizing which exercise is being performed • Counting repetitions ! • Online © Edward Dale, 2015 7
8. Defining the problem scope • Segmenting exercise from non-exercise • Recognizing which exercise is being performed • Counting repetitions ! • Online ! © Edward Dale, 2015 8
9. Which "Fitness Motion"? • Running • Swimming • Cycling © Edward Dale, 2015 9
10. This "Fitness Motion" • Pushups • Jumping Jacks • Burpees © Edward Dale, 2015 10
11. Pushups © Edward Dale, 2015 11
12. © Edward Dale, 2015 12
13. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) © Edward Dale, 2015 13
14. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR) © Edward Dale, 2015 14
15. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR) • Acceleration (Sensor.TYPE_ACCELEROMETER, Sensor.TYPE_LINEAR_ACCELERATION, Sensor.TYPE_GRAVITY) © Edward Dale, 2015 15
16. Pushup Sensors • Proximity (Sensor.TYPE_PROXIMITY) • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR, Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR, Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR) • Acceleration (Sensor.TYPE_ACCELEROMETER, Sensor.TYPE_LINEAR_ACCELERATION, Sensor.TYPE_GRAVITY) ! © Edward Dale, 2015 16
17. Acceleration Sensors TYPE_ACCELEROMETER uses the accelerometer and only the accelerometer. It returns raw accelerometer events, with minimal or no processing at all. TYPE_LINEAR_ACCELERATION and TYPE_GRAVITY ... are "fused" sensors — Mathias Agopian on android-developers Always returns 3 components of acceleration vector © Edward Dale, 2015 17
18. Acceleration Vector What to do with acceleration direction? Pushup acceleration happens in primary one direction Ignore acceleration direction and just use magnitude © Edward Dale, 2015 18
19. © Edward Dale, 2015 19
20. © Edward Dale, 2015 20
21. Not so fast Still have to count But there are well-known algorithms for that Google: Online peak detection algorithm © Edward Dale, 2015 21
22. Peakdet A point is considered a maximum peak if it has the maximal value, and was preceded (to the left) by a value lower by DELTA. -- http://www.billauer.co.il/peakdet.html © Edward Dale, 2015 22
23. © Edward Dale, 2015 23
24. © Edward Dale, 2015 24
25. Peakdet • Online ! • Efficient ! • Sensitive to DELTA parameter " © Edward Dale, 2015 25
26. Battery Efficiency • Analyze fewer samples • Do less analysis per sample • Analyze sample on the phone • Choose less power-hungry sensors • Watch the Power Optimization for Android talk from day 1 © Edward Dale, 2015 26
27. Battery Efficiency Analyze fewer samples • Register for sensor updates with lowest sampling frequency necessary • SENSOR_DELAY_NORMAL (5Hz) • SENSOR_DELAY_UI (15Hz) • SENSOR_DELAY_GAME (50Hz) • SENSOR_DELAY_FASTEST (~∞Hz) © Edward Dale, 2015 27
28. Battery Efficiency Analyze fewer samples • Register for sensor updates with lowest sampling frequency necessary • Also possible to suggest your own sampling frequency • Just a suggestion to the device © Edward Dale, 2015 28
29. Battery Efficiency Do less analysis per sample • Choose an efficient algorithm • Peakdet is relatively efficient • More efficient than algorithms using derivates © Edward Dale, 2015 29
30. Battery Efficiency Analyze samples on the phone Just use the watch as a wearable sensor that sends data to be analyzed on the phone. © Edward Dale, 2015 30
31. Battery Efficiency Choose Less Power-Hungry Sensors • Sensor power drain will differ on different hardware • Ask the sensor how much power the sensor uses Sensor.getPower() © Edward Dale, 2015 31
32. Thanks! Edward Dale (@scompt) Freeletics (We're hiring) © Edward Dale, 2015 32
33. Links • Walk Detection and Step Counting on Unconstrained Smartphones • RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises • Sample Project © Edward Dale, 2015 33