Fitness Motion Recognition with Android Wear

Counting is what computers do best and we should let them do it whenever possible. That's true for spreadsheets and it's true for fitness. Wearables exist to count your steps, measure the distance you run, and track how your pulse races after a workout. So why are we still counting pushups, situps, and burpees like they did in the Stone Age?

In this presentation, I will talk about the steps necessary to implement this kind of motion recognition on Android Wear:

- Measuring the motion being recorded by the device
- Deriving a pattern that represents the motion you want to recognize
- Implementing the pattern recognition in the most battery-efficient manner possible

Transcript

1. Fitness Motion
   Recognition
   with
   Android Wear
   Edward Dale
   Freeletics
   © Edward Dale, 2015 1
   

   View Slide

2. http://www.someecards.com/usercards/viewcard/MjAxMy1hMjIwMWUzMTc4NDgyOTA1
   © Edward Dale, 2015 2
   

   View Slide

3. Agenda
   • Define scope
   • Sensors
   • Algorithms
   • Battery Efficiency
   © Edward Dale, 2015 3
   

   View Slide

4. Defining the problem scope
   • Segmenting exercise from non-exercise
   • Recognizing which exercise is being performed
   • Counting repetitions
   • Online
   © Edward Dale, 2015 4
   

   View Slide

5. Defining the problem scope
   • Segmenting exercise from non-exercise
   • Recognizing which exercise is being performed
   • Counting repetitions
   • Online
   © Edward Dale, 2015 5
   

   View Slide

6. Defining the problem scope
   • Segmenting exercise from non-exercise
   • Recognizing which exercise is being performed
   • Counting repetitions
   • Online
   © Edward Dale, 2015 6
   

   View Slide

7. Defining the problem scope
   • Segmenting exercise from non-exercise
   • Recognizing which exercise is being performed
   • Counting repetitions !
   • Online
   © Edward Dale, 2015 7
   

   View Slide

8. Defining the problem scope
   • Segmenting exercise from non-exercise
   • Recognizing which exercise is being performed
   • Counting repetitions !
   • Online !
   © Edward Dale, 2015 8
   

   View Slide

9. Which "Fitness Motion"?
   • Running
   • Swimming
   • Cycling
   © Edward Dale, 2015 9
   

   View Slide

10. This "Fitness Motion"
    • Pushups
    • Jumping Jacks
    • Burpees
    © Edward Dale, 2015 10
    

    View Slide

11. Pushups
    © Edward Dale, 2015 11
    

    View Slide

12. © Edward Dale, 2015 12
    

    View Slide

13. Pushup Sensors
    • Proximity (Sensor.TYPE_PROXIMITY)
    © Edward Dale, 2015 13
    

    View Slide

14. Pushup Sensors
    • Proximity (Sensor.TYPE_PROXIMITY)
    • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR,
    Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR,
    Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR)
    © Edward Dale, 2015 14
    

    View Slide

15. Pushup Sensors
    • Proximity (Sensor.TYPE_PROXIMITY)
    • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR,
    Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR,
    Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR)
    • Acceleration (Sensor.TYPE_ACCELEROMETER,
    Sensor.TYPE_LINEAR_ACCELERATION,
    Sensor.TYPE_GRAVITY)
    © Edward Dale, 2015 15
    

    View Slide

16. Pushup Sensors
    • Proximity (Sensor.TYPE_PROXIMITY)
    • Rotation (Sensor.TYPE_GAME_ROTATION_VECTOR,
    Sensor.TYPE_GEOMAGNETIC_ROTATION_VECTOR,
    Sensor.TYPE_GYROSCOPE, Sensor.TYPE_ROTATION_VECTOR)
    • Acceleration (Sensor.TYPE_ACCELEROMETER,
    Sensor.TYPE_LINEAR_ACCELERATION,
    Sensor.TYPE_GRAVITY) !
    © Edward Dale, 2015 16
    

    View Slide

17. Acceleration Sensors
    TYPE_ACCELEROMETER uses the accelerometer and only the
    accelerometer. It returns raw accelerometer events, with minimal
    or no processing at all.
    TYPE_LINEAR_ACCELERATION and TYPE_GRAVITY ... are "fused"
    sensors
    — Mathias Agopian on android-developers
    Always returns 3 components of acceleration vector
    © Edward Dale, 2015 17
    

    View Slide

18. Acceleration Vector
    What to do with acceleration direction?
    Pushup acceleration happens in primary one direction
    Ignore acceleration direction and just use magnitude
    © Edward Dale, 2015 18
    

    View Slide

19. © Edward Dale, 2015 19
    

    View Slide

20. © Edward Dale, 2015 20
    

    View Slide

21. Not so fast
    Still have to count
    But there are well-known algorithms for that
    Google: Online peak detection algorithm
    © Edward Dale, 2015 21
    

    View Slide

22. Peakdet
    A point is considered a maximum peak if it has the maximal value,
    and was preceded (to the left) by a value lower by DELTA.
    -- http://www.billauer.co.il/peakdet.html
    © Edward Dale, 2015 22
    

    View Slide

23. © Edward Dale, 2015 23
    

    View Slide

24. © Edward Dale, 2015 24
    

    View Slide

25. Peakdet
    • Online !
    • Efficient !
    • Sensitive to DELTA parameter "
    © Edward Dale, 2015 25
    

    View Slide

26. Battery Efficiency
    • Analyze fewer samples
    • Do less analysis per sample
    • Analyze sample on the phone
    • Choose less power-hungry sensors
    • Watch the Power Optimization for Android talk from day 1
    © Edward Dale, 2015 26
    

    View Slide

27. Battery Efficiency
    Analyze fewer samples
    • Register for sensor updates with lowest sampling frequency
    necessary
    • SENSOR_DELAY_NORMAL (5Hz)
    • SENSOR_DELAY_UI (15Hz)
    • SENSOR_DELAY_GAME (50Hz)
    • SENSOR_DELAY_FASTEST (~∞Hz)
    © Edward Dale, 2015 27
    

    View Slide

28. Battery Efficiency
    Analyze fewer samples
    • Register for sensor updates with lowest sampling frequency
    necessary
    • Also possible to suggest your own sampling frequency
    • Just a suggestion to the device
    © Edward Dale, 2015 28
    

    View Slide

29. Battery Efficiency
    Do less analysis per sample
    • Choose an efficient algorithm
    • Peakdet is relatively efficient
    • More efficient than algorithms using derivates
    © Edward Dale, 2015 29
    

    View Slide

30. Battery Efficiency
    Analyze samples on the phone
    Just use the watch as a wearable sensor that sends data to be
    analyzed on the phone.
    © Edward Dale, 2015 30
    

    View Slide

31. Battery Efficiency
    Choose Less Power-Hungry Sensors
    • Sensor power drain will differ on different hardware
    • Ask the sensor how much power the sensor uses
    Sensor.getPower()
    © Edward Dale, 2015 31
    

    View Slide

32. Thanks!
    Edward Dale (@scompt)
    Freeletics (We're hiring)
    © Edward Dale, 2015 32
    

    View Slide

33. Links
    • Walk Detection and Step Counting on
    Unconstrained Smartphones
    • RecoFit: Using a Wearable Sensor to
    Find, Recognize, and Count Repetitive Exercises
    • Sample Project
    © Edward Dale, 2015 33
    

    View Slide