Compensating for On-Body Placement Effects in Activity Recognition

F44f01b6972482deceb7c0383ce49055?s=47 Kai
August 01, 2011

Compensating for On-Body Placement Effects in Activity Recognition

My phD. thesis presentation.

Abstract:
This thesis investigates, how placement variations of electronic devices influence the possibility of using sensors integrated in those devices for context recognition. The vast majority of context recognition research assumes well defined, fixed sen- sor locations. Although this might be acceptable for some application domains (e.g. in an industrial setting), users, in general, will have a hard time coping with these limitations. If one needs to remember to carry dedicated sensors and to adjust their orientation from time to time, the activity recognition system is more distracting than helpful. How can we deal with device location and orientation changes to make context sensing mainstream? This thesis presents a systematic evaluation of device placement effects in context recognition.

full thesis pdf available here: http://www.opus-bayern.de/uni-passau/volltexte/2012/2611/

latex sources are on github:
http://github.com/kkai/phdthesis

F44f01b6972482deceb7c0383ce49055?s=128

Kai

August 01, 2011
Tweet

Transcript

  1. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Compensating for On-Body Placement Effects in Activity Recognition Kai Kunze
  2. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Compensating for On-Body Placement Effects in Activity Recognition Kai Kunze
  3. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

  4. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

  5. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    A Zeiss HMD Study Kunze, K.,Wagner, F., Kartal, E., Morales Kluge, E., and Lukowicz, P. Does Context Matter ? - A Quantitative Evaluation in a Real World Maintenance Scenario. In Proceedings of the 7th international Conference on Pervasive Computing Nara, Japan, May 11 - 14, 2009.
  6. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Applications
  7. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    ... using environment and onbody sensors lower arm trouser pocket upper arm
  8. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Compensating for On-Body Placement Effects in Activity Recognition Kai Kunze
  9. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling Kunze, K. and Lukowicz, P. Symbolic object localization through active sampling of acceleration and sound signatures. In Proceedings of the 9th international Conference on Ubiquitous Computing. Innsbruck, Austria, September 16 - 19, 2007. nominated for best paper. (Acceptance rate: 14%)
  10. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Environmental Placement Detection -Active Sampling- § up to 96 % per room § up to 92 % for abstract classes
  11. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling K. Kunze and P. Lukowicz. Using acceleration signatures from everyday activities for on- body device location. 11th IEEE International Symposium on Wearable Computers, Sep 2007. K. Kunze, P. Lukowicz, H. Junker, and G. Troester. Where am i: Recognizing on- body positions of wearable sensors. LOCA’04: International Workshop on Location and Context Awareness , Jan 2005. yes Can we recognize the body part? On-body Placement Recognition
  12. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling Kunze, K. and Lukowicz, P. Dealing with sensor displacement in motion-based on-body activity recognition systems. In Proceedings of the 10th international con- ference on Ubiquitous computing (UbiComp ’08). Seoul, Korea, September, 2008. yes Can we recognize the body part? On-body Placement Recognition Is it displaced? no yes compensate use as trained Heuristics for Displacement
  13. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling Kai Kunze, Paul Lukowicz, Kurt Partridge, Bo Begole, Which Way Am I Facing: Inferring Horizontal Device Orientation from an Accelerometer Signal, 13th IEEE International Symposium on Wearable Computers. Linz, Austria, 2009. yes Can we recognize the body part? On-body Placement Recognition Is it displaced? no yes compensate use as trained Heuristics for Displacement Did the orientation change? yes no infere orientation Orientation Recognition while Walking
  14. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling yes Can we recognize the body part? On-body Placement Recognition Is it displaced? no yes compensate use as trained Heuristics for Displacement Did the orientation change? yes no infere orientation Orientation Recognition while Walking
  15. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling yes Can we recognize the body part? Is it displaced? no yes compensate use as trained Heuristics for Displacement Did the orientation change? yes no infere orientation Orientation Recognition while Walking On-body Placement Recognition
  16. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Related Work § first dealing with this problem § other work so far is supplementary Jonathan Lester, Tanzeem Choudhury, and Gaetano Borriello. A Practical Approach to Recognizing Physical Activities. Appears in the Proceedings of Pervasive. May 2006, Dublin, Ireland. Beverly Harrison, Sunny Consolvo, and Tanzeem Choudhury. Using Multi-modal Sensing for Human Activity Modeling in the Real World. Appears in the Handbook of Ambient Intelligence and Smart Environments, Springer Verlag 2009.
  17. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    On-Body Location Contributions § recognition architecture § basic approaches: walking/unconstraint § feature selection § algorithms § experimental evaluation
  18. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Walking lower arm trouser pocket
  19. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Method Using Walking walking? frame-by-frame walking recognition yes frame-by-frame placement majority decison no majority decison feature extraction 1 sec. sliding win § 1 sec. sliding window feature extraction § frame-by-frame walking classification § with penalty on false positives § best C4.5 (decision tree classifier) § 10 sec. jumping window majority decision § 6 Placement Features: § RMS § 75% Percentile § Inter Quartile Range § Frequency Range Power § Frequency Entropy § Sums Power Wave Det Coefficient
  20. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

  21. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Unconstraint Onbody Placement Recognition § rest periods need to be filtered out § “carrier” frequency is gone § probabilities of distinct movements for a given body part differ greatly § time-series approach necessary § smoothing § majority decision too crude § stochastic filtering needed
  22. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Time Series Approach: Hidden Markov Models § sequence of features as observations § hidden states with transitions represent the distinct movements and their sequence § train one for each location § 3-8 hidden states ... s1 s2 sn s3 ... s1 s2 sn s3 0 20 40 60 80 0.00 0.02 0.04 0.06 Heming Lake Pike: Distribution by Age Groups Length [cm] Probability Density 0 20 40 60 80 −0.005 0.000 0.005 0.010 1st Derivative x y' • • • • • • • • 0 20 40 60 80 −0.006 −0.002 0.002 2nd Derivative x y'' • • • • • • • • • • • observations hidden states transition probabilities ... s1 s2 sn s3 ... s1 s2 sn s3
  23. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Smoothing: Stochastic Filtering § stochastic filtering § 2 general approaches: § kalman (linear model, gaussian noise) § particle (arbitrary model and distribution) § posteriori using monte carlo estimation § particles hold the on-body location § strong prior on holding the state (95 %) § “measurements”: HMM classifications § no. of particles for tracking: 30 -120 § estimator: weighted average over particles numbers N. p(xk|y1:k 1) = f (xk|xk 1)p(xk 1|y1:k 1) dxk p(xk|y1:k) = g(yk|xk)p(xk|y1:k 1) p(yk|y1:k 1) where : p(yk|y1:k 1) = g(yk|xk)p(xk|y1:k 1)f (xk|xk f (xk)p(xk|y1:k)dxk ⇤ 1 N N i=1 wi f (xk,i) To reach a good estimate the particle filter performs iterat sampling steps given subsequently. 1. draw N particles from the proposed sampling distribution st ⇥ (xt|st 1 , yt) for t = 1 to k do 2. compute and normalize the importance weight updat surement yt according to: wi,t = wi,t 1 p(yt|st)p(st|st 1) (st|st 1 ,yt)
  24. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Unconstrained Method § acceleration features: § std, mean, fft center of mass, duration of rest period § gyro features: § pca angle, frequency range power § both: § sum of the differences in variance per axis calibration rest period? feature extraction particle filter smoothing no yes HMM classification acceleration angular velocity on this vertical axis and the norm vector of the if not indicated otherwise with the feature. A the length of the last calibration/ rest period. Our initial approach was to use a mixture the frame-by-frame case, presented in 3.1. Yet, the following, we describe the features we calc Table 3.2: Features used for the Hi Accelerometer Gyrosco standard deviation and mean PCA an fft center of mass frequen duration of the last rest period (below The sum of the norm of the differenc malized axes a1 , a2 , a3 divided by the v 1/2 n i=1 n j=1,j<i | var(ai) var(aj) | var(norm)
  25. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Experimental Evaluation § 5 data sets § house work to bicycle repair § 3 to 7 participants per data set § 1 real life data set § age range 17 - over 60 § 4-5 on-body placements ETH material 1. 5x Xsens (according to experiment description). 2. Xbus: S/N#: 00130157 (incl. Serial cable + USB converter). 3. 5x Xsens strap 4. 5x Xsens cable 5. Motion jacket around 30 hours of sensor data
  26. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Walking Method: Results § 2 data sets § contain sufficiently long walking segments § Evaluation: § 33/66 % training /testing § 10 fold cross validation § walking detection close to ~100 % § placement detection over 90 % 0 20 40 60 80 100 Nearest Neighbour C 4.5 Naive Bayes Simple Naive Bayes correctly classified in percent Classfication Algorithm rl fbf fbf fbf ws bs 0 20 40 60 80 100 N C 4.5 Naive Bayes Simple Naive Bayes correctly classified in percent Classfication Algorithm
  27. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    HMM Method without Particle Filtering § Evaluation: 33% training, 66 %testing 3. On-Body Placement Table 3.7: HMM Overview for several segment sizes Set / time 30 sec. 45 sec. 1 min. 3 min. 4 min. 5 min. bicycle 43 % 67 % - 83 % 83 % 84 % house 32 % 65 % 68 % 73 % 82 % 79 % opp. (accel) 20 % 59 % - 69 % 80 % 82 % drink and work 15 % 61 % - - 72 % 78 % Event Based Location Recognition In a final step majority decision was per- ormed in each segment leading to an event based recognition. Just like in the manually segmented case the recognition rate was 100 %. We have introduced a method that allows us to recognize where on the user’s body an acceleration sensor is located. The experimental results presented above ndicate that the method produces surprisingly reliable results. The method has ement Table 3.7: HMM Overview for several segment sizes time 30 sec. 45 sec. 1 min. 3 min. 4 min. 5 min. cycle 43 % 67 % - 83 % 83 % 84 % ouse 32 % 65 % 68 % 73 % 82 % 79 % ccel) 20 % 59 % - 69 % 80 % 82 % work 15 % 61 % - - 72 % 78 % ation Recognition In a final step majority decision was per- egment leading to an event based recognition. Just like in the nted case the recognition rate was 100 %. duced a method that allows us to recognize where on the user’s tion sensor is located. The experimental results presented above method produces surprisingly reliable results. The method has Confusion matrix for house work (4 min.)
  28. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Particle Filtering Evaluation § need placement switches for evaluation § approach: § take certain size segments from different placements and combine them. § no “switching” movement § NOT indicative for new location § makes the inference more difficult § for evaluation: § 100 segments picked randomly from training data. § approx. 10 min. long § stitched together 0 100 200 300 400 500 600 700 −60 −40 −20 0 20 40 60 0 100 200 300 400 500 600 700 −60 −40 −20 0 20 40 60 0 100 200 300 400 500 600 700 −60 −40 −20 0 20 40 60 0 100 200 300 400 500 600 700 −60 −40 −20 0 20 40 60 0 100 200 300 400 500 600 700 −60 −40 −20 0 20 40 60 hand switch pocket                        
  29. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Smoothing: Particle Filtering Time→ Classes
  30. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Results: Particle Filtering
  31. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    On-body Summary Achievements + large scale evaluation ~30 hours of different activities/ people variety of placements + universal architecture Limitations -takes relative long to stabilize alright for usage -costly to compute
  32. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling yes Can we recognize the body part? On-body Placement Recognition Is it displaced? no yes compensate use as trained Heuristics for Displacement Did the orientation change? yes no infere orientation Orientation Recognition while Walking
  33. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Overview and Contributions Is the device on the body? no Can we determine its placement in the environment? Environmental Placement: Active Sampling yes Can we recognize the body part? On-body Placement Recognition Is it displaced? no yes compensate use as trained Did the orientation change? yes no infere orientation Orientation Recognition while Walking Heuristics for Displacement
  34. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    The issue
  35. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    The Impact Gym Exercises Modality Acceleration Gyroscope Recognition algorithm trained on sensor 1 evaluated on sensor 1 100% 80% 63% 72% evaluated on displaced sensor
  36. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Related Work 4.8. Bibliography [5] K. Forster, D. Roggen, and G. Troster. Unsupervised classifier self-calibration through repeated context occurences: Is there robustness against sensor dis- placement to gain? In Wearable Computers, 2009. ISWC ’09. International Sym- posium on, pages 77 –84, 2009. [6] N. Kern, B. Schiele, H. Junker, P. Lukowicz, and G. Tröster. Wearable sensing to annotate meeting recordings. Personal and Ubiquitous Computing, 7(5):263– Device orientation is one of the major variations in sensor signals for context recognition. Compared to displacement and on-body placement, however, it is a bit better understood, as shown in the related work section 5.2. Large changes in device orientation with respect to the user’s body can influence the signal quality of sound and radio signals, if the microphone/antenna has strong directional characteristics. With respect to motion sensors, the vector norm can be used as an orientation-invariant feature. However, ignoring orientation means losing information, that can improve recognition and enable novel applications. Thus, we showed how to derive both vertical and horizontal orientation using only an accelerometer signal. Our proposed method is comparable to using the Euler Angles and GPS. Yet, of course it has the advantage that is uses an accelerometer only and is not limited to being outside (gps) or susceptible to metal (magnetic compass). Of course, the experimental setup and trials presented are only an initial result to underline the usefulness of the approach. 5.7 Bibliography [1] U. Blanke and B. Schiele. Sensing location in the pocket. Ubicomp Poster Session, page 2, Aug 2008. [2] F. Ichikawa, J. Chipchase, and R. Grignani. Where’s the phone? a study of mobile phone location in public spaces. Mobile Technology, Applications and Systems, 2005 2nd International Conference on, pages 1 – 8, Oct 2005. [3] K. Kunze and P. Lukowicz. Using acceleration signatures from everyday ac- tivities for on-body device location. Wearable Computers, 2007 11th IEEE Inter- national Symposium on, pages 115 – 116, Sep 2007. [4] K. Kunze and P. Lukowicz. Dealing with sensor displacement in motion- J. Shaffer, and F. L. Wong. Sensay: a context-aware mobile phone. In Wearable Computers, 2003. Proceedings. Seventh IEEE International Symposium on, pages 248 – 249, oct. 2003. [48] B. Smith, L. Bass, and J. Siegel. On site maintenance using a wearable com- puter system. In Conference on Human Factors in Computing Systems, pages 119–120. ACM New York, NY, USA, 1995. [49] M. Stager and P. Lukowicz. Power and accuracy trade-offs in sound-based context recognition systems. Pervasive and Mobile Computing, 2007. [50] T. Starner, B. Schiele, and A. Pentland. Visual contextual awareness in wear- able computing. In Proceeding of the Second Int. Symposium on Wearable Com- puting. Pittsburgh, October, 1998. [51] U. Steinhoff and B. Schiele. Dead reckoning from the pocket - an experimen- tal study. In Eighth Annual IEEE International Conference on Pervasive Comput- ing and Communications (PerCom 2010), Mannheim, Germany, 04/2010 2010. IEEE, IEEE. [52] J. Sunkpho, J. Garrett Jr, A. Smailagic, and D. Siewiorek. MIA: A Wearable Computer for Bridge Inspectors. In Proceedings of the 2nd IEEE International Symposium on Wearable Computers, page 160. IEEE Computer Society Wash- ington, DC, USA, 1998. [53] E. Tapia, S. Intille, and K. Larson. Activity recognition in the home using sim- ple and ubiquitous sensors. In A. Ferscha and F. Mattern, editors, Pervasive Computing, volume 3001 of Lecture Notes in Computer Science, pages 158–175. Springer Berlin / Heidelberg, 2004. [54] S. Thiemjarus. A device-orientation independent method for activity recog- most work so far discussed only displacement robust features, putting up with a loss of information.
  37. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Displacement Contributions § modeling based on physical principles § definition of a heuristic for dynamic feature selection § experimental evaluation
  38. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Rigid Body Approximation
  39. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Rigid Body Approximation p1 p2 p3 p1 p2 p3 €  d 1,2 €  d 1,3 €  d 2,3 €  d 1,2 €  d 2,3 €  d 1,3 €  x 1 €  x 2 €  x 3 €  r 1 €  r 2 €  r 3 €  θ 1 €  θ 2 €  θ 3 € α1,2 € α2,3 € α1,3 €  r 1 €  r 2 €  r 3 € α1,2 € α2,3 € α1,3 p1 p2 p3 p1 p2 p3 €  d 1,2 €  d 1,3 €  d 2,3 €  d 1,2 €  d 1,3 €  d 2,3 § model a body part as a rigid body § every movement can be described as a combination of rotations and translations § for Translation + Rotation: Angular Velocity is displacement indifferent § for Translation only: Acceleration is displacement indifferent § How do we decide when to choose what?
  40. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    0 100 200 300 400 500 600 5 0 5 10 15 20 25 30 35 40 45 N Accelerartion / N Angle Vel + N Angle Accel Norm Difference (in %) classification on angular velocity classification on acceleration A B || A || || B || Ratio between acceleration signals Norm Acceleration Norm Angle Velocity + Norm Angle Acceleration Principle
  41. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Locomotion Experiments § Setup: § 5 XSense Motion Sensors § 3 axis accelerometer / gyroscope § randomized position on upper leg § 3 users 8 Activities: walking running running uphill biking rowing climbing stairs skiing crosstrainer
  42. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Locomotion Results 10 slice majority decision over KNN frame-by-frame classification Modality Same Displaced Acceleration 100% 63% Gyroscope 80% 72% 4. Displacement Table 4.4: Joint Accleerometer and Gyro trained on two sensors, evaluated on three, 90 % decision boundary at 150. i j k l m n o p classified as 100 0 0 0 0 0 0 0 i = walking 0 76.9 23.1 0 0 0 0 0 j = running 0 20.3 79.7 0 0 0 0 0 k = uphill 0 0 0 90.4 9.6 0 0 0 l = biking 0 0 0 0 100 0 0 0 m = rowing 0 0 0 0 0 91.1 8.9 0 n = stairs 8.7 0 0 0 0 0 91.3 0 o = skiing 6.1 0 0 0 0 0 0 93.9 p = crosstrainer Table 4.5: Classification comparison for the locomotion exercises using a majority decision over the motions based on a Knn classifier. Acceleration cut-off Norm - 9.81 at larger than Heuristic 90%
  43. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Gym Exercises § Same setup only on lower arm § 3 users 8 Exercises: lat machine pectorial sholder press upper back arm extension arm curl pull down chestpress
  44. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Gym Exercise Results Modality Same Displaced Acceleration 97% 24% continuous HMMs 15 sec. sliding window 3 gaussians up to 4 hidden states 4.7. Conclusion Table 4.7: Confusion Matrix Joint Accleerometer and Gyro trained on 2 Sensors eval on 2 82 % decision boundary at 300 q r s t u v w x classified as 75.6 0 0 0 0 0 0 24.4 q = lat 0 81.6 0 0 0 0 18.4 0 r = pectorial 0 0 88.6 0 11.4 0 0 0 s = shoulder press 0 0 0 100 0 0 0 0 t = upper back 0 0 13.3 0 76.7 0 10.0 0 u = arm extension 0 0 0 0 22.2 77.8 0 0 v = arm curl 12.0 0 0 0 8.0 0 80 0 w = pull down 0 0 0 20.8 0 0 0 79.2 x = chestpress body part. Combined with two sensor training to force the classifier to ignore Heuristic 74%
  45. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Displacement Summary Achievements +easy to implement +computational overhead low +measurable improvements Limitations - needs gyro yet, more and more common - “only” heuristics - “only” displacement device orientation changes need to be handled separately
  46. Kai Kunze Compensating for On-Body Placement Effects in Activity Recognition

    Summary Is the device on the body? no yes Can we determine its placement in the environment? Can we recognize the body part? Is it displaced? Did the orientation change? yes no no yes infere orientation compensate use as trained Environmental Placement: Active Sampling On-body Placement Recognition Orientation Recognition while walking Heuristics for Displacement