Skeleton-based Human Action Recognition with Recurrent Neural Network

Transcript

1. Skeleton-based Human Action Recognition with Recurrent Neural Network University of Science, VNU-HCM Faculty of Information Technology Advanced Program in Computer Science Võ Trần Thanh Lương 1551020 Vũ Hoàng Quân 1551026 Thesis Advisors: Dr. Trần Thái Sơn Ho Chi Minh City Aug 18th 2019
2. Outlines • • • • • • •
3. Introduction Every human action is done to serve a purpose. Machines should be able to learn and understand it.
4. Introduction
5. Introduction • • ◦ ◦ ◦ ◦ • •
6. Introduction • ◦ ◦
7. Motivation • ◦ ◦ ◦ ◦ ◦ ◦ …
8. • Motivation
9. Contributions • • • •
10. Related Work • •
11. Related Work • ◦ ◦ Zhuowen Lv 1, Xianglei Xing 1,, Kejun Wang 1, and Donghai Guan 2 , "Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review," An example of local representations for human action
12. Related Work • ◦ ◦ Georgios D. Evangelidis, Gurkirt Singh, Radu Horaud, "Continuous Gesture Recognition from Articulated Poses," Spatial Temporal interest points. S.F. Wong, T.-K. Kim, and R. Cipolla, "Learning motion categories using both semantic and structural information”
13. Related Work • ◦ ◦ ▪ ▪ Example of 3D convolution. Karen Simonyan & Andrew Zisserman , "Two-Stream Convolutional Networks for Action Recognition in Videos,"
14. Related Work • ◦ ◦ Hybrid network for temporal modeling. Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell, "Long-term recurrent convolutional networks for visual recognition and description," Jun Liu, Gang Wang, Ling-Yu Duan, Kamila Abdiyeva, Alex C. Kot, "Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks,
15. Proposed Method Temporal RNN Spatial RNN
16. Temporal RNN • • •
17. Temporal RNN • • • ◦ ◦
20. Spatial RNN • • • •
21. Spatial RNN
22. Spatial RNN • • •
23. Spatial RNN
24. Spatial RNN
25. 3D Transformation
26. Training Flow Skeleton Dataset Training Set Testing Set Initialize RNN Feature Extraction Training Softmax Classification Trained Model NTU Dataset Kinetics Dataset Raw video Extract Skeleton Data Using OpenPose
27. Predicted Flow Raw Video Extract Skeleton Data Load Trained Model Predicted Value Using OpenPose
28. Experiments • ◦ ◦ ◦
29. NTU RGB+D and NTU RGB+D 120 Dataset • • • •
31. Skeleton Joints Position (NTU Dataset)
32. Kinetics Dataset • • •
33. Sample frames of Kinetics Dataset
34. Skeleton Joints Position (Coco Model)
35. Proposed Method Result
36. Comparison with the state-of-the-art
37. Accuracy Calculation
38. Problems with Kinetics dataset NTU RGB+D NTU RGB+D 120 Kinetics Raw Videos Yes Yes No (can obtain from given URLs) 3D skeleton data Yes Yes No Depth maps Yes Yes No
39. Problems with Kinetics dataset ◦ ◦
40. Demo
41. Conclusion • • • •
42. Future Work • • • Jun Liu, Gang Wang, Ling-Yu Duan, Kamila Abdiyeva, Alex C. Kot, "Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks, ASPR Framework. Liu, Jun and Shahroudy, Amir and Perez, Mauricio and Wang, Gang and Duan, Ling-Yu and Kot, Alex C., "NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding,"
43. THANK YOU FOR YOUR ATTENTION University of Science, VNU-HCM Faculty of Information Technology Advanced Program in Computer Science Võ Trần Thanh Lương 1551020 Vũ Hoàng Quân 1551026 Thesis Advisors: Dr. Trần Thái Sơn Ho Chi Minh City Aug 18th 2019