Telewheelchiar - SIGGRAPH Asia 2017 (E-tech Talk)

This slide was presented at SIGGRAPH Asia 2017 "Emerging Technologies Talk: Embodied Interaction" session.
https://sa2017.siggraph.org/attendees/emerging-technologies?view=session&sid=47

Telewheelchair: a demonstration of the intelligent electric wheelchair system towards human-machine
https://doi.org/10.1145/3132818.3132834

Telewheelchair: The intelligent electric wheelchair system towards human-machine combined environmental supports
https://doi.org/10.1145/3102163.3102238

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2017/03/telewheelchair/

【Project movie】
https://www.youtube.com/watch?v=e9bcp0elNFs

【Presenter】
Satoshi Hashizume (橋爪智)
University of Tsukuba,
Digital Nature Group (Yoichi Ochiai)

【Abstract】
Telewheelchair is divided into two parts: a wheelchair part and a base station part for operating a wheelchair. We propose a telepresence system which enables us to provide care from a remote place by installing telepresence function in a wheelchair. The caregiver who drives the wheelchair can see environments around the handicapped person through the omnidirectional camera mounted on wheelchair. The caregiver wears HMD to view this image, and hold controller to control wheelchair. In order to safely operation, we employ Human-detection system by using YOLO. If person come close to the wheelchair, system stop wheelchair and display a caution on HMD. To assist operation in narrow area such like corridor, we employ environment recognition using SLAM. In future work, if a plurality of wheelchairs are connected and one operator person can assist with switching among these wheelchairs, the cost of nursing care can be reduced, and more people can receive the nursing care.

1. Telewheelchair A Demonstration of the Intelligent Electric Wheelchair System towards Human-Machine 1 Satoshi Hashizume,IppeiSuzuki,KazukiTakazawa RyuichiroSasaki,Yoshikuni Hashimoto,Yoichi Ochiai UniversityofTsukuba,AISIN SeikiCo., Ltd.
2. Introduction Wheelchairs are important mobility. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction Problem Aging society Caregivers understaffing 3
4. Introduction Problem Aging society Caregivers understaffing Increasing burden of caregiver 4
5. Introduction Increasing burden of caregiver We tried to ease the strain on caregivers by developing a new electric wheelchair system. 5
6. Introduction Method to ease the strain on caregivers. Remote control Manipulation aid with automatic operation 6
7. RelatedWork: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration 7 RT Gundersen, Stephen J Smith, and Ben AAbbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
8. RelatedWork: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion 8 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
9. RelatedWork: manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 9
10. RelatedWork: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 10 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
11. RelatedWork: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 11 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
12. Implementation 12
13. Implementation: electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, hight 135cm 13
14. Implementation: electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 14
15. Implementation: electric wheelchair 15
16. Implementation: base station 16 Driver watched a image of the wheelchair viewpoint using HMD. Display an arrow indicating a direction of the wheelchair movement in VR.
17. Implementation: base station 17 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
18. Implementation: remote control Wireless transfer of video and operation signal. 18 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
19. Implementation: operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 19 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
20. Implementation: operation assistance Environmental map creation by SLAM 20 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
21. Limitation Latency  Transmission distance Monitoring passenger Passenger's fear 21
22. Research Member 22 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoshikuni Hashimoto2 Yoichi Ochiai1

Transcript

1. Telewheelchair A Demonstration of the Intelligent Electric Wheelchair System towards Human-Machine 1 Satoshi Hashizume, IppeiSuzuki, KazukiTakazawa Ryuichiro Sasaki, Yoshikuni Hashimoto, Yoichi Ochiai University of Tsukuba, AISIN Seiki Co., Ltd.
2. Introduction Wheelchairs are important mobility. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction Problem Aging society Caregivers understaffing 3
4. Introduction Problem Aging society Caregivers understaffing Increasing burden of caregiver 4
5. Introduction Increasing burden of caregiver We tried to ease the strain on caregivers by developing a new electric wheelchair system. 5
6. Introduction Method to ease the strain on caregivers. Remote control Manipulation aid with automatic operation 6
7. Related Work: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration 7 RT Gundersen, Stephen J Smith, and Ben A Abbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
8. Related Work: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion 8 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
9. Related Work: manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 9
10. Related Work: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 10 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
11. Related Work: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 11 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
12. Implementation 12
13. Implementation: electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, hight 135cm 13
14. Implementation: electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 14
15. Implementation: electric wheelchair 15
16. Implementation: base station 16 Driver watched a image of the wheelchair viewpoint using HMD. Display an arrow indicating a direction of the wheelchair movement in VR.
17. Implementation: base station 17 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
18. Implementation: remote control Wireless transfer of video and operation signal. 18 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
19. Implementation: operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 19 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
20. Implementation: operation assistance Environmental map creation by SLAM 20 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
21. Limitation Latency  Transmission distance Monitoring passenger Passenger's fear 21
22. Research Member 22 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoshikuni Hashimoto2 Yoichi Ochiai1