Telewheelchiar - Augmented Human 2018

Published on Feb. 7, 2018

This slide was presented at Augmented Human 2018.
http://www.sigah.org/AH2018/

Telewheelchair: the Remote Controllable Electric Wheelchair System combined Human and Machine Intelligence
https://dl.acm.org/citation.cfm?id=3174914

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2017/03/telewheelchair/

【Project movie】
https://www.youtube.com/watch?v=e9bcp0elNFs

【Presenter】
Satoshi Hashizume (橋爪智)
University of Tsukuba,
Digital Nature Group (Yoichi Ochiai)

【Abstract】
Wheelchairs are essential means of transport for the elderly people and the physically challenged. However, wheelchairs need to be accompanied by caregivers. As society ages and the number of care recipients increases, the burden on caregivers is expected to increase. In order to reduce the burden on caregivers, we present Telewheelchair, an electric wheelchair equipped with a remote control function and computational operation assistance function. The caregiver can remotely control the Telewheelchair by means of a head mounted display (HMD). In addition, the proposed system is equipped with a human detection system to stop the wheelchair automatically and avoid collisions. We conducted a user study on the wheelchair in four types of systems and investigated the time taken to achieve tasks. Telewheelchair will enhance geriatric mobility and improve society by combining human intelligence and machine intelligence.

1. Telewheelchair the Remote Controllable Electric Wheelchair System combined Human and Machine Intelligence 1 Satoshi Hashizume, Ippei Suzuki, KazukiTakazawa Ryuichiro Sasaki,Yoichi Ochiai University ofTsukuba, AISIN Seiki Co.,Ltd.
2. Introduction Wheelchairs are essential means of transport. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction 3 Problem Aging society Lack of caregivers Strain of caregiver is increasing.
4. Introduction Strain of caregiver is increasing. We tried to reduce the strain on caregivers by developing a new electric wheelchair system. 4
5. Introduction Method to reduce the strain on caregivers. Remote control Manipulation aid with automatic operation 5
6. Related Work: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration. 6 RT Gundersen, Stephen J Smith, and Ben AAbbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
7. Related Work: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion. 7 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
8. Related Work : manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 8
9. Related Work: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 9 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
10. Related Work: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 10 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
11. Implementation 11
12. Implementation : electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, height 135cm 12
13. Implementation : electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 13
14. Implementation : electric wheelchair 14
15. Implementation : base station 15 Driver watched a image of a wheelchair viewpoint using HMD. Display an arrow corresponding to the direction of the wheelchair movement in VR.
16. Implementation : base station 16 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
17. Implementation : remote control Wireless transfer of omnidirectional image and operation signal. 17 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
18. Implementation : operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 18 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
19. Implementation : operation assistance Environmental map creation by SLAM. 19 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
20. Evaluation Purpose｜ To investigate the operability of the remote operation. Design ｜We set 6 tasks and 4 operation methods. We measured the operation time for each task. Recording the position of the wheelchair using motion capture. Condition｜Weight of 52.5kg that same as average body weight of 60 year old women is loaded on wheelchair. 20
21. Evaluation Four operation methods. 21
22. Evaluation: Four operation methods Normal mode Operating Telewheelchair with the handle. 22
23. Evaluation: Four operation methods Standby mode Operating with the controller by standing next to wheelchair. 23
24. Evaluation: Four operation methods Display mode Participants watched the display image of only the front part of the omnidirectional image. Do not look the wheelchair directly. 24
25. Evaluation: Four operation methods HMD mode Wear HMD and operate wheelchair while watching the omnidirectional image. 25
26. Evaluation Six tasks of course Straight Right turn Left turn Rotation Door Obstacle 26
27. Evaluation Outline of course Tasks set in the course in a row. Based on equipment standards of nursing care facilities stipulated in Japan. 27 START GOAL 0.8m 1.4m
28. 28
29. Result: operation time of tasks 29 Straight Right turn Door Left turn Rotation Obstacle
30. Result: operation path of wheelchair 30
31. Result Operations using display mode are more difficult / discomfort than normal mode. 31 Difficulty of the operation Feeling of discomfort between movement and operation.
32. Discussion Users operated to turn squarely because they could not look around wheelchair in display mode. → Display mode takes a longer time. 32 Rotation Obstacle
33. Discussion Stand by and HMD modes have similar difficulty of control. We can choose the operation mode depends on the use case. Case of remote control → use HMD mode. Case of control near the wheelchair → use stand by mode. 33
34. Limitation Latency Specification of wheelchair and processing delay. Transmission distance Specification of wireless device. Monitoring passenger Passenger's fear 34
35. Contributions 1. We conducted experiments on the operability of the wheelchair using an HMD. 2. For remote operation, an immersive operation method using an HMD improved the stability more than an operation using a large display. 3. In realizing the semi-automatic operation of an electric wheelchair, we explored functions to be developed in future work. 35
36. Research Member 36 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoichi Ochiai1

Transcript

1. Telewheelchair the Remote Controllable Electric Wheelchair System combined Human and Machine Intelligence 1 Satoshi Hashizume, Ippei Suzuki, Kazuki Takazawa Ryuichiro Sasaki, Yoichi Ochiai University of Tsukuba, AISIN Seiki Co., Ltd.
2. Introduction Wheelchairs are essential means of transport. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction 3 Problem Aging society Lack of caregivers Strain of caregiver is increasing.
4. Introduction Strain of caregiver is increasing. We tried to reduce the strain on caregivers by developing a new electric wheelchair system. 4
5. Introduction Method to reduce the strain on caregivers. Remote control Manipulation aid with automatic operation 5
6. Related Work: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration. 6 RT Gundersen, Stephen J Smith, and Ben A Abbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
7. Related Work: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion. 7 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
8. Related Work : manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 8
9. Related Work: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 9 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
10. Related Work: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 10 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
11. Implementation 11
12. Implementation : electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, height 135cm 12
13. Implementation : electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 13
14. Implementation : electric wheelchair 14
15. Implementation : base station 15 Driver watched a image of a wheelchair viewpoint using HMD. Display an arrow corresponding to the direction of the wheelchair movement in VR.
16. Implementation : base station 16 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
17. Implementation : remote control Wireless transfer of omnidirectional image and operation signal. 17 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
18. Implementation : operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 18 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
19. Implementation : operation assistance Environmental map creation by SLAM. 19 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
20. Evaluation Purpose｜ To investigate the operability of the remote operation. Design ｜We set 6 tasks and 4 operation methods. We measured the operation time for each task. Recording the position of the wheelchair using motion capture. Condition｜Weight of 52.5kg that same as average body weight of 60 year old women is loaded on wheelchair. 20
21. Evaluation Four operation methods. 21
22. Evaluation: Four operation methods Normal mode Operating Telewheelchair with the handle. 22
23. Evaluation: Four operation methods Standby mode Operating with the controller by standing next to wheelchair. 23
24. Evaluation: Four operation methods Display mode Participants watched the display image of only the front part of the omnidirectional image. Do not look the wheelchair directly. 24
25. Evaluation: Four operation methods HMD mode Wear HMD and operate wheelchair while watching the omnidirectional image. 25
26. Evaluation Six tasks of course Straight Right turn Left turn Rotation Door Obstacle 26
27. Evaluation Outline of course Tasks set in the course in a row. Based on equipment standards of nursing care facilities stipulated in Japan. 27 START GOAL 0.8m 1.4m
29. Result: operation time of tasks 29 Straight Right turn Door Left turn Rotation Obstacle
30. Result: operation path of wheelchair 30
31. Result Operations using display mode are more difficult / discomfort than normal mode. 31 Difficulty of the operation Feeling of discomfort between movement and operation.
32. Discussion Users operated to turn squarely because they could not look around wheelchair in display mode. → Display mode takes a longer time. 32 Rotation Obstacle
33. Discussion Stand by and HMD modes have similar difficulty of control. We can choose the operation mode depends on the use case. Case of remote control → use HMD mode. Case of control near the wheelchair → use stand by mode. 33
34. Limitation Latency Specification of wheelchair and processing delay. Transmission distance Specification of wireless device. Monitoring passenger Passenger's fear 34
35. Contributions 1. We conducted experiments on the operability of the wheelchair using an HMD. 2. For remote operation, an immersive operation method using an HMD improved the stability more than an operation using a large display. 3. In realizing the semi-automatic operation of an electric wheelchair, we explored functions to be developed in future work. 35
36. Research Member 36 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoichi Ochiai1