Telewheelchiar - SIGGRAPH Asia 2017 (E-tech Talk)

This slide was presented at SIGGRAPH Asia 2017 "Emerging Technologies Talk: Embodied Interaction" session.
https://sa2017.siggraph.org/attendees/emerging-technologies?view=session&sid=47

Telewheelchair: a demonstration of the intelligent electric wheelchair system towards human-machine
https://doi.org/10.1145/3132818.3132834

Telewheelchair: The intelligent electric wheelchair system towards human-machine combined environmental supports
https://doi.org/10.1145/3102163.3102238

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2017/03/telewheelchair/

【Project movie】
https://www.youtube.com/watch?v=e9bcp0elNFs

【Presenter】
Satoshi Hashizume (橋爪智)
University of Tsukuba,
Digital Nature Group (Yoichi Ochiai)

【Abstract】
Telewheelchair is divided into two parts: a wheelchair part and a base station part for operating a wheelchair. We propose a telepresence system which enables us to provide care from a remote place by installing telepresence function in a wheelchair. The caregiver who drives the wheelchair can see environments around the handicapped person through the omnidirectional camera mounted on wheelchair. The caregiver wears HMD to view this image, and hold controller to control wheelchair. In order to safely operation, we employ Human-detection system by using YOLO. If person come close to the wheelchair, system stop wheelchair and display a caution on HMD. To assist operation in narrow area such like corridor, we employ environment recognition using SLAM. In future work, if a plurality of wheelchairs are connected and one operator person can assist with switching among these wheelchairs, the cost of nursing care can be reduced, and more people can receive the nursing care.

1. Telewheelchair A Demonstration of the Intelligent Electric Wheelchair System towards Human-Machine 1 Satoshi Hashizume,IppeiSuzuki,KazukiTakazawa RyuichiroSasaki,Yoshikuni Hashimoto,Yoichi Ochiai UniversityofTsukuba,AISIN SeikiCo., Ltd.
2. Introduction Wheelchairs are important mobility. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction Problem Aging society Caregivers understaffing 3
4. Introduction Problem Aging society Caregivers understaffing Increasing burden of caregiver 4
5. Introduction Increasing burden of caregiver We tried to ease the strain on caregivers by developing a new electric wheelchair system. 5
6. Introduction Method to ease the strain on caregivers. Remote control Manipulation aid with automatic operation 6
7. RelatedWork: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration 7 RT Gundersen, Stephen J Smith, and Ben AAbbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
8. RelatedWork: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion 8 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
9. RelatedWork: manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 9
10. RelatedWork: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 10 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
11. RelatedWork: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 11 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
12. Implementation 12
13. Implementation: electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, hight 135cm 13
14. Implementation: electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 14
15. Implementation: electric wheelchair 15
16. Implementation: base station 16 Driver watched a image of the wheelchair viewpoint using HMD. Display an arrow indicating a direction of the wheelchair movement in VR.
17. Implementation: base station 17 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
18. Implementation: remote control Wireless transfer of video and operation signal. 18 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
19. Implementation: operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 19 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
20. Implementation: operation assistance Environmental map creation by SLAM 20 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
21. Limitation Latency  Transmission distance Monitoring passenger Passenger's fear 21
22. Research Member 22 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoshikuni Hashimoto2 Yoichi Ochiai1

Transcript

1. Telewheelchair
   A Demonstration of the Intelligent Electric
   Wheelchair System towards Human-Machine
   1
   Satoshi Hashizume, IppeiSuzuki, KazukiTakazawa
   Ryuichiro Sasaki, Yoshikuni Hashimoto, Yoichi Ochiai
   University of Tsukuba, AISIN Seiki Co., Ltd.
   

   View Slide

2. Introduction
   Wheelchairs are important mobility.
   It is always necessary for carers to be located
   near the wheelchair.
   2
   

   View Slide

3. Introduction
   Problem
   Aging society
   Caregivers understaffing
   3
   

   View Slide

4. Introduction
   Problem
   Aging society
   Caregivers understaffing
   Increasing burden of caregiver
   4
   

   View Slide

5. Introduction
   Increasing burden of caregiver
   We tried to ease the strain on caregivers
   by developing a new electric wheelchair system.
   5
   

   View Slide

6. Introduction
   Method to ease the strain on caregivers.
   Remote control
   Manipulation aid with automatic operation
   6
   

   View Slide

7. Related Work: automatic operation
   Applications of virtual reality technology to
   wheelchair remote steering systems
   Remote control electric wheelchair using VR.
   Complex configuration
   7
   RT Gundersen, Stephen J Smith, and Ben A Abbott. 1996. Applications of virtual reality technology to wheelchair remote
   steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
   

   View Slide

8. Related Work: automatic operation
   Robotic Wheelchair Easy to Move and
   Communicate with Companions
   Using a laser range sensor to move with the
   companion
   8
   Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki.
   2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors
   in Computing Systems. ACM, 3079–3082.
   

   View Slide

9. Related Work: manipulation method
   Electrooculography (EOG)
   Electromyograph (EMG)
   Voice
   Hand gesture
   9
   

   View Slide

10. Related Work: manipulation method
    EOG guidance of a wheelchair using neural networks
    Identify Electrooculography using a neural network to operate a wheelchair.
    10
    Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair
    using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
    

    View Slide

11. Related Work: manipulation method
    Electronic control of a wheelchair guided by voice commands
    Using the voice commands to operate the wheelchair.
    11
    PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995),
    665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE,
    2007 annual conference. IEEE, 336–340.
    

    View Slide

12. Implementation
    12
    

    View Slide

13. Implementation: electric wheelchair
    Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD.
    22 inch, Max speed 6 km/h
    width 70cm, length 100cm, hight 135cm
    13
    

    View Slide

14. Implementation: electric wheelchair
    We recorded the image of the wheelchair
    viewpoint with the omnidirectional camera.
    A microcomputer is connected to the
    controller.
    14
    

    View Slide

15. Implementation: electric wheelchair
    15
    

    View Slide

16. Implementation: base station
    16
    Driver watched a image of the wheelchair
    viewpoint using HMD.
    Display an arrow indicating a direction of
    the wheelchair movement in VR.
    

    View Slide

17. Implementation: base station
    17
    Using HDMI splitter to share the image of the omnidirectional camera.
    HDMI Extender
    HDMI Splitter
    (b)Linux
    HDMI HDMI
    HDMI
    UDP
    (c) Linux
    Environment Recognition
    (LSD-SLAM; ROS)
    Object Recognition
    (YOLO; CUDA+OpenCV)
    (a) Windows
    VR Controll
    (Unity)
    Wheelchair Controll
    (Processing)
    

    View Slide

18. Implementation: remote control
    Wireless transfer of video and operation signal.
    18
    Video
    CW-1 (IDX Company, Ltd.)
    Full HD, Max 30m
    less than 1ms latency
    Operation signal
    Xbee ZB S2C
    (Digi International K.K.)
    Max 60m
    

    View Slide

19. Implementation: operation assistance
    Automatic stop by object identification.
    Using YOLO which is the real-time
    object detection system.
    19
    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI:
    http://dx.doi.org/10.1109/CVPR.2016.91
    

    View Slide

20. Implementation: operation assistance
    Environmental map creation by SLAM
    20
    Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer
    International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
    

    View Slide

21. Limitation
    Latency
     Transmission distance
    Monitoring passenger
    Passenger's fear
    21
    

    View Slide

22. Research Member
    22
    Satoshi Hashizume1
    Ryuichiro Sasaki2
    1University of Tsukuba, 2AISIN Seiki Co., Ltd.
    Ippei Suzuki1 Kazuki Takazawa1
    Yoshikuni Hashimoto2 Yoichi Ochiai1
    

    View Slide