$30 off During Our Annual Pro Sale. View Details »

Telewheelchiar - SIGGRAPH Asia 2017 (E-tech Talk)

Telewheelchiar - SIGGRAPH Asia 2017 (E-tech Talk)

This slide was presented at SIGGRAPH Asia 2017 "Emerging Technologies Talk: Embodied Interaction" session.
https://sa2017.siggraph.org/attendees/emerging-technologies?view=session&sid=47

Telewheelchair: a demonstration of the intelligent electric wheelchair system towards human-machine
https://doi.org/10.1145/3132818.3132834

Telewheelchair: The intelligent electric wheelchair system towards human-machine combined environmental supports
https://doi.org/10.1145/3102163.3102238

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2017/03/telewheelchair/

【Project movie】
https://www.youtube.com/watch?v=e9bcp0elNFs

【Presenter】
Satoshi Hashizume (橋爪智)
University of Tsukuba,
Digital Nature Group (Yoichi Ochiai)

【Abstract】
Telewheelchair is divided into two parts: a wheelchair part and a base station part for operating a wheelchair. We propose a telepresence system which enables us to provide care from a remote place by installing telepresence function in a wheelchair. The caregiver who drives the wheelchair can see environments around the handicapped person through the omnidirectional camera mounted on wheelchair. The caregiver wears HMD to view this image, and hold controller to control wheelchair. In order to safely operation, we employ Human-detection system by using YOLO. If person come close to the wheelchair, system stop wheelchair and display a caution on HMD. To assist operation in narrow area such like corridor, we employ environment recognition using SLAM. In future work, if a plurality of wheelchairs are connected and one operator person can assist with switching among these wheelchairs, the cost of nursing care can be reduced, and more people can receive the nursing care.

1. Telewheelchair A Demonstration of the Intelligent Electric Wheelchair System towards Human-Machine 1 Satoshi Hashizume,IppeiSuzuki,KazukiTakazawa RyuichiroSasaki,Yoshikuni Hashimoto,Yoichi Ochiai UniversityofTsukuba,AISIN SeikiCo., Ltd.
2. Introduction Wheelchairs are important mobility. It is always necessary for carers to be located near the wheelchair. 2
3. Introduction Problem Aging society Caregivers understaffing 3
4. Introduction Problem Aging society Caregivers understaffing Increasing burden of caregiver 4
5. Introduction Increasing burden of caregiver We tried to ease the strain on caregivers by developing a new electric wheelchair system. 5
6. Introduction Method to ease the strain on caregivers. Remote control Manipulation aid with automatic operation 6
7. RelatedWork: automatic operation Applications of virtual reality technology to wheelchair remote steering systems Remote control electric wheelchair using VR. Complex configuration 7 RT Gundersen, Stephen J Smith, and Ben AAbbott. 1996. Applications of virtual reality technology to wheelchair remote steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.
8. RelatedWork: automatic operation Robotic Wheelchair Easy to Move and Communicate with Companions Using a laser range sensor to move with the companion 8 Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki. 2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 3079–3082.
9. RelatedWork: manipulation method Electrooculography (EOG) Electromyograph (EMG) Voice Hand gesture 9
10. RelatedWork: manipulation method EOG guidance of a wheelchair using neural networks Identify Electrooculography using a neural network to operate a wheelchair. 10 Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.
11. RelatedWork: manipulation method Electronic control of a wheelchair guided by voice commands Using the voice commands to operate the wheelchair. 11 PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995), 665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE, 2007 annual conference. IEEE, 336–340.
12. Implementation 12
13. Implementation: electric wheelchair Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD. 22 inch, Max speed 6 km/h width 70cm, length 100cm, hight 135cm 13
14. Implementation: electric wheelchair We recorded the image of the wheelchair viewpoint with the omnidirectional camera. A microcomputer is connected to the controller. 14
15. Implementation: electric wheelchair 15
16. Implementation: base station 16 Driver watched a image of the wheelchair viewpoint using HMD. Display an arrow indicating a direction of the wheelchair movement in VR.
17. Implementation: base station 17 Using HDMI splitter to share the image of the omnidirectional camera. HDMI Extender HDMI Splitter (b)Linux HDMI HDMI HDMI UDP (c) Linux Environment Recognition (LSD-SLAM; ROS) Object Recognition (YOLO; CUDA+OpenCV) (a) Windows VR Controll (Unity) Wheelchair Controll (Processing)
18. Implementation: remote control Wireless transfer of video and operation signal. 18 Video CW-1 (IDX Company, Ltd.) Full HD, Max 30m less than 1ms latency Operation signal Xbee ZB S2C (Digi International K.K.) Max 60m
19. Implementation: operation assistance Automatic stop by object identification. Using YOLO which is the real-time object detection system. 19 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI: http://dx.doi.org/10.1109/CVPR.2016.91
20. Implementation: operation assistance Environmental map creation by SLAM 20 Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54
21. Limitation Latency  Transmission distance Monitoring passenger Passenger's fear 21
22. Research Member 22 Satoshi Hashizume1 Ryuichiro Sasaki2 1University of Tsukuba, 2AISIN Seiki Co., Ltd. Ippei Suzuki1 Kazuki Takazawa1 Yoshikuni Hashimoto2 Yoichi Ochiai1

Digital Nature Group

December 08, 2017
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. Telewheelchair
    A Demonstration of the Intelligent Electric
    Wheelchair System towards Human-Machine
    1
    Satoshi Hashizume, IppeiSuzuki, KazukiTakazawa
    Ryuichiro Sasaki, Yoshikuni Hashimoto, Yoichi Ochiai
    University of Tsukuba, AISIN Seiki Co., Ltd.

    View Slide

  2. Introduction
    Wheelchairs are important mobility.
    It is always necessary for carers to be located
    near the wheelchair.
    2

    View Slide

  3. Introduction
    Problem
    Aging society
    Caregivers understaffing
    3

    View Slide

  4. Introduction
    Problem
    Aging society
    Caregivers understaffing
    Increasing burden of caregiver
    4

    View Slide

  5. Introduction
    Increasing burden of caregiver
    We tried to ease the strain on caregivers
    by developing a new electric wheelchair system.
    5

    View Slide

  6. Introduction
    Method to ease the strain on caregivers.
    Remote control
    Manipulation aid with automatic operation
    6

    View Slide

  7. Related Work: automatic operation
    Applications of virtual reality technology to
    wheelchair remote steering systems
    Remote control electric wheelchair using VR.
    Complex configuration
    7
    RT Gundersen, Stephen J Smith, and Ben A Abbott. 1996. Applications of virtual reality technology to wheelchair remote
    steering systems. In Proc. of 1st Euro Conf of Disability, Virtual Reality & Assoc. Technology. 47–56.

    View Slide

  8. Related Work: automatic operation
    Robotic Wheelchair Easy to Move and
    Communicate with Companions
    Using a laser range sensor to move with the
    companion
    8
    Yoshinori Kobayashi, Ryota Suzuki, Yoshihisa Sato, Masaya Arai, Yoshinori Kuno, Akiko Yamazaki, and Keiichi Yamazaki.
    2013. Robotic wheelchair easy to move and communicate with companions. In CHI’13 Extended Abstracts on Human Factors
    in Computing Systems. ACM, 3079–3082.

    View Slide

  9. Related Work: manipulation method
    Electrooculography (EOG)
    Electromyograph (EMG)
    Voice
    Hand gesture
    9

    View Slide

  10. Related Work: manipulation method
    EOG guidance of a wheelchair using neural networks
    Identify Electrooculography using a neural network to operate a wheelchair.
    10
    Rafael Barea, Luciano Boquete, Manuel Mazo, Elena López, and Luis Miguel Bergasa. 2000. EOG guidance of a wheelchair
    using neural networks. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, Vol. 4. IEEE, 668–671.

    View Slide

  11. Related Work: manipulation method
    Electronic control of a wheelchair guided by voice commands
    Using the voice commands to operate the wheelchair.
    11
    PA Revenga. 1995. Electronic control of a wheelchair guided by voice commands. Control Engineering Practice 3, 5 (1995),
    665–674. 19. Masato Nishimori, Takeshi Saitoh, and Ryosuke Konishi. 2007. Voice controlled intelligent wheelchair. In SICE,
    2007 annual conference. IEEE, 336–340.

    View Slide

  12. Implementation
    12

    View Slide

  13. Implementation: electric wheelchair
    Based on TAO LIGHT II-m of AISIN SEIKI CO,. LTD.
    22 inch, Max speed 6 km/h
    width 70cm, length 100cm, hight 135cm
    13

    View Slide

  14. Implementation: electric wheelchair
    We recorded the image of the wheelchair
    viewpoint with the omnidirectional camera.
    A microcomputer is connected to the
    controller.
    14

    View Slide

  15. Implementation: electric wheelchair
    15

    View Slide

  16. Implementation: base station
    16
    Driver watched a image of the wheelchair
    viewpoint using HMD.
    Display an arrow indicating a direction of
    the wheelchair movement in VR.

    View Slide

  17. Implementation: base station
    17
    Using HDMI splitter to share the image of the omnidirectional camera.
    HDMI Extender
    HDMI Splitter
    (b)Linux
    HDMI HDMI
    HDMI
    UDP
    (c) Linux
    Environment Recognition
    (LSD-SLAM; ROS)
    Object Recognition
    (YOLO; CUDA+OpenCV)
    (a) Windows
    VR Controll
    (Unity)
    Wheelchair Controll
    (Processing)

    View Slide

  18. Implementation: remote control
    Wireless transfer of video and operation signal.
    18
    Video
    CW-1 (IDX Company, Ltd.)
    Full HD, Max 30m
    less than 1ms latency
    Operation signal
    Xbee ZB S2C
    (Digi International K.K.)
    Max 60m

    View Slide

  19. Implementation: operation assistance
    Automatic stop by object identification.
    Using YOLO which is the real-time
    object detection system.
    19
    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 779–788. DOI:
    http://dx.doi.org/10.1109/CVPR.2016.91

    View Slide

  20. Implementation: operation assistance
    Environmental map creation by SLAM
    20
    Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer
    International Publishing, Cham, 834–849. DOI: http://dx.doi.org/10.1007/978-3-319-10605-2_54

    View Slide

  21. Limitation
    Latency
     Transmission distance
    Monitoring passenger
    Passenger's fear
    21

    View Slide

  22. Research Member
    22
    Satoshi Hashizume1
    Ryuichiro Sasaki2
    1University of Tsukuba, 2AISIN Seiki Co., Ltd.
    Ippei Suzuki1 Kazuki Takazawa1
    Yoshikuni Hashimoto2 Yoichi Ochiai1

    View Slide