CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

Jenny Midwinter at May 21, 2019 event of montrealml.dev

Title: CNN Based Auto-Pilot for a Wheelchair: A brief story of the results of rapid development of a CNN based self driving wheelchair using open source software & low cost hardware

Summary: This lightening talk describes the development & results that were successfully used to create a proof-of-concept (POC) prototype for a CNN based auto-pilot function that was able to autonomously drive a wheelchair based on a single input image stream from forward facing camera . A brief explanation of how the CNN is used to autonomously drive the wheelchair is provided, including a short video demo. As well, a description of the low-cost hardware/software system, & the learnings that were obtained, now being used as a basis for the next stage of development, will also be presented.

Bio: Jenny is collaborating as Chief Scientist, Applied AI, at Eightfold Technologies, and responsible for the development of vision based self-driving technology for Eightfold’s SmartChair. She is the CEO and founder of Blue Horizon AI, a start up dedicated to applying AI in autonomous technologies to assist the disabled. Prior to this, she has 25 years experience in R&D in the Telecom Industry.

Transcript

1. CNN Based Auto-Pilot for a
   Wheelchair
   A brief story of rapid development of a proof-
   of-concept self driving wheelchair using open
   source software & low cost hardware.
   5/20/2019 jenny@bluehorizon.ai 1
   Jenny Midwinter
   

   View Slide

2. 5/20/2019 jenny@bluehorizon.ai 2
   Stewart Midwinter
   • Address the extreme
   difficulty some power
   wheelchair users have in
   getting out in their
   environment.
   • Cognitive & neurological
   challenges to drive a
   wheelchair. It is actually
   hard! (even for an able
   bodied person)
   

   View Slide

3. Self-Driving Wheelchair: Goals
   • Provide “lane assist” to autonomously follow a
   pathway, sidewalk, or hallway.
   • Low cost & easy to retrofit to user’s chair.
   • Open source the software.
   • Other applications with WC Autonomous Movement:
   – Follow-Me/Accompany-Me,
   – Automated Docking for Battery Recharging
   5/20/2019 jenny@bluehorizon.ai 3
   

   View Slide

4. Self-Driving Wheelchair – Early Decisions
   • Early Technical Decisions: “Strawman”
   – Use of CNN to predict a steering angle from a forward
   facing camera inspired by “End to End Learning for Self-
   Driving Cars” [1]
   – Use Eightfold Smart Chair a proven working product, able
   to remotely & programmatically deliver driving
   instructions to a wheelchair motor control system [5]
   – Use ROS (Robotic Operating System) [6]
   – Short-cut to POC … pre-trained CNN (“Learning to Fly by
   Driving” [2] ) since trained on a relevant & applicable open
   data sets. [3] [4]
   5/20/2019 jenny@bluehorizon.ai 4
   Build a Proof of Concept Prototype
   

   View Slide

5. WC POC Auto-Pilot Run-Time Process
   5/20/2019 jenny@bluehorizon.ai 5
   Image Pre-Processing
   CNN
   Joy Stick Equivalent Commands
   4) Create a “joy stick”
   equivalent command that
   is sent to WC Motor control
   to drive the wheelchair
   1) Input Images from a
   Single Forward Facing
   Camera
   2) Image Cropping, Re-size,
   Grayscale
   Steering Angle Collision Probability
   3) Trained Convolutional
   Neural Network produces
   steering Angle &
   probability of collision
   

   View Slide

6. Demo of
   Results
   “Beta”
   5/20/2019 jenny@bluehorizon.ai 6
   

   View Slide

7. Eightfold’s Smart Chair: A Remote Virtual Joystick
   5/20/2019
   jenny@bluehorizon.ai
   7
   Eightfold
   Technologies
   Custom
   Arduino Board
   SmartPhone
   Eightfold’s Smart Chair
   • “Virtual Joy Stick” Phone
   App
   • App sends “joy stick” type
   commands to Arduino over
   BLE
   • Arduino converts commands
   to electric signals for
   wheelchair motor control
   BLE
   Eightfold
   Technologies
   Joy Stick
   Connection
   Started With an Established Interface
   

   View Slide

8. Eightfold’s Custom Arduino Board
   5/20/2019 jenny@bluehorizon.ai 8
   Controls Voltage Lines to Motor Control
   Arduino
   & HW
   WC Control
   WC Motor
   Phone App
   JoyStick
   

   View Slide

9. Self-Driving Wheelchair (Alpha)
   5/20/2019
   jenny@bluehorizon.ai
   9
   Custom
   Arduino
   USB Webcam
   Laptop - CNN • USB Webcam & Arduino
   connected directly to the
   laptop
   • A CNN on laptop, produces
   the steering drive instructions
   from images
   • Arduino converts commands
   to electric signals for
   wheelchair motor control
   Replaced the Virtual Remote Joystick with CNN
   

   View Slide

10. Keras
    Tensorflow
    CUDA
    POC Prototype with Pi3 & CNN
    5/20/2019 jenny@bluehorizon.ai 10
    LapTop
    Pi3
    WC Control
    WC Motor
    Joystick
    Video stream opencv
    wifi
    usb
    usb
    USB Camera
    ROS Topics
    /cnn_out *
    /drive
    /chair
    /image
    “Deadman” Switch
    ROS Based Distributed Architecture & Open Source Packages
    * DroNet [2] modified for WC POC
    chairRx
    chairTx
    Perception *
    chairDrive
    chairReason
    ROS
    Linux
    ROS
    Linux
    SmartChair.ino
    Arduino + Shield
    

    View Slide

11. Results of the POC Prototype
    • Basic self-driving operation tested in a variety of
    environments (without navigation)
    • Drives on its own for short periods of time, usually
    stopping on its own well before running into any
    objects (except for low lying object like a curb).
    • Proven to be viable as a platform, but lots more
    work needed.
    • ** Helped to identify the next steps & focus.
    5/20/2019 jenny@bluehorizon.ai 11
    CNN as viable solution for Wheelchair Auto-Pilot
    

    View Slide

12. What was Learned from POC
    • WC drives more like a tank than a car! Need better
    data to train in order to get improvements.
    • Collision detection & stopping worked well overall,
    but missed low level objects like curbs.
    • Image pre-processing , mounting location/height &
    angle & type of camera
    5/20/2019 jenny@bluehorizon.ai 12
    Platform for Further Experimentation
    

    View Slide

13. Other Observations & Learnings
    • A lot of tedious “unglamorous” work:
    – installing software,
    – resolving inter-compatibility issues,
    – bugs in open source code,
    – working with the datasets,
    – setting up the development environments.
    • Lots of effort into instrumenting software &
    hardware to both test & debug to make sure that
    WC is doing the correct behavior
    5/20/2019 jenny@bluehorizon.ai 13
    Development & Testing Continues
    

    View Slide

14. Conclusions – Proven Possibilities
    • Allows those with neurological deficiencies to be
    able to accompany able-bodied companions.
    • Allow those with more severe mobility
    impairments more freedom & independence.
    • Autonomous driving techniques can be adapted
    for use in less structured areas.
    • ML algorithms, can be taken from academia in a
    low cost way, using standard engineering R&D
    practices.
    5/20/2019 jenny@bluehorizon.ai 14
    Applying ML to Create Independence
    

    View Slide

15. The Team
    5/20/2019 jenny@bluehorizon.ai 15
    Ke Wang
    Smart Chair
    Software
    Benoit Cote
    Smart Chair
    Hardware
    Najmeh Taleb
    DL Researcher,
    AI Algorithms
    Jenny Midwinter
    Chief Bottle
    Washer
    

    View Slide

16. Blooper –
    Running Into
    Curb
    5/20/2019 jenny@bluehorizon.ai 16
    

    View Slide

17. References
    1. End to End Learning for Self-Driving Cars, Mariusz
    Bojarski. NVIDIA Corporation. Holmdel, NJ 07735. Davide Del
    Testa. NVIDIA Corporation., April 16, 2016.
    https://images.nvidia.com/content/tegra/automotive/images/201
    6/solutions/pdf/end-to-end-dl-using-px.pdf
    2. “Learning to Fly by Driving”, A Loquercio et al, IEEE Robotics &
    Automation Letters. preprint version accepted January, 2018 .
    http://rpg.ifi.uzh.ch/docs/RAL18_Loquercio.pdf
    3. Udacity Open Data Set: https://github.com/udacity/self-driving-
    car/tree/master/datasets
    4. DroNet Open Data set: https://github.com/uzh-
    rpg/rpg_public_dronet
    5. Eightfold Technologies Inc: www.eightfoldtech.com
    6. ROS (Robotic Operating System) http://www.ros.org/
    7. Stewart Midwinter : https://xcmag.com/magazine-
    articles/stewarts-story-living-life-injury/
    5/20/2019 jenny@bluehorizon.ai 17
    

    View Slide