CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

Jenny Midwinter at May 21, 2019 event of

Title: CNN Based Auto-Pilot for a Wheelchair: A brief story of the results of rapid development of a CNN based self driving wheelchair using open source software & low cost hardware

Summary: This lightening talk describes the development & results that were successfully used to create a proof-of-concept (POC) prototype for a CNN based auto-pilot function that was able to autonomously drive a wheelchair based on a single input image stream from forward facing camera . A brief explanation of how the CNN is used to autonomously drive the wheelchair is provided, including a short video demo. As well, a description of the low-cost hardware/software system, & the learnings that were obtained, now being used as a basis for the next stage of development, will also be presented.

Bio: Jenny is collaborating as Chief Scientist, Applied AI, at Eightfold Technologies, and responsible for the development of vision based self-driving technology for Eightfold’s SmartChair. She is the CEO and founder of Blue Horizon AI, a start up dedicated to applying AI in autonomous technologies to assist the disabled. Prior to this, she has 25 years experience in R&D in the Telecom Industry.



May 21, 2019


  1. 1.

    CNN Based Auto-Pilot for a Wheelchair A brief story of

    rapid development of a proof- of-concept self driving wheelchair using open source software & low cost hardware. 5/20/2019 1 Jenny Midwinter
  2. 2.

    5/20/2019 2 Stewart Midwinter • Address the extreme difficulty

    some power wheelchair users have in getting out in their environment. • Cognitive & neurological challenges to drive a wheelchair. It is actually hard! (even for an able bodied person)
  3. 3.

    Self-Driving Wheelchair: Goals • Provide “lane assist” to autonomously follow

    a pathway, sidewalk, or hallway. • Low cost & easy to retrofit to user’s chair. • Open source the software. • Other applications with WC Autonomous Movement: – Follow-Me/Accompany-Me, – Automated Docking for Battery Recharging 5/20/2019 3
  4. 4.

    Self-Driving Wheelchair – Early Decisions • Early Technical Decisions: “Strawman”

    – Use of CNN to predict a steering angle from a forward facing camera inspired by “End to End Learning for Self- Driving Cars” [1] – Use Eightfold Smart Chair a proven working product, able to remotely & programmatically deliver driving instructions to a wheelchair motor control system [5] – Use ROS (Robotic Operating System) [6] – Short-cut to POC … pre-trained CNN (“Learning to Fly by Driving” [2] ) since trained on a relevant & applicable open data sets. [3] [4] 5/20/2019 4 Build a Proof of Concept Prototype
  5. 5.

    WC POC Auto-Pilot Run-Time Process 5/20/2019 5 Image Pre-Processing

    CNN Joy Stick Equivalent Commands 4) Create a “joy stick” equivalent command that is sent to WC Motor control to drive the wheelchair 1) Input Images from a Single Forward Facing Camera 2) Image Cropping, Re-size, Grayscale Steering Angle Collision Probability 3) Trained Convolutional Neural Network produces steering Angle & probability of collision
  6. 7.

    Eightfold’s Smart Chair: A Remote Virtual Joystick 5/20/2019 7

    Eightfold Technologies Custom Arduino Board SmartPhone Eightfold’s Smart Chair • “Virtual Joy Stick” Phone App • App sends “joy stick” type commands to Arduino over BLE • Arduino converts commands to electric signals for wheelchair motor control BLE Eightfold Technologies Joy Stick Connection Started With an Established Interface
  7. 8.

    Eightfold’s Custom Arduino Board 5/20/2019 8 Controls Voltage Lines

    to Motor Control Arduino & HW WC Control WC Motor Phone App JoyStick
  8. 9.

    Self-Driving Wheelchair (Alpha) 5/20/2019 9 Custom Arduino USB Webcam

    Laptop - CNN • USB Webcam & Arduino connected directly to the laptop • A CNN on laptop, produces the steering drive instructions from images • Arduino converts commands to electric signals for wheelchair motor control Replaced the Virtual Remote Joystick with CNN
  9. 10.

    Keras Tensorflow CUDA POC Prototype with Pi3 & CNN 5/20/2019 10 LapTop Pi3 WC Control WC Motor Joystick Video stream opencv wifi usb usb USB Camera ROS Topics /cnn_out * /drive /chair /image “Deadman” Switch ROS Based Distributed Architecture & Open Source Packages * DroNet [2] modified for WC POC chairRx chairTx Perception * chairDrive chairReason ROS Linux ROS Linux SmartChair.ino Arduino + Shield
  10. 11.

    Results of the POC Prototype • Basic self-driving operation tested

    in a variety of environments (without navigation) • Drives on its own for short periods of time, usually stopping on its own well before running into any objects (except for low lying object like a curb). • Proven to be viable as a platform, but lots more work needed. • ** Helped to identify the next steps & focus. 5/20/2019 11 CNN as viable solution for Wheelchair Auto-Pilot
  11. 12.

    What was Learned from POC • WC drives more like

    a tank than a car! Need better data to train in order to get improvements. • Collision detection & stopping worked well overall, but missed low level objects like curbs. • Image pre-processing , mounting location/height & angle & type of camera 5/20/2019 12 Platform for Further Experimentation
  12. 13.

    Other Observations & Learnings • A lot of tedious “unglamorous”

    work: – installing software, – resolving inter-compatibility issues, – bugs in open source code, – working with the datasets, – setting up the development environments. • Lots of effort into instrumenting software & hardware to both test & debug to make sure that WC is doing the correct behavior 5/20/2019 13 Development & Testing Continues
  13. 14.

    Conclusions – Proven Possibilities • Allows those with neurological deficiencies

    to be able to accompany able-bodied companions. • Allow those with more severe mobility impairments more freedom & independence. • Autonomous driving techniques can be adapted for use in less structured areas. • ML algorithms, can be taken from academia in a low cost way, using standard engineering R&D practices. 5/20/2019 14 Applying ML to Create Independence
  14. 15.

    The Team 5/20/2019 15 Ke Wang Smart Chair Software

    Benoit Cote Smart Chair Hardware Najmeh Taleb DL Researcher, AI Algorithms Jenny Midwinter Chief Bottle Washer
  15. 17.

    References 1. End to End Learning for Self-Driving Cars, Mariusz

    Bojarski. NVIDIA Corporation. Holmdel, NJ 07735. Davide Del Testa. NVIDIA Corporation., April 16, 2016. 6/solutions/pdf/end-to-end-dl-using-px.pdf 2. “Learning to Fly by Driving”, A Loquercio et al, IEEE Robotics & Automation Letters. preprint version accepted January, 2018 . 3. Udacity Open Data Set: car/tree/master/datasets 4. DroNet Open Data set: rpg/rpg_public_dronet 5. Eightfold Technologies Inc: 6. ROS (Robotic Operating System) 7. Stewart Midwinter : articles/stewarts-story-living-life-injury/ 5/20/2019 17