CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

Jenny Midwinter at May 21, 2019 event of montrealml.dev

Title: CNN Based Auto-Pilot for a Wheelchair: A brief story of the results of rapid development of a CNN based self driving wheelchair using open source software & low cost hardware

Summary: This lightening talk describes the development & results that were successfully used to create a proof-of-concept (POC) prototype for a CNN based auto-pilot function that was able to autonomously drive a wheelchair based on a single input image stream from forward facing camera . A brief explanation of how the CNN is used to autonomously drive the wheelchair is provided, including a short video demo. As well, a description of the low-cost hardware/software system, & the learnings that were obtained, now being used as a basis for the next stage of development, will also be presented.

Bio: Jenny is collaborating as Chief Scientist, Applied AI, at Eightfold Technologies, and responsible for the development of vision based self-driving technology for Eightfold’s SmartChair. She is the CEO and founder of Blue Horizon AI, a start up dedicated to applying AI in autonomous technologies to assist the disabled. Prior to this, she has 25 years experience in R&D in the Telecom Industry.

Transcript

1. CNN Based Auto-Pilot for a Wheelchair A brief story of rapid development of a proof- of-concept self driving wheelchair using open source software & low cost hardware. 5/20/2019 jenny@bluehorizon.ai 1 Jenny Midwinter
2. 5/20/2019 jenny@bluehorizon.ai 2 Stewart Midwinter • Address the extreme difficulty some power wheelchair users have in getting out in their environment. • Cognitive & neurological challenges to drive a wheelchair. It is actually hard! (even for an able bodied person)
3. Self-Driving Wheelchair: Goals • Provide “lane assist” to autonomously follow a pathway, sidewalk, or hallway. • Low cost & easy to retrofit to user’s chair. • Open source the software. • Other applications with WC Autonomous Movement: – Follow-Me/Accompany-Me, – Automated Docking for Battery Recharging 5/20/2019 jenny@bluehorizon.ai 3
4. Self-Driving Wheelchair – Early Decisions • Early Technical Decisions: “Strawman” – Use of CNN to predict a steering angle from a forward facing camera inspired by “End to End Learning for Self- Driving Cars” [1] – Use Eightfold Smart Chair a proven working product, able to remotely & programmatically deliver driving instructions to a wheelchair motor control system [5] – Use ROS (Robotic Operating System) [6] – Short-cut to POC … pre-trained CNN (“Learning to Fly by Driving” [2] ) since trained on a relevant & applicable open data sets. [3] [4] 5/20/2019 jenny@bluehorizon.ai 4 Build a Proof of Concept Prototype
5. WC POC Auto-Pilot Run-Time Process 5/20/2019 jenny@bluehorizon.ai 5 Image Pre-Processing CNN Joy Stick Equivalent Commands 4) Create a “joy stick” equivalent command that is sent to WC Motor control to drive the wheelchair 1) Input Images from a Single Forward Facing Camera 2) Image Cropping, Re-size, Grayscale Steering Angle Collision Probability 3) Trained Convolutional Neural Network produces steering Angle & probability of collision
6. Demo of Results “Beta” 5/20/2019 jenny@bluehorizon.ai 6
7. Eightfold’s Smart Chair: A Remote Virtual Joystick 5/20/2019 jenny@bluehorizon.ai 7 Eightfold Technologies Custom Arduino Board SmartPhone Eightfold’s Smart Chair • “Virtual Joy Stick” Phone App • App sends “joy stick” type commands to Arduino over BLE • Arduino converts commands to electric signals for wheelchair motor control BLE Eightfold Technologies Joy Stick Connection Started With an Established Interface
8. Eightfold’s Custom Arduino Board 5/20/2019 jenny@bluehorizon.ai 8 Controls Voltage Lines to Motor Control Arduino & HW WC Control WC Motor Phone App JoyStick
9. Self-Driving Wheelchair (Alpha) 5/20/2019 jenny@bluehorizon.ai 9 Custom Arduino USB Webcam Laptop - CNN • USB Webcam & Arduino connected directly to the laptop • A CNN on laptop, produces the steering drive instructions from images • Arduino converts commands to electric signals for wheelchair motor control Replaced the Virtual Remote Joystick with CNN
10. Keras Tensorflow CUDA POC Prototype with Pi3 & CNN 5/20/2019 jenny@bluehorizon.ai 10 LapTop Pi3 WC Control WC Motor Joystick Video stream opencv wifi usb usb USB Camera ROS Topics /cnn_out * /drive /chair /image “Deadman” Switch ROS Based Distributed Architecture & Open Source Packages * DroNet [2] modified for WC POC chairRx chairTx Perception * chairDrive chairReason ROS Linux ROS Linux SmartChair.ino Arduino + Shield
11. Results of the POC Prototype • Basic self-driving operation tested in a variety of environments (without navigation) • Drives on its own for short periods of time, usually stopping on its own well before running into any objects (except for low lying object like a curb). • Proven to be viable as a platform, but lots more work needed. • ** Helped to identify the next steps & focus. 5/20/2019 jenny@bluehorizon.ai 11 CNN as viable solution for Wheelchair Auto-Pilot
12. What was Learned from POC • WC drives more like a tank than a car! Need better data to train in order to get improvements. • Collision detection & stopping worked well overall, but missed low level objects like curbs. • Image pre-processing , mounting location/height & angle & type of camera 5/20/2019 jenny@bluehorizon.ai 12 Platform for Further Experimentation
13. Other Observations & Learnings • A lot of tedious “unglamorous” work: – installing software, – resolving inter-compatibility issues, – bugs in open source code, – working with the datasets, – setting up the development environments. • Lots of effort into instrumenting software & hardware to both test & debug to make sure that WC is doing the correct behavior 5/20/2019 jenny@bluehorizon.ai 13 Development & Testing Continues
14. Conclusions – Proven Possibilities • Allows those with neurological deficiencies to be able to accompany able-bodied companions. • Allow those with more severe mobility impairments more freedom & independence. • Autonomous driving techniques can be adapted for use in less structured areas. • ML algorithms, can be taken from academia in a low cost way, using standard engineering R&D practices. 5/20/2019 jenny@bluehorizon.ai 14 Applying ML to Create Independence
15. The Team 5/20/2019 jenny@bluehorizon.ai 15 Ke Wang Smart Chair Software Benoit Cote Smart Chair Hardware Najmeh Taleb DL Researcher, AI Algorithms Jenny Midwinter Chief Bottle Washer
16. Blooper – Running Into Curb 5/20/2019 jenny@bluehorizon.ai 16
17. References 1. End to End Learning for Self-Driving Cars, Mariusz Bojarski. NVIDIA Corporation. Holmdel, NJ 07735. Davide Del Testa. NVIDIA Corporation., April 16, 2016. https://images.nvidia.com/content/tegra/automotive/images/201 6/solutions/pdf/end-to-end-dl-using-px.pdf 2. “Learning to Fly by Driving”, A Loquercio et al, IEEE Robotics & Automation Letters. preprint version accepted January, 2018 . http://rpg.ifi.uzh.ch/docs/RAL18_Loquercio.pdf 3. Udacity Open Data Set: https://github.com/udacity/self-driving- car/tree/master/datasets 4. DroNet Open Data set: https://github.com/uzh- rpg/rpg_public_dronet 5. Eightfold Technologies Inc: www.eightfoldtech.com 6. ROS (Robotic Operating System) http://www.ros.org/ 7. Stewart Midwinter : https://xcmag.com/magazine- articles/stewarts-story-living-life-injury/ 5/20/2019 jenny@bluehorizon.ai 17