$30 off During Our Annual Pro Sale. View Details »

CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

CNN Based Auto-Pilot for a Wheelchair - Jenny Midwinter

Jenny Midwinter at May 21, 2019 event of montrealml.dev

Title: CNN Based Auto-Pilot for a Wheelchair: A brief story of the results of rapid development of a CNN based self driving wheelchair using open source software & low cost hardware

Summary: This lightening talk describes the development & results that were successfully used to create a proof-of-concept (POC) prototype for a CNN based auto-pilot function that was able to autonomously drive a wheelchair based on a single input image stream from forward facing camera . A brief explanation of how the CNN is used to autonomously drive the wheelchair is provided, including a short video demo. As well, a description of the low-cost hardware/software system, & the learnings that were obtained, now being used as a basis for the next stage of development, will also be presented.

Bio: Jenny is collaborating as Chief Scientist, Applied AI, at Eightfold Technologies, and responsible for the development of vision based self-driving technology for Eightfold’s SmartChair. She is the CEO and founder of Blue Horizon AI, a start up dedicated to applying AI in autonomous technologies to assist the disabled. Prior to this, she has 25 years experience in R&D in the Telecom Industry.

PatternedScience

May 21, 2019
Tweet

More Decks by PatternedScience

Other Decks in Technology

Transcript

  1. CNN Based Auto-Pilot for a
    Wheelchair
    A brief story of rapid development of a proof-
    of-concept self driving wheelchair using open
    source software & low cost hardware.
    5/20/2019 [email protected] 1
    Jenny Midwinter

    View Slide

  2. 5/20/2019 [email protected] 2
    Stewart Midwinter
    • Address the extreme
    difficulty some power
    wheelchair users have in
    getting out in their
    environment.
    • Cognitive & neurological
    challenges to drive a
    wheelchair. It is actually
    hard! (even for an able
    bodied person)

    View Slide

  3. Self-Driving Wheelchair: Goals
    • Provide “lane assist” to autonomously follow a
    pathway, sidewalk, or hallway.
    • Low cost & easy to retrofit to user’s chair.
    • Open source the software.
    • Other applications with WC Autonomous Movement:
    – Follow-Me/Accompany-Me,
    – Automated Docking for Battery Recharging
    5/20/2019 [email protected] 3

    View Slide

  4. Self-Driving Wheelchair – Early Decisions
    • Early Technical Decisions: “Strawman”
    – Use of CNN to predict a steering angle from a forward
    facing camera inspired by “End to End Learning for Self-
    Driving Cars” [1]
    – Use Eightfold Smart Chair a proven working product, able
    to remotely & programmatically deliver driving
    instructions to a wheelchair motor control system [5]
    – Use ROS (Robotic Operating System) [6]
    – Short-cut to POC … pre-trained CNN (“Learning to Fly by
    Driving” [2] ) since trained on a relevant & applicable open
    data sets. [3] [4]
    5/20/2019 [email protected] 4
    Build a Proof of Concept Prototype

    View Slide

  5. WC POC Auto-Pilot Run-Time Process
    5/20/2019 [email protected] 5
    Image Pre-Processing
    CNN
    Joy Stick Equivalent Commands
    4) Create a “joy stick”
    equivalent command that
    is sent to WC Motor control
    to drive the wheelchair
    1) Input Images from a
    Single Forward Facing
    Camera
    2) Image Cropping, Re-size,
    Grayscale
    Steering Angle Collision Probability
    3) Trained Convolutional
    Neural Network produces
    steering Angle &
    probability of collision

    View Slide

  6. Demo of
    Results
    “Beta”
    5/20/2019 [email protected] 6

    View Slide

  7. Eightfold’s Smart Chair: A Remote Virtual Joystick
    5/20/2019
    [email protected]
    7
    Eightfold
    Technologies
    Custom
    Arduino Board
    SmartPhone
    Eightfold’s Smart Chair
    • “Virtual Joy Stick” Phone
    App
    • App sends “joy stick” type
    commands to Arduino over
    BLE
    • Arduino converts commands
    to electric signals for
    wheelchair motor control
    BLE
    Eightfold
    Technologies
    Joy Stick
    Connection
    Started With an Established Interface

    View Slide

  8. Eightfold’s Custom Arduino Board
    5/20/2019 [email protected] 8
    Controls Voltage Lines to Motor Control
    Arduino
    & HW
    WC Control
    WC Motor
    Phone App
    JoyStick

    View Slide

  9. Self-Driving Wheelchair (Alpha)
    5/20/2019
    [email protected]
    9
    Custom
    Arduino
    USB Webcam
    Laptop - CNN • USB Webcam & Arduino
    connected directly to the
    laptop
    • A CNN on laptop, produces
    the steering drive instructions
    from images
    • Arduino converts commands
    to electric signals for
    wheelchair motor control
    Replaced the Virtual Remote Joystick with CNN

    View Slide

  10. Keras
    Tensorflow
    CUDA
    POC Prototype with Pi3 & CNN
    5/20/2019 [email protected] 10
    LapTop
    Pi3
    WC Control
    WC Motor
    Joystick
    Video stream opencv
    wifi
    usb
    usb
    USB Camera
    ROS Topics
    /cnn_out *
    /drive
    /chair
    /image
    “Deadman” Switch
    ROS Based Distributed Architecture & Open Source Packages
    * DroNet [2] modified for WC POC
    chairRx
    chairTx
    Perception *
    chairDrive
    chairReason
    ROS
    Linux
    ROS
    Linux
    SmartChair.ino
    Arduino + Shield

    View Slide

  11. Results of the POC Prototype
    • Basic self-driving operation tested in a variety of
    environments (without navigation)
    • Drives on its own for short periods of time, usually
    stopping on its own well before running into any
    objects (except for low lying object like a curb).
    • Proven to be viable as a platform, but lots more
    work needed.
    • ** Helped to identify the next steps & focus.
    5/20/2019 [email protected] 11
    CNN as viable solution for Wheelchair Auto-Pilot

    View Slide

  12. What was Learned from POC
    • WC drives more like a tank than a car! Need better
    data to train in order to get improvements.
    • Collision detection & stopping worked well overall,
    but missed low level objects like curbs.
    • Image pre-processing , mounting location/height &
    angle & type of camera
    5/20/2019 [email protected] 12
    Platform for Further Experimentation

    View Slide

  13. Other Observations & Learnings
    • A lot of tedious “unglamorous” work:
    – installing software,
    – resolving inter-compatibility issues,
    – bugs in open source code,
    – working with the datasets,
    – setting up the development environments.
    • Lots of effort into instrumenting software &
    hardware to both test & debug to make sure that
    WC is doing the correct behavior
    5/20/2019 [email protected] 13
    Development & Testing Continues

    View Slide

  14. Conclusions – Proven Possibilities
    • Allows those with neurological deficiencies to be
    able to accompany able-bodied companions.
    • Allow those with more severe mobility
    impairments more freedom & independence.
    • Autonomous driving techniques can be adapted
    for use in less structured areas.
    • ML algorithms, can be taken from academia in a
    low cost way, using standard engineering R&D
    practices.
    5/20/2019 [email protected] 14
    Applying ML to Create Independence

    View Slide

  15. The Team
    5/20/2019 [email protected] 15
    Ke Wang
    Smart Chair
    Software
    Benoit Cote
    Smart Chair
    Hardware
    Najmeh Taleb
    DL Researcher,
    AI Algorithms
    Jenny Midwinter
    Chief Bottle
    Washer

    View Slide

  16. Blooper –
    Running Into
    Curb
    5/20/2019 [email protected] 16

    View Slide

  17. References
    1. End to End Learning for Self-Driving Cars, Mariusz
    Bojarski. NVIDIA Corporation. Holmdel, NJ 07735. Davide Del
    Testa. NVIDIA Corporation., April 16, 2016.
    https://images.nvidia.com/content/tegra/automotive/images/201
    6/solutions/pdf/end-to-end-dl-using-px.pdf
    2. “Learning to Fly by Driving”, A Loquercio et al, IEEE Robotics &
    Automation Letters. preprint version accepted January, 2018 .
    http://rpg.ifi.uzh.ch/docs/RAL18_Loquercio.pdf
    3. Udacity Open Data Set: https://github.com/udacity/self-driving-
    car/tree/master/datasets
    4. DroNet Open Data set: https://github.com/uzh-
    rpg/rpg_public_dronet
    5. Eightfold Technologies Inc: www.eightfoldtech.com
    6. ROS (Robotic Operating System) http://www.ros.org/
    7. Stewart Midwinter : https://xcmag.com/magazine-
    articles/stewarts-story-living-life-injury/
    5/20/2019 [email protected] 17

    View Slide