Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RoboClippy

Tommy Falgout
April 04, 2019
13k

 RoboClippy

Why on God's green Earth, would you build a robotic Clippy made out of LEGO?

Details inside.

Tommy Falgout

April 04, 2019
Tweet

Transcript

  1. RoboClippy
    TOMMY FALGOUT
    CLOUD SOLUTION ARCHITECT

    View full-size slide

  2. Agenda
    u Background
    u Inspiration for Robo-Clippy
    u Hobby Electronics
    u Experiments and failures on the path
    u h/w & s/w solutions
    u Azure Cognitive Services
    u Google AIY Kit
    u Demo

    View full-size slide

  3. WARNING: We’re about to get
    into the weeds, up in here.
    EVEN FOR ME.

    View full-size slide

  4. Tommy Falgout
    u 20+ years as Unix/Linux developer
    u Graduated from Tulane University in CompSci
    u 8 years at Nortel
    u Developer on SMS (aka Text Messaging)
    u Found love for automation
    u 1 year at failed startup (checkbox met)
    u 9 years at Yahoo
    u Worked in DC/Network/Video/LiveStream automation
    u Broadcast Obama inauguration with 100k streams
    u 2 years at Microsoft
    u Hired as Technical Evangelist now Cloud Solution Architect

    View full-size slide

  5. Extra-curricular activities
    u Guest Trebuchet Expert on Nickelodeon’s “Dude Perfect”
    u https://www.imdb.com/name/nm9305352/
    u Built
    u Electronic Applause-o-Tron
    u LEGO Train set around my office
    u Whiskey Chandelier
    u Multiple Trebuchets
    u Organized
    u Board Member of TheLab.ms (Local Makerspace)
    u LEGO Contest for Texas Society of Architects
    u SlingFest (Local version of Punkin’ Chunkin’)
    u Classes
    u “How to solve a Rubik’s Cube”
    u “Kubernetes and Waffles”

    View full-size slide

  6. RoboClippy Inspiration
    I.E. WHY ON GOD’S GREEN EARTH WOULD YOU DO THIS?

    View full-size slide

  7. Credit: https://www.instructables.com/id/Animate-a-Billy-Bass-Mouth-With-Any-Audio-Source/

    View full-size slide

  8. https://www.amazon.com/gp/product/B07657LKNZ/

    View full-size slide

  9. Goal: Use Azure AI in an OCP P20
    Hackathon

    View full-size slide

  10. Requirements Resources
    u Portable
    u Utilize Microsoft AI
    u “Wow” factor
    u LEGO
    u Makerspace
    u Used LEGO retailer
    u Friends
    u OSS
    u Arduino
    u Raspberry Pi

    View full-size slide

  11. Arduino
    u What is Arduino?
    u Arduino is an open-source electronics platform based on easy-to-use
    hardware and software.
    u Arduino boards are able to read inputs - light on a sensor, a finger on a
    button, or a Twitter message - and turn it into an output - activating a motor,
    turning on an LED, publishing something online. You can tell your board
    what to do by sending a set of instructions to the microcontroller on the
    board.
    u Benefits
    u Easy to understand and create
    u Cheap-ish
    u Good IDE
    u Write code in C/C++
    u Large ecosystem
    u Works with Azure IoT
    u Many I/O ports

    View full-size slide

  12. Raspberry Pi
    u What is Raspberry Pi?
    u The Raspberry Pi is a series of small single-board computers developed in
    the United Kingdom by the Raspberry Pi Foundation to promote teaching of
    basic computer science in schools and in developing countries
    u Benefits
    u Ports: USB, HDMI, SD Card, Audio jack
    u Cheap-ish
    u Can run Linux, BSD, Win10 IoT Core
    u Write code in whatever
    u Full networking, CPU, Memory stack
    u Works with Azure IoT
    u Downsides
    u Fewer I/O ports
    u Less focused on motor controls

    View full-size slide

  13. Important Questions
    u How do you make RoboClippy?
    u How big should RoboClippy be?
    u How do I build RoboClippy?
    u Materials?
    u Articulation?
    u Portability?

    View full-size slide

  14. RoboClippy Mark I

    View full-size slide

  15. Status
    u Cute, but uses a competitor
    u Definitively, not a paperclip
    u Eyebrows wiggle, but only manually

    View full-size slide

  16. Uncanny Valley
    u From Wikipedia: The concept of the uncanny valley
    suggests humanoid objects which appear almost, but not exactly,
    like real human beings elicit uncanny, or strangely familiar, feelings
    of eeriness and revulsion in observers.

    View full-size slide

  17. Uncanny Valley
    u From Wikipedia: The concept of the uncanny valley
    suggests humanoid objects which appear almost, but not exactly,
    like real human beings elicit uncanny, or strangely familiar, feelings
    of eeriness and revulsion in observers.
    u Surprise! Same goes for paperclips

    View full-size slide

  18. RoboClippy Mark II

    View full-size slide

  19. Status
    u More like a paperclip!
    u Flimsy, clumsy body
    u Difficult to articulate

    View full-size slide

  20. Animatronics done wrong is
    nightmare fuel

    View full-size slide

  21. RoboClippy Mark III

    View full-size slide

  22. Status
    u Eyebrows articulate!
    u But very basic. One side goes up, other goes down
    u Doesn’t look natural
    u No fine motor control

    View full-size slide

  23. LEGO Mindstorms

    View full-size slide

  24. Proprietary interfaces = Difficult
    hacking
    u Was designed to only interact with LEGO power supplies.
    u LEGO Motors Uses 9V. USB uses 5V
    u LEGO connectors have weird wiring
    u Whitepapers have been written on LEGO Motor specs
    u https://www.egr.msu.edu/classes/ece480/capstone/fall10/group02/do
    c/Brouker%20Application%20Note.pdf
    u Cabling is not standard

    View full-size slide

  25. Almost like CAT 5. But not

    View full-size slide

  26. Not all motors are created equal
    u Servo motors – Good for broad movements
    u Apply voltage -> turn left/right
    u Can control torque
    u Pulsate back and forth
    u Stepper motors – Good for fine movements
    u Apply voltage -> go to position X
    u Use Pulse-Width-Modulation (PWM) for communication
    u Cannot control torque
    u Able to stay completely still

    View full-size slide

  27. Controlling multiple PWM motors
    u I2C PWM Driver Board
    u I2C = two-wire Protocol controlling low-speed devices
    u PWM – Pulse Width Modulation

    View full-size slide

  28. Controlling the Servos
    u Use I2C Driver Board to send PWM signals to motors
    u Each motor has a different address
    u 3 Motors
    u Mouth
    u Left Eye
    u Right Eye

    View full-size slide

  29. Arduino Code

    View full-size slide

  30. RoboClippy Mark IV

    View full-size slide

  31. RoboClippy Mark IV

    View full-size slide

  32. Status
    u Great structure!
    u Mouth articulates!
    u Eyes articulate! (using stilts + motors)
    u Lacks a ”soul”
    u Soul of a New Machine?

    View full-size slide

  33. Making RoboClippy “alive”
    u Goals
    u Move mouth when speaking
    u Simulate Clippy’s Voice
    u Articulating eyebrows

    View full-size slide

  34. Azure Cognitive Services
    u Speech
    u Speech to Text
    u Text to Speech
    u Speaker Recognition
    u Speech Translation
    u Vision
    u Video Indexer – Used for my Ring Doorbell
    u Face Detection
    u Image Classification – Used for Whisk(e)y recommendation
    u Knowledge
    u Search
    u Language

    View full-size slide

  35. Calculating volume
    u Fact: Speakers usually use two wires.
    u Assumption: Measure the voltage, calculate
    volume, right?

    View full-size slide

  36. Calculating volume
    u Fact: Speakers usually use two wires.
    u Assumption: Measure the voltage, calculate
    volume, right?

    View full-size slide

  37. Sound is a wave
    u Measuring sound at any point only
    gets you a snapshot.
    u Varies greatly on frequency
    u Most microphones measure -2.5V
    to 2.5V
    u Arduino measures 0V to 5V
    u Missing half the data!

    View full-size slide

  38. Step-up Converter
    u From Wikipedia: A boost converter (step-up converter) is a DC-to-DC
    power converter that steps up voltage (while stepping down
    current) from its input (supply) to its output (load).
    u TL;DR: Convert -2.5V to 2.5V to 0V to 5V

    View full-size slide

  39. Step-up Converter

    View full-size slide

  40. RoboClippy Mark IV

    View full-size slide

  41. Status
    u Needed 2 audio out
    u Speaker
    u Arduino
    u Audio splitter wasn’t working (reduced current by half)
    u Wasn’t easily portable
    u Took 5-10 minutes to hook up each time
    u Arduino is powered by USB connected to computer
    u Sometimes drew too much current (potential laptop killer)
    u Flakey results

    View full-size slide

  42. Google’s AIY Voice Kit

    View full-size slide

  43. Google AIY Voice Kit

    View full-size slide

  44. Making RoboClippy “talk”
    u How do we know when start listening?
    u How do we know when to stop listening?
    u What’s the quickest way to respond?
    u How can we best utilize Azure?

    View full-size slide

  45. Speech Recognition
    u Python library for speech recognition
    u Provides online and offline engines
    u Microsoft Bing, Google, IBM, Snowboy, CMU Sphinx
    u Combined with Snowboy, does a good job of capturing speech
    u PR to Replace Bing Speech API with Azure Speech API
    u https://github.com/Uberi/speech_recognition/pull/389
    u https://github.com/Uberi/speech_recognition
    u Tried offline T2S.
    u Took about 1 minute for 5 second audio clip
    u SaaS model is the way to go!

    View full-size slide

  46. Snowboy
    u Hotwork Detection
    u E.g. “Alexa”, “OK Google”, “Hey Clippy”
    u Record and download your own model for free
    u https://snowboy.kitt.ai/hotword/27993
    u https://snowboy.kitt.ai/

    View full-size slide

  47. RoboClippy Mark V

    View full-size slide

  48. Status
    u MUCH more extensible!
    u Hybrid solution
    u Use Google h/w for audio
    u Use Azure PaaS for speech
    u I2C for Motor control
    u Raspberry Pi for orchestration
    u Decoupled laptop from RoboClippy
    u Power + Control (ssh + python) is now remote
    u 5 easy-to-connect wires (4 for I2C, 1 USB for power)
    u Written in Python now OSS
    u https://github.com/lastcoolnameleft/robo-clippy

    View full-size slide

  49. Behind the scenes
    u Wiring
    u Repurposed Apple USB cable for motor power
    u 2 wires for controller power
    u 2 wires for controller signals
    u RaspberryPi
    u Listens on microphone
    u Uses speaker
    u Coordinates with Azure Cognitive Services

    View full-size slide

  50. Technologies
    u Raspberry Pi 3B+
    u Azure Cognitive Services – T2S/S2T
    u Google AIY RPi kit – Voice Hardware-Attached-on-Top (HAT)
    u LEGO - structure
    u I2C PWM Driver – Servo manipulation
    u Snowboy – Hotword detection
    u Python 3 – Glue code
    https://github.com/lastcoolnameleft/robo-clippy

    View full-size slide

  51. Next Steps
    u Where do I go from here?
    u Will RoboClippy become Skynet?
    u Coordinate with MS Marketing?
    u Do I need to replace Google AIY Voice kit or is the inter-op cool?
    u Do I integrate LUIS for NLP?

    View full-size slide

  52. Social Media
    u RoboClippy at Bricks and Minifigs
    u Steven Sinofsky (Clippy Creator) Retweeted RoboClippy videos

    View full-size slide

  53. Summary
    u Learned WAY more than I expected
    u Electronics, RPi, Arduino, Sound, NLP, T2S, S2T, I2C, PWM
    u Builders love a challenge
    u LEGO RoboClippy skeleton was done in a few hours
    u You have more resources available than you expect
    u People are both horrified and excited about Clippy’s resurgence
    u Embrace it

    View full-size slide