Tweet
Tweet

Slide 1

Slide 1 text

RoboClippy TOMMY FALGOUT CLOUD SOLUTION ARCHITECT

Slide 2

Slide 2 text

Agenda u Background u Inspiration for Robo-Clippy u Hobby Electronics u Experiments and failures on the path u h/w & s/w solutions u Azure Cognitive Services u Google AIY Kit u Demo

Slide 3

Slide 3 text

WARNING: We’re about to get into the weeds, up in here. EVEN FOR ME.

Slide 4

Slide 4 text

Tommy Falgout u 20+ years as Unix/Linux developer u Graduated from Tulane University in CompSci u 8 years at Nortel u Developer on SMS (aka Text Messaging) u Found love for automation u 1 year at failed startup (checkbox met) u 9 years at Yahoo u Worked in DC/Network/Video/LiveStream automation u Broadcast Obama inauguration with 100k streams u 2 years at Microsoft u Hired as Technical Evangelist now Cloud Solution Architect

Slide 5

Slide 5 text

Extra-curricular activities u Guest Trebuchet Expert on Nickelodeon’s “Dude Perfect” u https://www.imdb.com/name/nm9305352/ u Built u Electronic Applause-o-Tron u LEGO Train set around my office u Whiskey Chandelier u Multiple Trebuchets u Organized u Board Member of TheLab.ms (Local Makerspace) u LEGO Contest for Texas Society of Architects u SlingFest (Local version of Punkin’ Chunkin’) u Classes u “How to solve a Rubik’s Cube” u “Kubernetes and Waffles”

Slide 6

Slide 6 text

RoboClippy Inspiration I.E. WHY ON GOD’S GREEN EARTH WOULD YOU DO THIS?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Credit: https://www.instructables.com/id/Animate-a-Billy-Bass-Mouth-With-Any-Audio-Source/

Slide 9

Slide 9 text

https://www.amazon.com/gp/product/B07657LKNZ/

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Goal: Use Azure AI in an OCP P20 Hackathon

Slide 12

Slide 12 text

Requirements Resources u Portable u Utilize Microsoft AI u “Wow” factor u LEGO u Makerspace u Used LEGO retailer u Friends u OSS u Arduino u Raspberry Pi

Slide 13

Slide 13 text

QED

Slide 14

Slide 14 text

Arduino u What is Arduino? u Arduino is an open-source electronics platform based on easy-to-use hardware and software. u Arduino boards are able to read inputs - light on a sensor, a finger on a button, or a Twitter message - and turn it into an output - activating a motor, turning on an LED, publishing something online. You can tell your board what to do by sending a set of instructions to the microcontroller on the board. u Benefits u Easy to understand and create u Cheap-ish u Good IDE u Write code in C/C++ u Large ecosystem u Works with Azure IoT u Many I/O ports

Slide 15

Slide 15 text

Raspberry Pi u What is Raspberry Pi? u The Raspberry Pi is a series of small single-board computers developed in the United Kingdom by the Raspberry Pi Foundation to promote teaching of basic computer science in schools and in developing countries u Benefits u Ports: USB, HDMI, SD Card, Audio jack u Cheap-ish u Can run Linux, BSD, Win10 IoT Core u Write code in whatever u Full networking, CPU, Memory stack u Works with Azure IoT u Downsides u Fewer I/O ports u Less focused on motor controls

Slide 16

Slide 16 text

Important Questions u How do you make RoboClippy? u How big should RoboClippy be? u How do I build RoboClippy? u Materials? u Articulation? u Portability?

Slide 17

Slide 17 text

PROTOTYPING

Slide 18

Slide 18 text

RoboClippy Mark I

Slide 19

Slide 19 text

Status u Cute, but uses a competitor u Definitively, not a paperclip u Eyebrows wiggle, but only manually

Slide 20

Slide 20 text

Uncanny Valley u From Wikipedia: The concept of the uncanny valley suggests humanoid objects which appear almost, but not exactly, like real human beings elicit uncanny, or strangely familiar, feelings of eeriness and revulsion in observers.

Slide 21

Slide 21 text

Uncanny Valley u From Wikipedia: The concept of the uncanny valley suggests humanoid objects which appear almost, but not exactly, like real human beings elicit uncanny, or strangely familiar, feelings of eeriness and revulsion in observers. u Surprise! Same goes for paperclips

Slide 22

Slide 22 text

RoboClippy Mark II

Slide 23

Slide 23 text

Status u More like a paperclip! u Flimsy, clumsy body u Difficult to articulate

Slide 24

Slide 24 text

Animatronics done wrong is nightmare fuel

Slide 25

Slide 25 text

RoboClippy Mark III

Slide 26

Slide 26 text

Status u Eyebrows articulate! u But very basic. One side goes up, other goes down u Doesn’t look natural u No fine motor control

Slide 27

Slide 27 text

LEGO Mindstorms

Slide 28

Slide 28 text

Proprietary interfaces = Difficult hacking u Was designed to only interact with LEGO power supplies. u LEGO Motors Uses 9V. USB uses 5V u LEGO connectors have weird wiring u Whitepapers have been written on LEGO Motor specs u https://www.egr.msu.edu/classes/ece480/capstone/fall10/group02/do c/Brouker%20Application%20Note.pdf u Cabling is not standard

Slide 29

Slide 29 text

Almost like CAT 5. But not

Slide 30

Slide 30 text

Not all motors are created equal u Servo motors – Good for broad movements u Apply voltage -> turn left/right u Can control torque u Pulsate back and forth u Stepper motors – Good for fine movements u Apply voltage -> go to position X u Use Pulse-Width-Modulation (PWM) for communication u Cannot control torque u Able to stay completely still

Slide 31

Slide 31 text

Controlling multiple PWM motors u I2C PWM Driver Board u I2C = two-wire Protocol controlling low-speed devices u PWM – Pulse Width Modulation

Slide 32

Slide 32 text

Controlling the Servos u Use I2C Driver Board to send PWM signals to motors u Each motor has a different address u 3 Motors u Mouth u Left Eye u Right Eye

Slide 33

Slide 33 text

Arduino Code

Slide 34

Slide 34 text

RoboClippy Mark IV

Slide 35

Slide 35 text

RoboClippy Mark IV

Slide 36

Slide 36 text

Status u Great structure! u Mouth articulates! u Eyes articulate! (using stilts + motors) u Lacks a ”soul” u Soul of a New Machine?

Slide 37

Slide 37 text

Making RoboClippy “alive” u Goals u Move mouth when speaking u Simulate Clippy’s Voice u Articulating eyebrows

Slide 38

Slide 38 text

Azure Cognitive Services u Speech u Speech to Text u Text to Speech u Speaker Recognition u Speech Translation u Vision u Video Indexer – Used for my Ring Doorbell u Face Detection u Image Classification – Used for Whisk(e)y recommendation u Knowledge u Search u Language

Slide 39

Slide 39 text

Calculating volume u Fact: Speakers usually use two wires. u Assumption: Measure the voltage, calculate volume, right?

Slide 40

Slide 40 text

Calculating volume u Fact: Speakers usually use two wires. u Assumption: Measure the voltage, calculate volume, right?

Slide 41

Slide 41 text

Sound is a wave u Measuring sound at any point only gets you a snapshot. u Varies greatly on frequency u Most microphones measure -2.5V to 2.5V u Arduino measures 0V to 5V u Missing half the data!

Slide 42

Slide 42 text

Step-up Converter u From Wikipedia: A boost converter (step-up converter) is a DC-to-DC power converter that steps up voltage (while stepping down current) from its input (supply) to its output (load). u TL;DR: Convert -2.5V to 2.5V to 0V to 5V

Slide 43

Slide 43 text

Step-up Converter

Slide 44

Slide 44 text

RoboClippy Mark IV

Slide 45

Slide 45 text

Status u Needed 2 audio out u Speaker u Arduino u Audio splitter wasn’t working (reduced current by half) u Wasn’t easily portable u Took 5-10 minutes to hook up each time u Arduino is powered by USB connected to computer u Sometimes drew too much current (potential laptop killer) u Flakey results

Slide 46

Slide 46 text

Google’s AIY Voice Kit

Slide 47

Slide 47 text

Google AIY Voice Kit

Slide 48

Slide 48 text

Making RoboClippy “talk” u How do we know when start listening? u How do we know when to stop listening? u What’s the quickest way to respond? u How can we best utilize Azure?

Slide 49

Slide 49 text

Speech Recognition u Python library for speech recognition u Provides online and offline engines u Microsoft Bing, Google, IBM, Snowboy, CMU Sphinx u Combined with Snowboy, does a good job of capturing speech u PR to Replace Bing Speech API with Azure Speech API u https://github.com/Uberi/speech_recognition/pull/389 u https://github.com/Uberi/speech_recognition u Tried offline T2S. u Took about 1 minute for 5 second audio clip u SaaS model is the way to go!

Slide 50

Slide 50 text

Snowboy u Hotwork Detection u E.g. “Alexa”, “OK Google”, “Hey Clippy” u Record and download your own model for free u https://snowboy.kitt.ai/hotword/27993 u https://snowboy.kitt.ai/

Slide 51

Slide 51 text

RoboClippy Mark V

Slide 52

Slide 52 text

Status u MUCH more extensible! u Hybrid solution u Use Google h/w for audio u Use Azure PaaS for speech u I2C for Motor control u Raspberry Pi for orchestration u Decoupled laptop from RoboClippy u Power + Control (ssh + python) is now remote u 5 easy-to-connect wires (4 for I2C, 1 USB for power) u Written in Python now OSS u https://github.com/lastcoolnameleft/robo-clippy

Slide 53

Slide 53 text

Behind the scenes u Wiring u Repurposed Apple USB cable for motor power u 2 wires for controller power u 2 wires for controller signals u RaspberryPi u Listens on microphone u Uses speaker u Coordinates with Azure Cognitive Services

Slide 54

Slide 54 text

Technologies u Raspberry Pi 3B+ u Azure Cognitive Services – T2S/S2T u Google AIY RPi kit – Voice Hardware-Attached-on-Top (HAT) u LEGO - structure u I2C PWM Driver – Servo manipulation u Snowboy – Hotword detection u Python 3 – Glue code https://github.com/lastcoolnameleft/robo-clippy

Slide 55

Slide 55 text

DEMO TIME!

Slide 56

Slide 56 text

Next Steps u Where do I go from here? u Will RoboClippy become Skynet? u Coordinate with MS Marketing? u Do I need to replace Google AIY Voice kit or is the inter-op cool? u Do I integrate LUIS for NLP?

Slide 57

Slide 57 text

Social Media u RoboClippy at Bricks and Minifigs u Steven Sinofsky (Clippy Creator) Retweeted RoboClippy videos

Slide 58

Slide 58 text

Summary u Learned WAY more than I expected u Electronics, RPi, Arduino, Sound, NLP, T2S, S2T, I2C, PWM u Builders love a challenge u LEGO RoboClippy skeleton was done in a few hours u You have more resources available than you expect u People are both horrified and excited about Clippy’s resurgence u Embrace it

Slide 59

Slide 59 text

Thank you!