Let's Get Chatty with Conversational Interface in JavaScript

Slide 1

Slide 1 text

Let's Get Chatty with Conversational Interface in JavaScript BOTS, AI, AND JAVASCRIPT Tomomi Imura (@girlie_mac) #RenderConf

Slide 2

Slide 2 text

@ girlie_mac ● an open web & tech advocate ● code JavaScript / Node.js ● work at Slack ● an advisor at Code Chrysalis ● a cat lady of the InterWeb tomomi imura @girlie_mac

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

"Bots are like new applications that you can converse with. " -- Satya Nadella, Microsoft

Slide 5

Slide 5 text

@ girlie_mac “We will evolve in computing from a mobile first to an AI first world.” -- Sundar Pichai, Google

Slide 6

Slide 6 text

@ girlie_mac CC BY-SA: nextdayblinds.com

Slide 7

Slide 7 text

@ girlie_mac Traditional Web & App Interactions

Slide 8

Slide 8 text

@ girlie_mac Modern Web & Apps with Social Interactions

Slide 9

Slide 9 text

@ girlie_mac Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri In various form-factors

Slide 10

Slide 10 text

@ girlie_mac Conversational User Interactions for Kids - with Voice CogniToys Dino - https://cognitoys.com

Slide 11

Slide 11 text

@ girlie_mac Conversational User Interactions in a robot shape - with Voice

Slide 12

Slide 12 text

@ girlie_mac Conversational User Interactions: Google Assistant (Voice & Text) (GIF animation)

Slide 13

Slide 13 text

@ girlie_mac Conversational User Interactions: Slack Bots (Text with GUI) !

Slide 14

Slide 14 text

Graphic Interface to Conversational Interface

Slide 15

Slide 15 text

@ girlie_mac Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust cha ching!

Slide 16

Slide 16 text

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

Slide 17

Slide 17 text

No UI Clutter. Less Time Spent.

Slide 18

Slide 18 text

@ girlie_mac Where is the address of your home? It’s 325 Ruby Street. Request a ride Come over to my place!

Slide 19

Slide 19 text

@ girlie_mac Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius!

Slide 20

Slide 20 text

@ girlie_mac Alexa UX (“Voice Chrome” Examples) https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html Listening Speaking (GIF animation)

Slide 21

Slide 21 text

Conversational Interface is:

Slide 22

Slide 22 text

● Intuitive ● Accessible ● Productive

Slide 23

Slide 23 text

@ girlie_mac Messaging Platforms ● Slack ● Facebook Messenger ● Telegram ● WeChat ● Kik ● Viver ● LINE etc.

Slide 24

Slide 24 text

Messaging + Bots for More Interactive Communications

Slide 25

Slide 25 text

Slack bots at

Slide 26

Slide 26 text

Slack bots at

Slide 27

Slide 27 text

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Slide 28

Slide 28 text

@ girlie_mac Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai ● Chatbots.io ● Converse AI ● Landbot.io ● Recast.ai ● ManyChat ● Fluent.ai (Voice UI) etc.

Slide 29

Slide 29 text

@ girlie_mac

Slide 30

Slide 30 text

AIaaS Motion.ai DialogFlow (API.ai) Artificial Intelligence as a Service

Slide 31

Slide 31 text

@ girlie_mac ManyChat

Slide 32

Slide 32 text

@ girlie_mac Hi, I want to book a flight! Yes, from SFO. Where are you flying to? London Hi Linda! Are you flying from San Francisco, as usual? An airline customer service bot Linda may not know she is talking to a bot!

Slide 33

Slide 33 text

@ girlie_mac Using DialogFlow

Slide 34

Slide 34 text

@ girlie_mac More Powerful NLP Platforms & APIs Natural Language Processing & Cognitive platforms: ● IBM Watson ● Google Cloud Natural Language API ● Microsoft LUIS ● Amazon Lex ● Baidu UNIT

Slide 35

Slide 35 text

Conversational Interface With JavaScript

Slide 36

Slide 36 text

@ girlie_mac { REST } JS SDK The APIs are mostly accessible with JS

Slide 37

Slide 37 text

@ girlie_mac Example: DialogFlow (API.ai) + FB Messenger const app = require('apiai')(CLIENT_ACCESS_TOKEN); function sendMessage(event) { let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on('response', (response) => { // Got a response from api.ai let aiText = response.result.fulfillment.speech; // Then POST to https://graph.facebook.com/v2.6/me/messages ... }); ai.end(); } API.ai Node.js SDK

Slide 38

Slide 38 text

@ girlie_mac Example: DialogFlow (API.ai) + FB Messenger

Slide 39

Slide 39 text

@ girlie_mac Example: Slack Message Sentiment Analysis with IBM Watson (GIF animation)

Slide 40

Slide 40 text

@ girlie_mac Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion

Slide 41

Slide 41 text

@ girlie_mac Example: IBM Watson + Slack app.post('/events', (req, res) => { let q = req.body; if (q.type === 'event_callback') { if(!q.event.text) return; analyzeTone(q.event); } }); Use Slack Events API to grab the text when a user post a message Pass the text data to Watson to analyze HTTP POST using ExpressJS

Slide 42

Slide 42 text

@ girlie_mac Example: IBM Watson + Slack const watson = require('watson-developer-cloud'); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === 'emotion_tone'){ tonecategory.tones.forEach((emotion) => { if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); }); Returns emotions score in 0 to 1 Just initializing it w/ your API credentials Post the result on Slack

Slide 43

Slide 43 text

@ girlie_mac Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === 'anger') { setLED(red); } else if(emotion.tone_id === 'joy') { setLED(yellow); } else if(emotion.tone_id === 'fear') { setLED(purple); } else if(emotion.tone_id === 'disgust') { setLED(green); } else if(emotion.tone_id === 'sadness') { setLED(blue); } } Change the LED color to match the emotion

Slide 44

Slide 44 text

@ girlie_mac

Slide 45

Slide 45 text

@ girlie_mac Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human!

Slide 46

Slide 46 text

Conversational Interface with Voice in Browser?

Slide 47

Slide 47 text

@ girlie_mac Project: Artificial Voice Chat Hello! 1 2 Howdy 2. Generate Artificial reply Using the 3rd party NLP API 1. User talk to browser Voice command: Voice -> Text 3 3. Browser speaks back Text -> Synthetic Voice Text

Slide 48

Slide 48 text

@ girlie_mac Web Speech API Speech Recognition & Speech Synthesis http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis 1 3

Slide 49

Slide 49 text

@ girlie_mac Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; const recognition = new SpeechRecognition(); Get an instance of the SpeechRecognition, the controller interface In the current Chromium, it is still prefixed

Slide 50

Slide 50 text

@ girlie_mac Web Speech API: Speech Recognition (Cont’d) recognition.lang = 'en-US'; recognition.interimResults = false; recognition.start(); recognition.addEventListener('result', (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); Some properties Methods: start(), stop(), abort() Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Slide 51

Slide 51 text

@ girlie_mac Web Speech API: Speech Recognition - Mic Access Permission

Slide 52

Slide 52 text

@ girlie_mac Web Speech API: Speech Recognition Hello! VOICE from a user TEXT “hello” Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv

Slide 53

Slide 53 text

@ girlie_mac Web Speech API: Speech Synthesis const synth = window.speechSynthesis; const utterance = new SpeechSynthesisUtterance(); utterance.text = 'I am a robot'; utterance.pitch = 1.5; utterance.lang = 'ja-JP'; synth.speak(utterance); No vendor prefix Properties of the SpeechSynthesisUtterance interface

Slide 54

Slide 54 text

@ girlie_mac Web Speech API: Speech Synthesis (voice by Alex or whoever it is) TEXT to VOICE “howdy” Howdy Speech Recognition demo on CodePen https://codepen.io/girliemac/pen/dmpxgv

Slide 55

Slide 55 text

@ girlie_mac Project: Artificial Voice Chat

Slide 56

Slide 56 text

Demo time! https://webspeech.herokuapp.com/

Slide 57

Slide 57 text

@ girlie_mac Article: smashingmagazine.com/ 2017/08/ai-chatbot-webspeech-api-node-js/ github.com/girliemac/ web-speech-ai

Slide 58

Slide 58 text

Conversational Interface is for human.

Slide 59

Slide 59 text

A bots interface really is a human interface.

Slide 60

Slide 60 text

Remember me? Clippy 1997 - 2007

Slide 61

Slide 61 text

@ girlie_mac Thank you... Ohhhh, wait!. There’s one more slide...

Slide 62

Slide 62 text

@ girlie_mac AI or cats, which will take over us first?

Slide 63

Slide 63 text

@ girlie_mac Thank you, really! @girlie_mac girliemac.com github.com/girliemac speakerdeck.com/girlie_mac BY-SA

Slide 64

Slide 64 text

@ girlie_mac Attribution: Open Emoji by Emoji-One (CC-BY 4.0)