Future of the Web with Conversational Interface (in JavaScript!)

Slide 1

Slide 1 text

Future of the Web with Conversational Interface Tomomi Imura (@girlie_mac)

Slide 2

Slide 2 text

@ ● advocate open web & technology ● code JavaScript & Node.js ● write & speak about tech ● hack useless stuff ● A cat lady of the InterWeb ● dev relations at Slack tomomi imura

Slide 3

Slide 3 text

Talking to non-human

Slide 4

Slide 4 text

@ Photo by Jeopardy Productions, Inc.

Slide 5

Slide 5 text

-- Satya Nadella, Microsoft

Slide 6

Slide 6 text

@ -- Sundar Pichai, Google

Slide 7

Slide 7 text

@ CC BY-SA: nextdayblinds.com

Slide 8

Slide 8 text

@ Traditional Web & App Interactions

Slide 9

Slide 9 text

@ Modern Web & Apps with Social Interactions

Slide 10

Slide 10 text

@ Conversational User Interactions: Siri and Alexa (Voice Assistants) Alexa, how is the weather? Hey Siri In various form-factors

Slide 11

Slide 11 text

@ Conversational User Interactions for Kids - with Voice CogniToys Dino - https://cognitoys.com

Slide 12

Slide 12 text

@ Conversational User Interactions in a robot shape - with Voice

Slide 13

Slide 13 text

Slide 14

Slide 14 text

@ Conversational User Interactions: Google Assistant (Voice & Text)

Slide 15

Slide 15 text

@ Conversational User Interactions: Slack Bots (Text) !

Slide 16

Slide 16 text

Slack Integrations & Bots for Better Productivity + + +

Slide 17

Slide 17 text

Graphic Interface to Conversational Interface

Slide 18

Slide 18 text

@ Deliver me a large margherita pizza! What kind of crust do you want? 1. Regular crust 2. Thin crust 3. Gluten free crust Thin crust cha ching!

Slide 19

Slide 19 text

Conversational Interface achieves: Natural user interactions with a minimal visual interface.

Slide 20

Slide 20 text

No UI Clutter. Less Time Spent.

Slide 21

Slide 21 text

@ Where is the address of your home? It’s 325 Ruby Street. Request a ride Come over to my place!

Slide 22

Slide 22 text

@ Yes, get me a ride now Your driver, Sam will pick you up in 6 minutes. Look for the red Toyota Prius!

Slide 23

Slide 23 text

@ Alexa UX (“Voice Chrome” Examples) https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html Listening Speaking

Slide 24

Slide 24 text

Old concepts. Technology caught up with the ideas.

Slide 25

Slide 25 text

@ HAL9000: “I'm sorry Dave, I'm afraid I can't do that”

Slide 26

Slide 26 text

Slide 27

Slide 27 text

@ Nuh, we’re still in control over AI. (I hope)

Slide 28

Slide 28 text

Conversational Interface is:

Slide 29

Slide 29 text

● Intuitive ● Accessible ● Productive

Slide 30

Slide 30 text

@ Messaging Platforms ● Slack ● Facebook Messenger ● Telegram ● WeChat ● Kik ● Viver ● LINE etc.

Slide 31

Slide 31 text

Messaging + Bots for More Interactive Communications

Slide 32

Slide 32 text

Slack bots at

Slide 33

Slide 33 text

Slack bots at

Slide 34

Slide 34 text

TacoBot by Taco Bell https://www.tacobell.com/feed/tacobot

Slide 35

Slide 35 text

@ Natural Conversation APIs ● DialogFlow (API.ai /Google) ● Wit.ai (Facebook) ● Microsoft Bot Framework ● Motion.ai ● Chatbots.io ● Converse AI ● Recast.ai ● Flow XO ● ManyChat ● Fluent.ai (Voice UI) etc.

Slide 36

Slide 36 text

AIaaS Motion.ai DialogFlow (API.ai) Artificial Intelligence as a Service

Slide 37

Slide 37 text

@ Hi, I want to book a flight! Yes, from SFO. Where are you flying to? London Hi Linda! Are you flying from San Francisco, as usual? An airline customer service bot Linda may not know she is talking to a bot!

Slide 38

Slide 38 text

Slide 39

Slide 39 text

@ NLP Platforms / APIs Natural Language Processing & Cognitive platforms: ● IBM Watson ● Google Cloud Natural Language API ● Microsoft LUIS ● Amazon Lex ● Baidu UNIT

Slide 40

Slide 40 text

Build Your Own Conversational Interface

Slide 41

Slide 41 text

...with JavaScript

Slide 42

Slide 42 text

@ { REST } The APIs are mostly accessible with JS

Slide 43

Slide 43 text

@ Example: API.ai + FB Messenger const app = require('apiai')(CLIENT_ACCESS_TOKEN); function sendMessage(event) { let sender = event.sender.id; let text = event.message.text; let ai = app.textRequest(text, { sessionId: SESSION_STRING }); ai.on('response', (response) => { // Got a response. Let's POST to Facebook Messenger }); ai.end(); } API.ai Node.js SDK

Slide 44

Slide 44 text

@ Example: IBM Watson + Slack

Slide 45

Slide 45 text

Slide 46

Slide 46 text

@ Example: IBM Watson + Slack The bot workflow: 1. Read each message on a Slack channel 2. Send the message to IBM Watson for examination 3. If the likelihood of an emotion is above the given confidence threshold post the most prominent emotion

Slide 47

Slide 47 text

@ Example: IBM Watson + Slack app.post('/events', (req, res) => { let q = req.body; if (q.type === 'event_callback') { if(!q.event.text) return; analyzeTone(q.event); } }); Use Slack Events API to grab the text when a user post a message Pass the text data to Watson to analyze HTTP POST using ExpressJS

Slide 48

Slide 48 text

@ Example: IBM Watson + Slack const watson = require('watson-developer-cloud'); let tone_analyzer = watson.tone_analyzer({ username: process.env.WATSON_USERNAME, password: process.env.WATSON_PASSWORD, }); const confidencethreshold = 0.55; tone_analyzer.tone({text: text}, (err, tone) => { tone.document_tone.tone_categories.forEach((tonecategory) => { if(tonecategory.category_id === 'emotion_tone'){ tonecategory.tones.forEach((emotion) => { if(emotion.score >= confidencethreshold) { postEmotionOnSlackChannel(emotion); }}); }}); }); Returns emotions score in 0 to 1 Just initializing it w/ your API credentials Post the result on Slack

Slide 49

Slide 49 text

@ Example: IBM Watson + Slack + Raspberry Pi (for fun) function colorEmotion(emotion) { if (emotion.tone_id === 'anger') { setLED(red); } else if(emotion.tone_id === 'joy') { setLED(yellow); } else if(emotion.tone_id === 'fear') { setLED(purple); } else if(emotion.tone_id === 'disgust') { setLED(green); } else if(emotion.tone_id === 'sadness') { setLED(blue); } } Change the LED color to match the emotion

Slide 50

Slide 50 text

@ Ack, this sucks! I want my money back! An angry customer detected. Connect the customer with a human!

Slide 51

Slide 51 text

Conversational Interface with Voice in Browser?

Slide 52

Slide 52 text

@ Project: Artificial Voice Chat Hello! 1 2 Howdy 2. Generate Artificial reply Using the 3rd party API 1. User talk to browser Voice command: Voice -> Text 3 3. Browser speaks back Text -> Synthetic Voice Text

Slide 53

Slide 53 text

@ Web Speech API & http://caniuse.com/#feat=speech-recognition http://caniuse.com/#feat=speech-synthesis 1 3

Slide 54

Slide 54 text

@ Web Speech API: Speech Recognition const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; const recognition = new SpeechRecognition(); Get an instance of the SpeechRecognition, the controller interface In the current Chromium, it is still prefixed

Slide 55

Slide 55 text

@ Web Speech API: Speech Recognition (Cont’d) recognition.lang = 'en-US'; recognition.interimResults = false; recognition.start(); recognition.addEventListener('result', (e) => { let last = e.results.length - 1; let text = e.results[last][0].transcript; }); Some properties Methods: start(), stop(), abort() Events: onresult, onerror, onaudiostarted, onaudioend, etc.

Slide 56

Slide 56 text

@ Web Speech API: Speech Recognition Hello! VOICE from a user TEXT “hello”

Slide 57

Slide 57 text

@ Web Speech API: Speech Synthesis const synth = window.speechSynthesis; const utterance = new SpeechSynthesisUtterance(); utterance.text = 'I am a robot'; utterance.pitch = 1.5; utterance.voice = 'Alex'; synth.speak(utterance); No vendor prefix Properties of the SpeechSynthesisUtterance interface Get available voices with synth.getVoices() method

Slide 58

Slide 58 text

@ Web Speech API: Speech Synthesis (voice by Alex or whoever browser is) TEXT to VOICE “howdy” Howdy

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Demo on Chrome https://webspeech.herokuapp.com/

Slide 61

Slide 61 text

@ https://www.smashingmagazine.com/2017/08/ai-chatbot-web-speech-api-node-js/

Slide 62

Slide 62 text

Conversational Interface is for human.

Slide 63

Slide 63 text

A bots interface really is a human interface.

Slide 64

Slide 64 text

Clippy 1997 - 2007

Slide 65

Slide 65 text

@ @girlie_mac girliemac.com github.com/girliemac speakerdeck.com/girlie_mac

Slide 66

Slide 66 text

@ Attribution: Open Emoji by Emoji-One (CC-BY 4.0)