Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let's Get Chatty with Conversational Interface in JavaScript

Let's Get Chatty with Conversational Interface in JavaScript

My talk at RenderConf in Oxford, UK

Tomomi Imura

March 23, 2018
Tweet

More Decks by Tomomi Imura

Other Decks in Technology

Transcript

  1. Let's Get Chatty with Conversational
    Interface in JavaScript
    BOTS, AI, AND JAVASCRIPT
    Tomomi Imura (@girlie_mac)
    #RenderConf

    View full-size slide

  2. @ girlie_mac
    ● an open web & tech advocate
    ● code JavaScript / Node.js
    ● work at Slack
    ● an advisor at Code Chrysalis
    ● a cat lady of the InterWeb
    tomomi imura
    @girlie_mac

    View full-size slide

  3. "Bots are like new
    applications that
    you can converse
    with. "
    -- Satya Nadella, Microsoft

    View full-size slide

  4. @ girlie_mac
    “We will evolve
    in computing
    from a mobile
    first to an AI first
    world.”
    -- Sundar Pichai, Google

    View full-size slide

  5. @ girlie_mac CC BY-SA: nextdayblinds.com

    View full-size slide

  6. @ girlie_mac
    Traditional Web & App Interactions

    View full-size slide

  7. @ girlie_mac
    Modern Web & Apps with Social Interactions

    View full-size slide

  8. @ girlie_mac
    Conversational User Interactions: Siri and Alexa (Voice Assistants)
    Alexa, how is
    the weather?
    Hey Siri
    In various form-factors

    View full-size slide

  9. @ girlie_mac
    Conversational User Interactions for Kids - with Voice
    CogniToys Dino - https://cognitoys.com

    View full-size slide

  10. @ girlie_mac
    Conversational User Interactions in a robot shape - with Voice

    View full-size slide

  11. @ girlie_mac
    Conversational User Interactions: Google Assistant (Voice & Text)
    (GIF animation)

    View full-size slide

  12. @ girlie_mac
    Conversational User Interactions: Slack Bots (Text with GUI)
    !

    View full-size slide

  13. Graphic
    Interface to
    Conversational
    Interface

    View full-size slide

  14. @ girlie_mac
    Deliver me a large margherita pizza!
    What kind of crust do you
    want?
    1. Regular crust
    2. Thin crust
    3. Gluten free crust
    Thin crust
    cha ching!

    View full-size slide

  15. Conversational
    Interface achieves:
    Natural user
    interactions with a
    minimal visual
    interface.

    View full-size slide

  16. No UI Clutter.
    Less Time Spent.

    View full-size slide

  17. @ girlie_mac
    Where is the address
    of your home?
    It’s 325 Ruby Street.
    Request a ride
    Come over to my place!

    View full-size slide

  18. @ girlie_mac
    Yes, get me a ride now
    Your driver, Sam will
    pick you up in 6 minutes.
    Look for the red Toyota
    Prius!

    View full-size slide

  19. @ girlie_mac
    Alexa UX (“Voice Chrome” Examples)
    https://developer.amazon.com/docs/alexa-voice-service/ux-design-overview.html
    Listening
    Speaking
    (GIF animation)

    View full-size slide

  20. Conversational
    Interface is:

    View full-size slide

  21. ● Intuitive
    ● Accessible
    ● Productive

    View full-size slide

  22. @ girlie_mac
    Messaging Platforms
    ● Slack
    ● Facebook Messenger
    ● Telegram
    ● WeChat
    ● Kik
    ● Viver
    ● LINE etc.

    View full-size slide

  23. Messaging + Bots for More Interactive Communications

    View full-size slide

  24. Slack bots at

    View full-size slide

  25. Slack bots at

    View full-size slide

  26. TacoBot by Taco Bell
    https://www.tacobell.com/feed/tacobot

    View full-size slide

  27. @ girlie_mac
    Natural Conversation APIs
    ● DialogFlow (API.ai
    /Google)
    ● Wit.ai (Facebook)
    ● Microsoft Bot
    Framework
    ● Motion.ai
    ● Chatbots.io
    ● Converse AI
    ● Landbot.io
    ● Recast.ai
    ● ManyChat
    ● Fluent.ai (Voice UI)
    etc.

    View full-size slide

  28. @ girlie_mac

    View full-size slide

  29. AIaaS
    Motion.ai
    DialogFlow (API.ai)
    Artificial
    Intelligence as
    a Service

    View full-size slide

  30. @ girlie_mac
    ManyChat

    View full-size slide

  31. @ girlie_mac
    Hi,
    I want to book a flight!
    Yes, from SFO.
    Where are you flying to?
    London
    Hi Linda! Are you flying from
    San Francisco, as usual?
    An airline customer
    service bot
    Linda may not
    know she is talking
    to a bot!

    View full-size slide

  32. @ girlie_mac
    Using
    DialogFlow

    View full-size slide

  33. @ girlie_mac
    More Powerful NLP Platforms & APIs
    Natural Language Processing & Cognitive platforms:
    ● IBM Watson
    ● Google Cloud Natural Language API
    ● Microsoft LUIS
    ● Amazon Lex
    ● Baidu UNIT

    View full-size slide

  34. Conversational
    Interface
    With JavaScript

    View full-size slide

  35. @ girlie_mac
    { REST }
    JS
    SDK
    The APIs are
    mostly
    accessible with
    JS

    View full-size slide

  36. @ girlie_mac
    Example: DialogFlow (API.ai) + FB Messenger
    const app = require('apiai')(CLIENT_ACCESS_TOKEN);
    function sendMessage(event) {
    let sender = event.sender.id;
    let text = event.message.text;
    let ai = app.textRequest(text, {
    sessionId: SESSION_STRING });
    ai.on('response', (response) => { // Got a response from api.ai
    let aiText = response.result.fulfillment.speech;
    // Then POST to https://graph.facebook.com/v2.6/me/messages
    ...
    });
    ai.end();
    }
    API.ai Node.js SDK

    View full-size slide

  37. @ girlie_mac
    Example: DialogFlow (API.ai) + FB Messenger

    View full-size slide

  38. @ girlie_mac
    Example: Slack Message Sentiment Analysis with IBM Watson
    (GIF animation)

    View full-size slide

  39. @ girlie_mac
    Example: IBM Watson + Slack
    The bot workflow:
    1. Read each message on a Slack channel
    2. Send the message to IBM Watson for
    examination
    3. If the likelihood of an emotion is above the given
    confidence threshold post the most prominent
    emotion

    View full-size slide

  40. @ girlie_mac
    Example: IBM Watson + Slack
    app.post('/events', (req, res) => {
    let q = req.body;
    if (q.type === 'event_callback') {
    if(!q.event.text) return;
    analyzeTone(q.event);
    }
    });
    Use Slack Events API to grab the
    text when a user post a message
    Pass the text data to Watson to analyze
    HTTP POST using ExpressJS

    View full-size slide

  41. @ girlie_mac
    Example: IBM Watson + Slack
    const watson = require('watson-developer-cloud');
    let tone_analyzer = watson.tone_analyzer({
    username: process.env.WATSON_USERNAME,
    password: process.env.WATSON_PASSWORD,
    });
    const confidencethreshold = 0.55;
    tone_analyzer.tone({text: text}, (err, tone) => {
    tone.document_tone.tone_categories.forEach((tonecategory) => {
    if(tonecategory.category_id === 'emotion_tone'){
    tonecategory.tones.forEach((emotion) => {
    if(emotion.score >= confidencethreshold) {
    postEmotionOnSlackChannel(emotion);
    }});
    }});
    });
    Returns
    emotions score in
    0 to 1
    Just initializing it w/
    your API credentials
    Post the result on Slack

    View full-size slide

  42. @ girlie_mac
    Example: IBM Watson + Slack + Raspberry Pi (for fun)
    function colorEmotion(emotion) {
    if (emotion.tone_id === 'anger') {
    setLED(red);
    } else if(emotion.tone_id === 'joy') {
    setLED(yellow);
    } else if(emotion.tone_id === 'fear') {
    setLED(purple);
    } else if(emotion.tone_id === 'disgust') {
    setLED(green);
    } else if(emotion.tone_id === 'sadness') {
    setLED(blue);
    }
    }
    Change the LED color to
    match the emotion

    View full-size slide

  43. @ girlie_mac

    View full-size slide

  44. @ girlie_mac
    Ack, this sucks! I want my
    money back!
    An angry customer detected. Connect
    the customer with a human!

    View full-size slide

  45. Conversational
    Interface with Voice
    in Browser?

    View full-size slide

  46. @ girlie_mac
    Project: Artificial Voice Chat
    Hello!
    1
    2
    Howdy
    2. Generate Artificial reply
    Using the 3rd party NLP API
    1. User talk to browser
    Voice command:
    Voice -> Text
    3 3. Browser speaks back
    Text -> Synthetic Voice
    Text

    View full-size slide

  47. @ girlie_mac
    Web Speech API
    Speech Recognition & Speech Synthesis
    http://caniuse.com/#feat=speech-recognition
    http://caniuse.com/#feat=speech-synthesis
    1 3

    View full-size slide

  48. @ girlie_mac
    Web Speech API: Speech
    Recognition
    const SpeechRecognition =
    window.SpeechRecognition || window.webkitSpeechRecognition;
    const recognition = new SpeechRecognition();
    Get an instance of the
    SpeechRecognition, the controller
    interface
    In the current Chromium, it is
    still prefixed

    View full-size slide

  49. @ girlie_mac
    Web Speech API: Speech Recognition (Cont’d)
    recognition.lang = 'en-US';
    recognition.interimResults = false;
    recognition.start();
    recognition.addEventListener('result', (e) => {
    let last = e.results.length - 1;
    let text = e.results[last][0].transcript;
    });
    Some properties
    Methods: start(), stop(),
    abort()
    Events: onresult,
    onerror,
    onaudiostarted,
    onaudioend, etc.

    View full-size slide

  50. @ girlie_mac
    Web Speech API: Speech Recognition - Mic Access Permission

    View full-size slide

  51. @ girlie_mac
    Web Speech API: Speech Recognition
    Hello!
    VOICE
    from a user
    TEXT
    “hello”
    Speech Recognition demo on CodePen
    https://codepen.io/girliemac/pen/dmpxgv

    View full-size slide

  52. @ girlie_mac
    Web Speech API: Speech Synthesis
    const synth = window.speechSynthesis;
    const utterance = new SpeechSynthesisUtterance();
    utterance.text = 'I am a robot';
    utterance.pitch = 1.5;
    utterance.lang = 'ja-JP';
    synth.speak(utterance);
    No vendor prefix
    Properties of the
    SpeechSynthesisUtterance interface

    View full-size slide

  53. @ girlie_mac
    Web Speech API: Speech Synthesis
    (voice by Alex or
    whoever it is)
    TEXT to VOICE
    “howdy”
    Howdy
    Speech Recognition demo on CodePen
    https://codepen.io/girliemac/pen/dmpxgv

    View full-size slide

  54. @ girlie_mac
    Project: Artificial Voice Chat

    View full-size slide

  55. Demo time!
    https://webspeech.herokuapp.com/

    View full-size slide

  56. @ girlie_mac
    Article:
    smashingmagazine.com/
    2017/08/ai-chatbot-web-
    speech-api-node-js/
    github.com/girliemac/
    web-speech-ai

    View full-size slide

  57. Conversational
    Interface is
    for human.

    View full-size slide

  58. A bots interface really is
    a human interface.

    View full-size slide

  59. Remember me?
    Clippy 1997 - 2007

    View full-size slide

  60. @ girlie_mac
    Thank you...
    Ohhhh, wait!. There’s one more slide...

    View full-size slide

  61. @ girlie_mac
    AI or cats,
    which will take
    over us first?

    View full-size slide

  62. @ girlie_mac
    Thank you,
    really!
    @girlie_mac
    girliemac.com
    github.com/girliemac
    speakerdeck.com/girlie_mac
    BY-SA

    View full-size slide

  63. @ girlie_mac
    Attribution:
    Open Emoji by Emoji-One (CC-BY 4.0)

    View full-size slide