Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Actions with Google

Bryan Herbst
February 04, 2017

Actions with Google

Learn how to create custom actions for the Google Assistant

Bryan Herbst

February 04, 2017
Tweet

More Decks by Bryan Herbst

Other Decks in Programming

Transcript

  1. Actions with Google
    Bryan Herbst
    Android Engineer @ Target

    View Slide

  2. The Google
    Assistant
    “Ok Google” Allo
    Google Home Android Auto
    Android Wear

    View Slide

  3. The Google Assistant
    Google Home

    View Slide

  4. Design principles
    Voice interaction is different

    View Slide

  5. What actions
    can I take?

    View Slide

  6. What actions
    can I take?

    View Slide

  7. What should I have
    for dinner?
    What are you in the mood
    for? You can say Italian,
    American, Mexican, or
    Chinese.
    !

    View Slide

  8. What should I have
    for dinner?
    What are you in the mood
    for? You can say Italian,
    American, Mexican, or
    Chinese.
    ! Take turns.

    View Slide

  9. “Guide me downtown.”

    View Slide

  10. “Guide me downtown.”
    Downtown where?
    New York?
    Minneapolis?

    View Slide

  11. “Guide me downtown.”
    Driving or walking?
    Starting from where?

    View Slide

  12. Speech uses shortcuts
    Filling in the blanks requires context.

    View Slide

  13. Go scriptless
    You are (not) a robot.

    View Slide

  14. Freedom of speech
    Say “help” for more options

    View Slide

  15. Freedom of speech
    Say “help” for more options

    View Slide

  16. Anticipate errors
    Shortcuts can cause ambiguity

    View Slide

  17. ”Buy me a KitKat bar.”

    View Slide

  18. ”Buy me a KitKat bar.”

    View Slide

  19. ”Buy me a KitKat bar.”

    View Slide

  20. ”Buy me a KitKat bar.”

    View Slide

  21. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”

    View Slide

  22. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “Yes.”

    View Slide

  23. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “No, make it a regular sized KitKat bar.”

    View Slide

  24. Anticipate errors
    Shortcuts can cause ambiguity

    View Slide

  25. Be Flexible
    Shortcuts can cause ambiguity

    View Slide

  26. Types of actions
    Conversation and direct

    View Slide

  27. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    Availability

    View Slide

  28. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability

    View Slide

  29. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now

    View Slide

  30. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now

    View Slide

  31. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now

    View Slide

  32. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now Soon

    View Slide

  33. Conversation Actions
    " Less ambiguity
    # Limited trigger

    View Slide

  34. Making Conversation
    Actions

    View Slide

  35. Conversation Actions Format
    $ Your Action
    Input

    View Slide

  36. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request

    View Slide

  37. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request
    JSON
    Response

    View Slide

  38. Conversation Actions Format
    $ Your Action
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response

    View Slide

  39. Conversation Actions Format
    Your Action
    JSON Request
    JSON Response
    Conversation
    API

    View Slide

  40. Invocation
    Dialogs
    Fulfillment
    “Ask AwesomeFlightApp
    to book me a flight.”

    View Slide

  41. Invocation
    Dialogs
    Fulfillment
    “Ok, where would you
    like to go?”
    “Paris.”

    View Slide

  42. Invocation
    Dialogs
    Fulfillment
    “Okay, booking a flight
    to Paris!”

    View Slide

  43. Invocation - name
    Unique

    View Slide

  44. Invocation - name
    Unique | Easy to say

    View Slide

  45. Name restrictions
    Two words (with exceptions)

    View Slide

  46. Name restrictions
    Two words (with exceptions) | Avoid reserved words

    View Slide

  47. Trigger Phrases
    • Let me talk to…
    • I want to talk to...
    • Can I talk to...
    • Talk to...
    • Ask...
    • At...
    • Tell...
    • Tell me...

    View Slide

  48. Deep links
    “Talk to service about topic”

    View Slide

  49. Not deep links
    “Talk to service.”

    View Slide

  50. Conversations
    You need to end them

    View Slide

  51. API.AI
    Actions SDK

    View Slide

  52. API.AI
    Actions SDK
    GUI

    View Slide

  53. API.AI
    Actions SDK
    GUI
    Natural Language Understanding

    View Slide

  54. API.AI
    Actions SDK
    GUI
    Natural Language Understanding
    Integrations

    View Slide

  55. API.AI
    Actions SDK
    It’s all on you
    GUI
    Natural Language Understanding
    Integrations

    View Slide

  56. View Slide

  57. $
    API.AI Agent
    Entities Actions

    View Slide

  58. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View Slide

  59. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View Slide

  60. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View Slide

  61. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View Slide

  62. Welcome
    Intent
    Talk to Clocky

    View Slide

  63. Fallback
    Intent
    Tell Clocky to order me a pizza

    View Slide

  64. Fallback
    Intent
    Tell Clocky to order me a pizza
    I don’t know how to help with that.

    View Slide

  65. Action
    Capture parameters

    View Slide

  66. Action
    Capture parameters | Prompt for parameters

    View Slide

  67. System
    Entitities
    • Number
    • Name
    • Location
    • Date/time
    • Airport
    • And so much more!

    View Slide

  68. Custom
    Entities
    • Food menu
    • Team names
    • Fantasy names

    View Slide

  69. Response
    These are logic-less

    View Slide

  70. Demo time

    View Slide

  71. Advanced Fulfillment
    For everything else, there’s webhooks

    View Slide

  72. $
    (Your action)
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response

    View Slide

  73. $ Your server

    View Slide

  74. $ Your server
    API.AI
    Webhook

    View Slide

  75. $ Your server
    API.AI
    Webhook
    (also JSON)

    View Slide

  76. Hosting
    Host anywhere

    View Slide

  77. Hosting
    Host anywhere | Node.js recommended

    View Slide

  78. Simple webhook server
    let Assistant = require('actions-on-google').ApiAiAssistant;
    let app = express();
    app.post('/', function (req, res) {
    const assistant = new Assistant({request: req, response: res}); }
    }

    View Slide

  79. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }

    View Slide

  80. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }

    View Slide

  81. Webhook with arguments
    function withArguments() {
    let time= assistant.getArgument(”time");
    assistant.tell(”I will set an alarm for" + time);
    }

    View Slide

  82. SSML- Speech Synthesis Markup Language
    function withSsml() {
    assistant.tell("This response uses”
    + "SSML.”
    + "Isn't it awesome?");
    }

    View Slide

  83. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)

    View Slide

  84. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)
    Requires permission

    View Slide

  85. Context
    Context allows for natural conversations

    View Slide

  86. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !

    View Slide

  87. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°

    View Slide

  88. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°

    View Slide

  89. Interaction 1 Interaction 2
    Context
    Output Input

    View Slide

  90. Output Context
    Contains all parameters from the action

    View Slide

  91. Input Context
    Intents only trigger if context exists

    View Slide

  92. Input Context
    Intents only trigger if context exists
    Allows for branching flows

    View Slide

  93. Context - Lifetime
    Duration (in requests)

    View Slide

  94. Context - Lifetime
    Duration (in requests) | Ends with conversation

    View Slide

  95. Demo time

    View Slide

  96. Publishing

    View Slide

  97. Actions
    Directory

    View Slide

  98. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View Slide

  99. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View Slide

  100. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View Slide

  101. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View Slide