Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Actions with Google

Bryan Herbst
February 04, 2017

Actions with Google

Learn how to create custom actions for the Google Assistant

Bryan Herbst

February 04, 2017
Tweet

More Decks by Bryan Herbst

Other Decks in Programming

Transcript

  1. Actions with Google
    Bryan Herbst
    Android Engineer @ Target

    View full-size slide

  2. The Google
    Assistant
    “Ok Google” Allo
    Google Home Android Auto
    Android Wear

    View full-size slide

  3. The Google Assistant
    Google Home

    View full-size slide

  4. Design principles
    Voice interaction is different

    View full-size slide

  5. What actions
    can I take?

    View full-size slide

  6. What actions
    can I take?

    View full-size slide

  7. What should I have
    for dinner?
    What are you in the mood
    for? You can say Italian,
    American, Mexican, or
    Chinese.
    !

    View full-size slide

  8. What should I have
    for dinner?
    What are you in the mood
    for? You can say Italian,
    American, Mexican, or
    Chinese.
    ! Take turns.

    View full-size slide

  9. “Guide me downtown.”

    View full-size slide

  10. “Guide me downtown.”
    Downtown where?
    New York?
    Minneapolis?

    View full-size slide

  11. “Guide me downtown.”
    Driving or walking?
    Starting from where?

    View full-size slide

  12. Speech uses shortcuts
    Filling in the blanks requires context.

    View full-size slide

  13. Go scriptless
    You are (not) a robot.

    View full-size slide

  14. Freedom of speech
    Say “help” for more options

    View full-size slide

  15. Freedom of speech
    Say “help” for more options

    View full-size slide

  16. Anticipate errors
    Shortcuts can cause ambiguity

    View full-size slide

  17. ”Buy me a KitKat bar.”

    View full-size slide

  18. ”Buy me a KitKat bar.”

    View full-size slide

  19. ”Buy me a KitKat bar.”

    View full-size slide

  20. ”Buy me a KitKat bar.”

    View full-size slide

  21. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”

    View full-size slide

  22. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “Yes.”

    View full-size slide

  23. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “No, make it a regular sized KitKat bar.”

    View full-size slide

  24. Anticipate errors
    Shortcuts can cause ambiguity

    View full-size slide

  25. Be Flexible
    Shortcuts can cause ambiguity

    View full-size slide

  26. Types of actions
    Conversation and direct

    View full-size slide

  27. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    Availability

    View full-size slide

  28. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability

    View full-size slide

  29. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now

    View full-size slide

  30. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now

    View full-size slide

  31. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now

    View full-size slide

  32. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now Soon

    View full-size slide

  33. Conversation Actions
    " Less ambiguity
    # Limited trigger

    View full-size slide

  34. Making Conversation
    Actions

    View full-size slide

  35. Conversation Actions Format
    $ Your Action
    Input

    View full-size slide

  36. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request

    View full-size slide

  37. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request
    JSON
    Response

    View full-size slide

  38. Conversation Actions Format
    $ Your Action
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response

    View full-size slide

  39. Conversation Actions Format
    Your Action
    JSON Request
    JSON Response
    Conversation
    API

    View full-size slide

  40. Invocation
    Dialogs
    Fulfillment
    “Ask AwesomeFlightApp
    to book me a flight.”

    View full-size slide

  41. Invocation
    Dialogs
    Fulfillment
    “Ok, where would you
    like to go?”
    “Paris.”

    View full-size slide

  42. Invocation
    Dialogs
    Fulfillment
    “Okay, booking a flight
    to Paris!”

    View full-size slide

  43. Invocation - name
    Unique

    View full-size slide

  44. Invocation - name
    Unique | Easy to say

    View full-size slide

  45. Name restrictions
    Two words (with exceptions)

    View full-size slide

  46. Name restrictions
    Two words (with exceptions) | Avoid reserved words

    View full-size slide

  47. Trigger Phrases
    • Let me talk to…
    • I want to talk to...
    • Can I talk to...
    • Talk to...
    • Ask...
    • At...
    • Tell...
    • Tell me...

    View full-size slide

  48. Deep links
    “Talk to service about topic”

    View full-size slide

  49. Not deep links
    “Talk to service.”

    View full-size slide

  50. Conversations
    You need to end them

    View full-size slide

  51. API.AI
    Actions SDK

    View full-size slide

  52. API.AI
    Actions SDK
    GUI

    View full-size slide

  53. API.AI
    Actions SDK
    GUI
    Natural Language Understanding

    View full-size slide

  54. API.AI
    Actions SDK
    GUI
    Natural Language Understanding
    Integrations

    View full-size slide

  55. API.AI
    Actions SDK
    It’s all on you
    GUI
    Natural Language Understanding
    Integrations

    View full-size slide

  56. $
    API.AI Agent
    Entities Actions

    View full-size slide

  57. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View full-size slide

  58. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View full-size slide

  59. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View full-size slide

  60. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm

    View full-size slide

  61. Welcome
    Intent
    Talk to Clocky

    View full-size slide

  62. Fallback
    Intent
    Tell Clocky to order me a pizza

    View full-size slide

  63. Fallback
    Intent
    Tell Clocky to order me a pizza
    I don’t know how to help with that.

    View full-size slide

  64. Action
    Capture parameters

    View full-size slide

  65. Action
    Capture parameters | Prompt for parameters

    View full-size slide

  66. System
    Entitities
    • Number
    • Name
    • Location
    • Date/time
    • Airport
    • And so much more!

    View full-size slide

  67. Custom
    Entities
    • Food menu
    • Team names
    • Fantasy names

    View full-size slide

  68. Response
    These are logic-less

    View full-size slide

  69. Advanced Fulfillment
    For everything else, there’s webhooks

    View full-size slide

  70. $
    (Your action)
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response

    View full-size slide

  71. $ Your server

    View full-size slide

  72. $ Your server
    API.AI
    Webhook

    View full-size slide

  73. $ Your server
    API.AI
    Webhook
    (also JSON)

    View full-size slide

  74. Hosting
    Host anywhere

    View full-size slide

  75. Hosting
    Host anywhere | Node.js recommended

    View full-size slide

  76. Simple webhook server
    let Assistant = require('actions-on-google').ApiAiAssistant;
    let app = express();
    app.post('/', function (req, res) {
    const assistant = new Assistant({request: req, response: res}); }
    }

    View full-size slide

  77. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }

    View full-size slide

  78. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }

    View full-size slide

  79. Webhook with arguments
    function withArguments() {
    let time= assistant.getArgument(”time");
    assistant.tell(”I will set an alarm for" + time);
    }

    View full-size slide

  80. SSML- Speech Synthesis Markup Language
    function withSsml() {
    assistant.tell("This response uses”
    + "SSML.”
    + "Isn't it awesome?");
    }

    View full-size slide

  81. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)

    View full-size slide

  82. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)
    Requires permission

    View full-size slide

  83. Context
    Context allows for natural conversations

    View full-size slide

  84. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !

    View full-size slide

  85. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°

    View full-size slide

  86. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°

    View full-size slide

  87. Interaction 1 Interaction 2
    Context
    Output Input

    View full-size slide

  88. Output Context
    Contains all parameters from the action

    View full-size slide

  89. Input Context
    Intents only trigger if context exists

    View full-size slide

  90. Input Context
    Intents only trigger if context exists
    Allows for branching flows

    View full-size slide

  91. Context - Lifetime
    Duration (in requests)

    View full-size slide

  92. Context - Lifetime
    Duration (in requests) | Ends with conversation

    View full-size slide

  93. Actions
    Directory

    View full-size slide

  94. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View full-size slide

  95. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View full-size slide

  96. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View full-size slide

  97. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review

    View full-size slide