Actions with Google

108056ccba92f98fdbbabad534537573?s=47 Bryan Herbst
February 04, 2017

Actions with Google

Learn how to create custom actions for the Google Assistant

108056ccba92f98fdbbabad534537573?s=128

Bryan Herbst

February 04, 2017
Tweet

Transcript

  1. Actions with Google Bryan Herbst Android Engineer @ Target

  2. The Google Assistant “Ok Google” Allo Google Home Android Auto

    Android Wear
  3. The Google Assistant Google Home

  4. Design principles Voice interaction is different

  5. What actions can I take?

  6. What actions can I take?

  7. What should I have for dinner? What are you in

    the mood for? You can say Italian, American, Mexican, or Chinese. !
  8. What should I have for dinner? What are you in

    the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
  9. “Guide me downtown.”

  10. “Guide me downtown.” Downtown where? New York? Minneapolis?

  11. “Guide me downtown.” Driving or walking? Starting from where?

  12. Speech uses shortcuts Filling in the blanks requires context.

  13. Go scriptless You are (not) a robot.

  14. Freedom of speech Say “help” for more options

  15. Freedom of speech Say “help” for more options

  16. Anticipate errors Shortcuts can cause ambiguity

  17. ”Buy me a KitKat bar.”

  18. ”Buy me a KitKat bar.”

  19. ”Buy me a KitKat bar.”

  20. ”Buy me a KitKat bar.”

  21. ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?”
  22. ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?” “Yes.”
  23. ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
  24. Anticipate errors Shortcuts can cause ambiguity

  25. Be Flexible Shortcuts can cause ambiguity

  26. Types of actions Conversation and direct

  27. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability

  28. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You

    handle interaction and fulfillment Availability
  29. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You

    handle interaction and fulfillment Availability Now
  30. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Availability Now
  31. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
  32. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
  33. Conversation Actions " Less ambiguity # Limited trigger

  34. Making Conversation Actions

  35. Conversation Actions Format $ Your Action Input

  36. Conversation Actions Format $ Your Action Input JSON Request

  37. Conversation Actions Format $ Your Action Input JSON Request JSON

    Response
  38. Conversation Actions Format $ Your Action Input Action Response JSON

    Request JSON Response
  39. Conversation Actions Format Your Action JSON Request JSON Response Conversation

    API
  40. Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”

  41. Invocation Dialogs Fulfillment “Ok, where would you like to go?”

    “Paris.”
  42. Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”

  43. Invocation - name Unique

  44. Invocation - name Unique | Easy to say

  45. Name restrictions Two words (with exceptions)

  46. Name restrictions Two words (with exceptions) | Avoid reserved words

  47. Trigger Phrases • Let me talk to… • I want

    to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
  48. Deep links “Talk to service about topic”

  49. Not deep links “Talk to service.”

  50. Conversations You need to end them

  51. API.AI Actions SDK

  52. API.AI Actions SDK GUI

  53. API.AI Actions SDK GUI Natural Language Understanding

  54. API.AI Actions SDK GUI Natural Language Understanding Integrations

  55. API.AI Actions SDK It’s all on you GUI Natural Language

    Understanding Integrations
  56. None
  57. $ API.AI Agent Entities Actions

  58. Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
  59. Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
  60. Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
  61. Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
  62. Welcome Intent Talk to Clocky

  63. Fallback Intent Tell Clocky to order me a pizza

  64. Fallback Intent Tell Clocky to order me a pizza I

    don’t know how to help with that.
  65. Action Capture parameters

  66. Action Capture parameters | Prompt for parameters

  67. System Entitities • Number • Name • Location • Date/time

    • Airport • And so much more!
  68. Custom Entities • Food menu • Team names • Fantasy

    names
  69. Response These are logic-less

  70. Demo time

  71. Advanced Fulfillment For everything else, there’s webhooks

  72. $ (Your action) Input Action Response JSON Request JSON Response

  73. $ Your server

  74. $ Your server API.AI Webhook

  75. $ Your server API.AI Webhook (also JSON)

  76. Hosting Host anywhere

  77. Hosting Host anywhere | Node.js recommended

  78. Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =

    express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
  79. Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);

    function setAlarm() { assistant.tell("Hello from the webhook!"); }
  80. Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);

    function setAlarm() { assistant.tell("Hello from the webhook!"); }
  81. Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I

    will set an alarm for" + time); }
  82. SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response

    uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
  83. User Identity • ID (resettable!) • Display name • Given

    name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
  84. User Identity • ID (resettable!) • Display name • Given

    name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
  85. Context Context allows for natural conversations

  86. What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. !
  87. What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
  88. What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
  89. Interaction 1 Interaction 2 Context Output Input

  90. Output Context Contains all parameters from the action

  91. Input Context Intents only trigger if context exists

  92. Input Context Intents only trigger if context exists Allows for

    branching flows
  93. Context - Lifetime Duration (in requests)

  94. Context - Lifetime Duration (in requests) | Ends with conversation

  95. Demo time

  96. Publishing

  97. Actions Directory

  98. Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review
  99. Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review
  100. Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review
  101. Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review