Actions with Google

Learn how to create custom actions for the Google Assistant

1. Actions with Google Bryan Herbst Android Engineer @ Target
2. The Google Assistant “Ok Google” Allo Google Home Android Auto Android Wear
3. The Google Assistant Google Home
4. Design principles Voice interaction is different
5. What actions can I take?
6. What actions can I take?
7. What should I have for dinner? What are you in the mood for? You can say Italian, American, Mexican, or Chinese. !
8. What should I have for dinner? What are you in the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
9. “Guide me downtown.”
10. “Guide me downtown.” Downtown where? New York? Minneapolis?
11. “Guide me downtown.” Driving or walking? Starting from where?
12. Speech uses shortcuts Filling in the blanks requires context.
13. Go scriptless You are (not) a robot.
14. Freedom of speech Say “help” for more options
15. Freedom of speech Say “help” for more options
16. Anticipate errors Shortcuts can cause ambiguity
17. ”Buy me a KitKat bar.”
18. ”Buy me a KitKat bar.”
19. ”Buy me a KitKat bar.”
20. ”Buy me a KitKat bar.”
21. ”Buy me a KitKat bar.” ”Okay, ordering a king sized KitKat bar from Amazon. Is that correct?”
22. ”Buy me a KitKat bar.” ”Okay, ordering a king sized KitKat bar from Amazon. Is that correct?” “Yes.”
23. ”Buy me a KitKat bar.” ”Okay, ordering a king sized KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
24. Anticipate errors Shortcuts can cause ambiguity
25. Be Flexible Shortcuts can cause ambiguity
26. Types of actions Conversation and direct
27. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
28. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You handle interaction and fulfillment Availability
29. Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You handle interaction and fulfillment Availability Now
30. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this action” Responsibilities You handle interaction and fulfillment Availability Now
31. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
32. Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
33. Conversation Actions " Less ambiguity # Limited trigger
34. Making Conversation Actions
35. Conversation Actions Format $ Your Action Input
36. Conversation Actions Format $ Your Action Input JSON Request
37. Conversation Actions Format $ Your Action Input JSON Request JSON Response
38. Conversation Actions Format $ Your Action Input Action Response JSON Request JSON Response
39. Conversation Actions Format Your Action JSON Request JSON Response Conversation API
40. Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
41. Invocation Dialogs Fulfillment “Ok, where would you like to go?” “Paris.”
42. Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
43. Invocation - name Unique
44. Invocation - name Unique | Easy to say
45. Name restrictions Two words (with exceptions)
46. Name restrictions Two words (with exceptions) | Avoid reserved words
47. Trigger Phrases • Let me talk to… • I want to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
48. Deep links “Talk to service about topic”
49. Not deep links “Talk to service.”
50. Conversations You need to end them
51. API.AI Actions SDK
52. API.AI Actions SDK GUI
53. API.AI Actions SDK GUI Natural Language Understanding
54. API.AI Actions SDK GUI Natural Language Understanding Integrations
55. API.AI Actions SDK It’s all on you GUI Natural Language Understanding Integrations
57. $ API.AI Agent Entities Actions
58. Intent examples • Set an alarm for 6:00am • Wake me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
59. Intent examples • Set an alarm for 6:00am • Wake me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
60. Intent examples • Set an alarm for 6:00am • Wake me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
61. Intent examples • Set an alarm for 6:00am • Wake me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
62. Welcome Intent Talk to Clocky
63. Fallback Intent Tell Clocky to order me a pizza
64. Fallback Intent Tell Clocky to order me a pizza I don’t know how to help with that.
65. Action Capture parameters
66. Action Capture parameters | Prompt for parameters
67. System Entitities • Number • Name • Location • Date/time • Airport • And so much more!
68. Custom Entities • Food menu • Team names • Fantasy names
69. Response These are logic-less
70. Demo time
71. Advanced Fulfillment For everything else, there’s webhooks
72. $ (Your action) Input Action Response JSON Request JSON Response
73. $ Your server
74. $ Your server API.AI Webhook
75. $ Your server API.AI Webhook (also JSON)
76. Hosting Host anywhere
77. Hosting Host anywhere | Node.js recommended
78. Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app = express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
79. Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap); function setAlarm() { assistant.tell("Hello from the webhook!"); }
80. Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap); function setAlarm() { assistant.tell("Hello from the webhook!"); }
81. Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I will set an alarm for" + time); }
82. SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
83. User Identity • ID (resettable!) • Display name • Given name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
84. User Identity • ID (resettable!) • Display name • Given name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
85. Context Context allows for natural conversations
86. What’s the weather like in Minneapolis? It is currently 10° in Minneapolis. !
87. What’s the weather like in Minneapolis? It is currently 10° in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
88. What’s the weather like in Minneapolis? It is currently 10° in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
89. Interaction 1 Interaction 2 Context Output Input
90. Output Context Contains all parameters from the action
91. Input Context Intents only trigger if context exists
92. Input Context Intents only trigger if context exists Allows for branching flows
93. Context - Lifetime Duration (in requests)
94. Context - Lifetime Duration (in requests) | Ends with conversation
95. Demo time
96. Publishing
97. Actions Directory
98. Create a Google Cloud Project Register the Action Deploy via API.AI Wait for review
99. Create a Google Cloud Project Register the Action Deploy via API.AI Wait for review
100. Create a Google Cloud Project Register the Action Deploy via API.AI Wait for review
101. Create a Google Cloud Project Register the Action Deploy via API.AI Wait for review