Actions with Google

Transcript

1. 1.

   Actions with Google Bryan Herbst Android Engineer @ Target

2. 2.

   The Google Assistant “Ok Google” Allo Google Home Android Auto

   Android Wear
3. 3.

   The Google Assistant Google Home

4. 4.

   Design principles Voice interaction is different

5. 5.

   What actions can I take?

6. 6.

   What actions can I take?

7. 7.

   What should I have for dinner? What are you in

   the mood for? You can say Italian, American, Mexican, or Chinese. !
8. 8.

   What should I have for dinner? What are you in

   the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
9. 9.

   “Guide me downtown.”

10. 10.

    “Guide me downtown.” Downtown where? New York? Minneapolis?

11. 11.

    “Guide me downtown.” Driving or walking? Starting from where?

12. 12.

    Speech uses shortcuts Filling in the blanks requires context.

13. 13.

    Go scriptless You are (not) a robot.

14. 14.

    Freedom of speech Say “help” for more options

15. 15.

    Freedom of speech Say “help” for more options

16. 16.

    Anticipate errors Shortcuts can cause ambiguity

17. 17.

    ”Buy me a KitKat bar.”

18. 18.

    ”Buy me a KitKat bar.”

19. 19.

    ”Buy me a KitKat bar.”

20. 20.

    ”Buy me a KitKat bar.”

21. 21.

    ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?”
22. 22.

    ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?” “Yes.”
23. 23.

    ”Buy me a KitKat bar.” ”Okay, ordering a king sized

    KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
24. 24.

    Anticipate errors Shortcuts can cause ambiguity

25. 25.

    Be Flexible Shortcuts can cause ambiguity

26. 26.

    Types of actions Conversation and direct

27. 27.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability

28. 28.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You

    handle interaction and fulfillment Availability
29. 29.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You

    handle interaction and fulfillment Availability Now
30. 30.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Availability Now
31. 31.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
32. 32.

    Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this

    action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
33. 33.

    Conversation Actions " Less ambiguity # Limited trigger

34. 34.

    Making Conversation Actions

35. 35.

    Conversation Actions Format $ Your Action Input

36. 36.

    Conversation Actions Format $ Your Action Input JSON Request

37. 37.

    Conversation Actions Format $ Your Action Input JSON Request JSON

    Response
38. 38.

    Conversation Actions Format $ Your Action Input Action Response JSON

    Request JSON Response
39. 39.

    Conversation Actions Format Your Action JSON Request JSON Response Conversation

    API
40. 40.

    Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”

41. 41.

    Invocation Dialogs Fulfillment “Ok, where would you like to go?”

    “Paris.”
42. 42.

    Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”

43. 43.

    Invocation - name Unique

44. 44.

    Invocation - name Unique | Easy to say

45. 45.

    Name restrictions Two words (with exceptions)

46. 46.

    Name restrictions Two words (with exceptions) | Avoid reserved words

47. 47.

    Trigger Phrases • Let me talk to… • I want

    to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
48. 48.

    Deep links “Talk to service about topic”

49. 49.

    Not deep links “Talk to service.”

50. 50.

    Conversations You need to end them

51. 51.

    API.AI Actions SDK

52. 52.

    API.AI Actions SDK GUI

53. 53.

    API.AI Actions SDK GUI Natural Language Understanding

54. 54.

    API.AI Actions SDK GUI Natural Language Understanding Integrations

55. 55.

    API.AI Actions SDK It’s all on you GUI Natural Language

    Understanding Integrations
56. 56.

    None
57. 57.

    $ API.AI Agent Entities Actions

58. 58.

    Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
59. 59.

    Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
60. 60.

    Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
61. 61.

    Intent examples • Set an alarm for 6:00am • Wake

    me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
62. 62.

    Welcome Intent Talk to Clocky

63. 63.

    Fallback Intent Tell Clocky to order me a pizza

64. 64.

    Fallback Intent Tell Clocky to order me a pizza I

    don’t know how to help with that.
65. 65.

    Action Capture parameters

66. 66.

    Action Capture parameters | Prompt for parameters

67. 67.

    System Entitities • Number • Name • Location • Date/time

    • Airport • And so much more!
68. 68.

    Custom Entities • Food menu • Team names • Fantasy

    names
69. 69.

    Response These are logic-less

70. 70.

    Demo time

71. 71.

    Advanced Fulfillment For everything else, there’s webhooks

72. 72.

    $ (Your action) Input Action Response JSON Request JSON Response

73. 73.

    $ Your server

74. 74.

    $ Your server API.AI Webhook

75. 75.

    $ Your server API.AI Webhook (also JSON)

76. 76.

    Hosting Host anywhere

77. 77.

    Hosting Host anywhere | Node.js recommended

78. 78.

    Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =

    express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
79. 79.

    Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);

    function setAlarm() { assistant.tell("Hello from the webhook!"); }
80. 80.

    Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);

    function setAlarm() { assistant.tell("Hello from the webhook!"); }
81. 81.

    Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I

    will set an alarm for" + time); }
82. 82.

    SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response

    uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
83. 83.

    User Identity • ID (resettable!) • Display name • Given

    name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
84. 84.

    User Identity • ID (resettable!) • Display name • Given

    name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
85. 85.

    Context Context allows for natural conversations

86. 86.

    What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. !
87. 87.

    What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
88. 88.

    What’s the weather like in Minneapolis? It is currently 10°

    in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
89. 89.

    Interaction 1 Interaction 2 Context Output Input

90. 90.

    Output Context Contains all parameters from the action

91. 91.

    Input Context Intents only trigger if context exists

92. 92.

    Input Context Intents only trigger if context exists Allows for

    branching flows
93. 93.

    Context - Lifetime Duration (in requests)

94. 94.

    Context - Lifetime Duration (in requests) | Ends with conversation

95. 95.

    Demo time

96. 96.

    Publishing

97. 97.

    Actions Directory

98. 98.

    Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review
99. 99.

    Create a Google Cloud Project Register the Action Deploy via

    API.AI Wait for review
100. 100.

     Create a Google Cloud Project Register the Action Deploy via

     API.AI Wait for review
101. 101.

     Create a Google Cloud Project Register the Action Deploy via

     API.AI Wait for review