Actions with Google

Learn how to create custom actions for the Google Assistant

Transcript

1. Actions with Google
   Bryan Herbst
   Android Engineer @ Target
   

   View Slide

2. The Google
   Assistant
   “Ok Google” Allo
   Google Home Android Auto
   Android Wear
   

   View Slide

3. The Google Assistant
   Google Home
   

   View Slide

4. Design principles
   Voice interaction is different
   

   View Slide

5. What actions
   can I take?
   

   View Slide

6. What actions
   can I take?
   

   View Slide

7. What should I have
   for dinner?
   What are you in the mood
   for? You can say Italian,
   American, Mexican, or
   Chinese.
   !
   

   View Slide

8. What should I have
   for dinner?
   What are you in the mood
   for? You can say Italian,
   American, Mexican, or
   Chinese.
   ! Take turns.
   

   View Slide

9. “Guide me downtown.”
   

   View Slide

10. “Guide me downtown.”
    Downtown where?
    New York?
    Minneapolis?
    

    View Slide

11. “Guide me downtown.”
    Driving or walking?
    Starting from where?
    

    View Slide

12. Speech uses shortcuts
    Filling in the blanks requires context.
    

    View Slide

13. Go scriptless
    You are (not) a robot.
    

    View Slide

14. Freedom of speech
    Say “help” for more options
    

    View Slide

15. Freedom of speech
    Say “help” for more options
    

    View Slide

16. Anticipate errors
    Shortcuts can cause ambiguity
    

    View Slide

17. ”Buy me a KitKat bar.”
    

    View Slide

18. ”Buy me a KitKat bar.”
    

    View Slide

19. ”Buy me a KitKat bar.”
    

    View Slide

20. ”Buy me a KitKat bar.”
    

    View Slide

21. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    

    View Slide

22. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “Yes.”
    

    View Slide

23. ”Buy me a KitKat bar.”
    ”Okay, ordering a king sized KitKat
    bar from Amazon. Is that correct?”
    “No, make it a regular sized KitKat bar.”
    

    View Slide

24. Anticipate errors
    Shortcuts can cause ambiguity
    

    View Slide

25. Be Flexible
    Shortcuts can cause ambiguity
    

    View Slide

26. Types of actions
    Conversation and direct
    

    View Slide

27. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    Availability
    

    View Slide

28. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability
    

    View Slide

29. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now
    

    View Slide

30. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Availability Now
    

    View Slide

31. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now
    

    View Slide

32. Conversation Direct
    Trigger
    “Talk to SERVICE about
    TOPIC”
    “Do this action”
    Responsibilities
    You handle interaction
    and fulfillment
    Google handles interaction,
    you handle fulfillment.
    Availability Now Soon
    

    View Slide

33. Conversation Actions
    " Less ambiguity
    # Limited trigger
    

    View Slide

34. Making Conversation
    Actions
    

    View Slide

35. Conversation Actions Format
    $ Your Action
    Input
    

    View Slide

36. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request
    

    View Slide

37. Conversation Actions Format
    $ Your Action
    Input
    JSON
    Request
    JSON
    Response
    

    View Slide

38. Conversation Actions Format
    $ Your Action
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response
    

    View Slide

39. Conversation Actions Format
    Your Action
    JSON Request
    JSON Response
    Conversation
    API
    

    View Slide

40. Invocation
    Dialogs
    Fulfillment
    “Ask AwesomeFlightApp
    to book me a flight.”
    

    View Slide

41. Invocation
    Dialogs
    Fulfillment
    “Ok, where would you
    like to go?”
    “Paris.”
    

    View Slide

42. Invocation
    Dialogs
    Fulfillment
    “Okay, booking a flight
    to Paris!”
    

    View Slide

43. Invocation - name
    Unique
    

    View Slide

44. Invocation - name
    Unique | Easy to say
    

    View Slide

45. Name restrictions
    Two words (with exceptions)
    

    View Slide

46. Name restrictions
    Two words (with exceptions) | Avoid reserved words
    

    View Slide

47. Trigger Phrases
    • Let me talk to…
    • I want to talk to...
    • Can I talk to...
    • Talk to...
    • Ask...
    • At...
    • Tell...
    • Tell me...
    

    View Slide

48. Deep links
    “Talk to service about topic”
    

    View Slide

49. Not deep links
    “Talk to service.”
    

    View Slide

50. Conversations
    You need to end them
    

    View Slide

51. API.AI
    Actions SDK
    

    View Slide

52. API.AI
    Actions SDK
    GUI
    

    View Slide

53. API.AI
    Actions SDK
    GUI
    Natural Language Understanding
    

    View Slide

54. API.AI
    Actions SDK
    GUI
    Natural Language Understanding
    Integrations
    

    View Slide

55. API.AI
    Actions SDK
    It’s all on you
    GUI
    Natural Language Understanding
    Integrations
    

    View Slide

56. View Slide

57. $
    API.AI Agent
    Entities Actions
    

    View Slide

58. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm
    

    View Slide

59. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm
    

    View Slide

60. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm
    

    View Slide

61. Intent
    examples
    • Set an alarm for 6:00am
    • Wake me up at 6:00am
    • I need an alarm at 6:00am
    • Set an alarm for March 1st at 6:00pm
    

    View Slide

62. Welcome
    Intent
    Talk to Clocky
    

    View Slide

63. Fallback
    Intent
    Tell Clocky to order me a pizza
    

    View Slide

64. Fallback
    Intent
    Tell Clocky to order me a pizza
    I don’t know how to help with that.
    

    View Slide

65. Action
    Capture parameters
    

    View Slide

66. Action
    Capture parameters | Prompt for parameters
    

    View Slide

67. System
    Entitities
    • Number
    • Name
    • Location
    • Date/time
    • Airport
    • And so much more!
    

    View Slide

68. Custom
    Entities
    • Food menu
    • Team names
    • Fantasy names
    

    View Slide

69. Response
    These are logic-less
    

    View Slide

70. Demo time
    

    View Slide

71. Advanced Fulfillment
    For everything else, there’s webhooks
    

    View Slide

72. $
    (Your action)
    Input
    Action
    Response
    JSON
    Request
    JSON
    Response
    

    View Slide

73. $ Your server
    

    View Slide

74. $ Your server
    API.AI
    Webhook
    

    View Slide

75. $ Your server
    API.AI
    Webhook
    (also JSON)
    

    View Slide

76. Hosting
    Host anywhere
    

    View Slide

77. Hosting
    Host anywhere | Node.js recommended
    

    View Slide

78. Simple webhook server
    let Assistant = require('actions-on-google').ApiAiAssistant;
    let app = express();
    app.post('/', function (req, res) {
    const assistant = new Assistant({request: req, response: res}); }
    }
    

    View Slide

79. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }
    

    View Slide

80. Basic webhook
    let actionMap = new Map();
    actionMap.set(”set_alarm", setAlarm);
    assistant.handleRequest(actionMap);
    function setAlarm() {
    assistant.tell("Hello from the webhook!");
    }
    

    View Slide

81. Webhook with arguments
    function withArguments() {
    let time= assistant.getArgument(”time");
    assistant.tell(”I will set an alarm for" + time);
    }
    

    View Slide

82. SSML- Speech Synthesis Markup Language
    function withSsml() {
    assistant.tell("This response uses”
    + "SSML.”
    + "Isn't it awesome?");
    }
    

    View Slide

83. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)
    

    View Slide

84. User
    Identity
    • ID (resettable!)
    • Display name
    • Given name
    • Family name
    • Coarse device location (zip code and city)
    • Precise device location (lat/lng, address)
    Requires permission
    

    View Slide

85. Context
    Context allows for natural conversations
    

    View Slide

86. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    

    View Slide

87. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°
    

    View Slide

88. What’s the weather
    like in Minneapolis?
    It is currently 10°
    in Minneapolis.
    !
    Context
    Location: Minneapolis
    Temperature: 10°
    

    View Slide

89. Interaction 1 Interaction 2
    Context
    Output Input
    

    View Slide

90. Output Context
    Contains all parameters from the action
    

    View Slide

91. Input Context
    Intents only trigger if context exists
    

    View Slide

92. Input Context
    Intents only trigger if context exists
    Allows for branching flows
    

    View Slide

93. Context - Lifetime
    Duration (in requests)
    

    View Slide

94. Context - Lifetime
    Duration (in requests) | Ends with conversation
    

    View Slide

95. Demo time
    

    View Slide

96. Publishing
    

    View Slide

97. Actions
    Directory
    

    View Slide

98. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review
    

    View Slide

99. Create a Google Cloud Project
    Register the Action
    Deploy via API.AI
    Wait for review
    

    View Slide

100. Create a Google Cloud Project
     Register the Action
     Deploy via API.AI
     Wait for review
     

     View Slide

101. Create a Google Cloud Project
     Register the Action
     Deploy via API.AI
     Wait for review
     

     View Slide