Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Actions on Google Assistant and DialogFlow

Actions on Google Assistant and DialogFlow

How does Google Assistant work? How to design and add new capabilities? In this talk, we'll see how to interact with the Assistant in Google Home and other devices, how to add custom Actions and how to understand what users are saying and reply to them using DialogFlow service to understand user's request.



March 20, 2018


  1. Alfredo Morresi @rainbowbreeze

  2. The Google Assistant

  3. The Google Assistant

  4. Ready to help across 400M+ devices

  5. Actions on Google

  6. Alfredo Morresi ROLE Developer Relations Reach me rainbowbreeze@google.com @rainbowbreeze Community,

    Development, Snowboarding, Tiramisu' PASSIONS
  7. None
  8. None
  9. Assistant App

  10. None
  11. None
  12. Action

  13. Well, it’s kind of cold outside, so I'd like something

    to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
  14. Well, it’s kind of cold outside, so I'd like something

    to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
  15. “ Intent Matching — Entity Extraction — @

  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. const App = require('actions-on-google').ApiAiApp; exports.yourApp = (request, response) => {

    const app = new App({request, response}); console.log('Request headers: ' + JSON.stringify(request.headers)); console.log('Request body: ' + JSON.stringify(request.body)); // Fulfill action business logic function responseHandler (app) { // Complete your fulfillment logic and send a response app.ask('Hello, World!'); } const actionMap = new Map(); actionMap.set('<API.AI_action_name>', responseHandler); app.handleRequest(actionMap); };
  34. const app = new ApiAiApp({request: request, response: response}); const WELCOME_INTENT

    = 'input.welcome'; // the action name from the API.AI intent const NUMBER_INTENT = 'input.number'; // the action name from the API.AI intent const NUMBER_ARGUMENT = 'input.mynum'; // the action name from the API.AI intent function welcomeIntent (app) { app.ask('Welcome to action snippets! Say a number.'); } function numberIntent (app) { let number = app.getArgument(NUMBER_ARGUMENT); app.tell('You said ' + number); } let actionMap = new Map(); actionMap.set(WELCOME_INTENT, welcomeIntent); actionMap.set(NUMBER_INTENT, numberIntent); app.handleRequest(actionMap);
  35. None
  36. None
  37. None
  38. g.co/actionswebsim

  39. None
  40. None
  41. None
  42. if (app.hasSurfaceCapability(app.SurfaceCapabilities.SCREEN_OUTPUT)) { app.ask(app.buildRichResponse() .addSimpleResponse(`Here's a fact for you. ${fact}

    Which one ` + `do you want to hear about next, Google's history or headquarters?`) .addBasicCard( app.buildBasicCard('Google is an amazing company.') .setImage(GOOGLE_LOGO_SRC)) .addSuggestions(['History', 'Headquarters'])); } else { app.ask(`Here's a fact for you. ${fact} Which one ` + `do you want to hear about next, Google's history or headquarters?`); }
  43. None
  44. None
  45. g.co/dev/ActionsDesign

  46. Hey! This is ___ Welcome to ___ Ready to play

    ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
  47. <speak> <!-- Must be at the start of the string

    --> <say-as interpret-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <say-as interpret-as="cardinal">10</say-as>. <say-as interpret-as="ordinal">10</say-as> <say-as interpret-as="characters">10</say-as>. <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> “S S M L” [3 second pause] [audio file plays] “Ten” “Tenth” “One Oh” World Wide Web [two sentences]
  48. function welcome (app) { return isPreviousUser(app.getUser().userId).then((userHasVisited) => { if (userHasVisited)

    { app.ask(`Welcome to Number Genie!...`, NO_INPUT_PROMPTS); } else { app.ask(`Hey you're back...`, NO_INPUT_PROMPTS); } }); }
  49. Support different surface capabilities https://developers.google.com/actions/assistant/surface-capabilities

  50. Support speech and display text https://developers.google.com/actions/assistant/responses function simpleResponse (app) {

    app.ask({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number, like 42. What do you have in mind?', displayText: 'Howdy! I can tell you fun facts about ' + 'almost any number. What do you have in mind?' }); }
  51. Guide the user (suggestion chips) https://developers.google.com/actions/assistant/responses function suggestionChips (app) {

    app.ask(app.buildRichResponse() .addSimpleResponse({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number like 0, 42, or 100. What number do you have ' + 'in mind?', displayText: 'Howdy! I can tell you fun facts about almost any ' + 'number. What number do you have in mind?' }) .addSuggestions(['0', '42', '100', 'Never mind']) ); }
  52. Display basic cards https://developers.google.com/actions/assistant/responses function basicCard (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Math

    and prime numbers it is!') .addBasicCard( app.buildBasicCard(`42 is an even composite number. It ` + `is composed of three distinct prime numbers multiplied together. It ` + `has a total of eight divisors. 42 is an abundant number, because the ` + `sum of its proper divisors 54 is greater than itself. To count from ` + `1 to 42 would take you about twenty-one…`) .setTitle('Math & prime numbers') .addButton('Read more') .setImage('https://example.google.com/42.png', 'Image alternate text') ) ); }
  53. Used for easy selection Used for comparison Lists and carousels

    for selection https://developers.google.com/actions/assistant/responses
  54. Ask for information https://developers.google.com/actions/assistant/helpers#user_information let permission = app.SupportedPermissions.DEVICE_COARSE_LOCATION; app.askForPermission('To find

    bookstores near you', permission);
  55. Link an account to your OAuth2 https://developers.google.com/actions/identity/

  56. • Build orders • Use Google provided payment instrument •

    Use your payment processor (Stripe, Braintree, Vantiv, more coming) • Update order statu Transact with the user https://developers.google.com/actions/transactions (check policies and guidelines for availability)
  57. Transact with the user https://developers.google.com/actions/identity

  58. Smart Home Device Integration developers.google.com/actions/smarthome/

  59. Discovery in the Google Assistant Directory

  60. Link to your Action From anywhere

  61. Google Assistant SDK developers.google.com/assistant/sdk

  62. None
  63. Alfredo Morresi @rainbowbreeze