Actions on Google Assistant and DialogFlow

How does Google Assistant work? How to design and add new capabilities? In this talk, we'll see how to interact with the Assistant in Google Home and other devices, how to add custom Actions and how to understand what users are saying and reply to them using DialogFlow service to understand user's request.

Transcript

1. Alfredo Morresi @rainbowbreeze
2. The Google Assistant
3. The Google Assistant
4. Ready to help across 400M+ devices
5. Actions on Google
6. Alfredo Morresi ROLE Developer Relations Reach me rainbowbreeze@google.com @rainbowbreeze Community, Development, Snowboarding, Tiramisu' PASSIONS
9. Assistant App
12. Action
13. Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
14. Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
15. “ Intent Matching — Entity Extraction — @
33. const App = require('actions-on-google').ApiAiApp; exports.yourApp = (request, response) => { const app = new App({request, response}); console.log('Request headers: ' + JSON.stringify(request.headers)); console.log('Request body: ' + JSON.stringify(request.body)); // Fulfill action business logic function responseHandler (app) { // Complete your fulfillment logic and send a response app.ask('Hello, World!'); } const actionMap = new Map(); actionMap.set('<API.AI_action_name>', responseHandler); app.handleRequest(actionMap); };
34. const app = new ApiAiApp({request: request, response: response}); const WELCOME_INTENT = 'input.welcome'; // the action name from the API.AI intent const NUMBER_INTENT = 'input.number'; // the action name from the API.AI intent const NUMBER_ARGUMENT = 'input.mynum'; // the action name from the API.AI intent function welcomeIntent (app) { app.ask('Welcome to action snippets! Say a number.'); } function numberIntent (app) { let number = app.getArgument(NUMBER_ARGUMENT); app.tell('You said ' + number); } let actionMap = new Map(); actionMap.set(WELCOME_INTENT, welcomeIntent); actionMap.set(NUMBER_INTENT, numberIntent); app.handleRequest(actionMap);
38. g.co/actionswebsim
42. if (app.hasSurfaceCapability(app.SurfaceCapabilities.SCREEN_OUTPUT)) { app.ask(app.buildRichResponse() .addSimpleResponse(`Here's a fact for you. ${fact} Which one ` + `do you want to hear about next, Google's history or headquarters?`) .addBasicCard( app.buildBasicCard('Google is an amazing company.') .setImage(GOOGLE_LOGO_SRC)) .addSuggestions(['History', 'Headquarters'])); } else { app.ask(`Here's a fact for you. ${fact} Which one ` + `do you want to hear about next, Google's history or headquarters?`); }
45. g.co/dev/ActionsDesign
46. Hey! This is ___ Welcome to ___ Ready to play ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
47. <speak> <!-- Must be at the start of the string --> <say-as interpret-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <say-as interpret-as="cardinal">10</say-as>. <say-as interpret-as="ordinal">10</say-as> <say-as interpret-as="characters">10</say-as>. <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> “S S M L” [3 second pause] [audio file plays] “Ten” “Tenth” “One Oh” World Wide Web [two sentences]
48. function welcome (app) { return isPreviousUser(app.getUser().userId).then((userHasVisited) => { if (userHasVisited) { app.ask(`Welcome to Number Genie!...`, NO_INPUT_PROMPTS); } else { app.ask(`Hey you're back...`, NO_INPUT_PROMPTS); } }); }
49. Support different surface capabilities https://developers.google.com/actions/assistant/surface-capabilities
50. Support speech and display text https://developers.google.com/actions/assistant/responses function simpleResponse (app) { app.ask({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number, like 42. What do you have in mind?', displayText: 'Howdy! I can tell you fun facts about ' + 'almost any number. What do you have in mind?' }); }
51. Guide the user (suggestion chips) https://developers.google.com/actions/assistant/responses function suggestionChips (app) { app.ask(app.buildRichResponse() .addSimpleResponse({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number like 0, 42, or 100. What number do you have ' + 'in mind?', displayText: 'Howdy! I can tell you fun facts about almost any ' + 'number. What number do you have in mind?' }) .addSuggestions(['0', '42', '100', 'Never mind']) ); }
52. Display basic cards https://developers.google.com/actions/assistant/responses function basicCard (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Math and prime numbers it is!') .addBasicCard( app.buildBasicCard(`42 is an even composite number. It ` + `is composed of three distinct prime numbers multiplied together. It ` + `has a total of eight divisors. 42 is an abundant number, because the ` + `sum of its proper divisors 54 is greater than itself. To count from ` + `1 to 42 would take you about twenty-one…`) .setTitle('Math & prime numbers') .addButton('Read more') .setImage('https://example.google.com/42.png', 'Image alternate text') ) ); }
53. Used for easy selection Used for comparison Lists and carousels for selection https://developers.google.com/actions/assistant/responses
54. Ask for information https://developers.google.com/actions/assistant/helpers#user_information let permission = app.SupportedPermissions.DEVICE_COARSE_LOCATION; app.askForPermission('To find bookstores near you', permission);
55. Link an account to your OAuth2 https://developers.google.com/actions/identity/
56. • Build orders • Use Google provided payment instrument • Use your payment processor (Stripe, Braintree, Vantiv, more coming) • Update order statu Transact with the user https://developers.google.com/actions/transactions (check policies and guidelines for availability)
57. Transact with the user https://developers.google.com/actions/identity
58. Smart Home Device Integration developers.google.com/actions/smarthome/
59. Discovery in the Google Assistant Directory
60. Link to your Action From anywhere
61. Google Assistant SDK developers.google.com/assistant/sdk
63. Alfredo Morresi @rainbowbreeze