Introduction to Google Assistant

I participated the first DevFest1 Veneto giving this introduction to Google Assistant.
In this talk I covered what Assistant is and how developers can extend its functionalities by custom Conversational UX.

1. Introduction to Google Assistant Daniele Bonaldo
2. Daniele Bonaldo @danybony_ danybony danielebonaldo.com
4. Google Home/Mobile device — The surface to interact with the Assistant. The Google Assistant — A conversation between you and Google that helps you get things done in your world. Actions on Google — How developers can extend the assistant (via Assistant apps)
5. Assistant App
6. Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
7. How does it work?
8. Assistant app { conversation api request } { conversation api response } user input app response
9. Ok Google, talk to Personal Chef Well, it’s kind of cold outside, so I’d like... Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text NLP Knowledge Graph ML Ranking User Profile Text to Speech Speech to Text Text to Speech ... Invoke Personal Chef action Parse query and generate response
10. Well, it’s kind of cold outside, so I'd like something to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
11. “ Intent Matching — Match and categorize user utterances to an intent. Entity Extraction — Identify key words and phrases spoken by the user. @
13. Ok Google, talk to Personal Chef Well, it’s kind of cold outside, so I’d like... Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text NLP Knowledge Graph ML Ranking User Profile Speech to Text Text to Speech ... ... ... Trigger WELCOME event ... NLP: Intent Matching Entity Extraction
16. Intent Triggered via a series of “user says” phrases or platform based events Can collects entity values Matched at every turn of conversation
17. Intent name Name to use in Dialogflow console Only used by the developer, not referenced in your code
18. Action name String passed to the webhook that indicates what action is being performed Can share the same action name across different Intents Dialogflow specific use of “Action” compared to Google Assistant “Conversation Action”
19. Input and Output Contexts You can require a context to be available before an Intent is enabled Intent can set context to enable other Intents Implement flow control between some Intents
20. Parameters Values that we are trying to capture from the user phrases Can specify a parameter name and a type of value Values can be optional Values can be a list of fixed values
21. Text response Spoken when the Intent is complete Embed entity values if desired - “I found a recipe for $protein and $vegetable!” Webhook can dynamically adjust this response
23. Developer entities
24. Specify follow up questions if a user doesn’t specify certain values Read out in random order to make it more natural Prompts
25. Fulfillment
26. Test Console Test your agent by entering text or voice requests. Voice testing is available in Chrome and Firefox browsers.
28. Actions Console console.actions.google.com
29. g.co/actionswebsim
30. Action naming and policies https://developers.google.com/actions/distribute/general-policies Policies about what Conversation Actions are allowed See the guidelines for more information Invocation names must be unique since it is a global name Direct Actions not widely available to build and deploy, register for interest
31. From ‘app’ to ‘experience’...
32. Preserve and reinforce your persona by engaging the user as a separate entity from the Google Assistant. Own it. Hey! This is ___ Welcome to ___ Ready to play ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
33. <speak> <!-- Must be at the start of the string --> <say-as interpret-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <say-as interpret-as="cardinal">10</say-as>. <say-as interpret-as="ordinal">10</say-as> <say-as interpret-as="characters">10</say-as>. <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> Reinforce it with SSML: Speech Synthesis Markup Language https://developers.google.com/actions/reference/ssml “S S M L” [3 second pause] [audio file plays] “Ten” “Tenth” “One Oh” World Wide Web [two sentences]
34. function welcome (app) { return isPreviousUser(app.getUser().userId).then((userHasVisited) => { if (!userHasVisited) { app.ask(`Welcome to Number Genie!...`, NO_INPUT_PROMPTS); } else { app.ask(`Hey you're back...`, NO_INPUT_PROMPTS); } }); } Reinforce it with persistence https://developers.google.com/actions/assistant/best-practices
35. Support different surface capabilities https://developers.google.com/actions/assistant/surface-capabilities AUDIO_OUTPUT SCREEN_OUTPUT
36. function simpleResponse (app) { app.ask({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number, like 42. What do you have in mind?', displayText: 'Howdy! I can tell you fun facts about ' + 'almost any number. What do you have in mind?' }); } Support speech and display text https://developers.google.com/actions/assistant/responses Chat text should be a subset of audio
37. function suggestionChips (app) { app.ask(app.buildRichResponse() .addSimpleResponse({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number like 0, 42, or 100. What number do you have ' + 'in mind?', displayText: 'Howdy! I can tell you fun facts about almost any ' + 'number. What number do you have in mind?' }) .addSuggestions(['0', '42', '100', 'Never mind']) ); } Guide the user (suggestion chips) https://developers.google.com/actions/assistant/responses
38. function basicCard (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Math and prime numbers it is!') .addBasicCard( app.buildBasicCard(`42 is an even composite number. It ` + `is composed of three distinct prime numbers multiplied together. It ` + `has a total of eight divisors. 42 is an abundant number, because the ` + `sum of its proper divisors 54 is greater than itself. To count from ` + `1 to 42 would take you about twenty-one…`) .setTitle('Math & prime numbers') .addButton('Read more') .setImage('https://example.google.com/42.png', 'Image alternate text') ) ); } Display basic cards https://developers.google.com/actions/assistant/responses
39. Lists and carousels for selection https://developers.google.com/actions/assistant/responses Used for easy selection <10 items Used for comparison <30 items
40. Top 3 Design Tips
41. Create a persona. 1
42. Persona is conveyed through: Tone Word and phrase choices Functional design Style Technique Voice ...and based on: Your user population and their needs The imagery & qualities associated with your brand
43. Think outside the box. Literally. 2
44. code code code code code code code code code code code “ dialog string” code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code code code STRUCTURE + CODE
45. code code code code code code code code code code code “ dialog string” code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code code code STRUCTURE + CODE
46. DIALOG + STRUCTURE
47. In conversations, there are no “errors”. 3
48. When a so-called “error” occurs in a conversation, it should be treated simply as a new turn in the dialog, only with different conditions.
49. More info The Conversational UI and Why It Matters developers.google.com/actions/design/ Codelabs codelabs.developers.google.com
50. Daniele Bonaldo @danybony_ danybony danielebonaldo.com Thank you!