Introduction to Google Assistant

D4be3ad792b57408b3ab6fe98caef08e?s=47 danybony
December 02, 2017

Introduction to Google Assistant

I participated the first DevFest1 Veneto giving this introduction to Google Assistant.
In this talk I covered what Assistant is and how developers can extend its functionalities by custom Conversational UX.

D4be3ad792b57408b3ab6fe98caef08e?s=128

danybony

December 02, 2017
Tweet

Transcript

  1. Introduction to Google Assistant Daniele Bonaldo

  2. Daniele Bonaldo @danybony_ danybony danielebonaldo.com

  3. None
  4. Google Home/Mobile device — The surface to interact with the

    Assistant. The Google Assistant — A conversation between you and Google that helps you get things done in your world. Actions on Google — How developers can extend the assistant (via Assistant apps)
  5. Assistant App

  6. Well, it’s kind of cold outside, so I'd like something

    to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
  7. How does it work?

  8. Assistant app { conversation api request } { conversation api

    response } user input app response
  9. Ok Google, talk to Personal Chef Well, it’s kind of

    cold outside, so I’d like... Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text NLP Knowledge Graph ML Ranking User Profile Text to Speech Speech to Text Text to Speech ... Invoke Personal Chef action Parse query and generate response
  10. Well, it’s kind of cold outside, so I'd like something

    to warm me up, like a hot soup, and I want it fast. I have some chicken, and also canned tomatoes.
  11. “ Intent Matching — Match and categorize user utterances to

    an intent. Entity Extraction — Identify key words and phrases spoken by the user. @
  12. None
  13. Ok Google, talk to Personal Chef Well, it’s kind of

    cold outside, so I’d like... Sure, here’s Personal Chef What are you in the mood for? What protein would you like to use? Speech to Text NLP Knowledge Graph ML Ranking User Profile Speech to Text Text to Speech ... ... ... Trigger WELCOME event ... NLP: Intent Matching Entity Extraction
  14. None
  15. None
  16. Intent Triggered via a series of “user says” phrases or

    platform based events Can collects entity values Matched at every turn of conversation
  17. Intent name Name to use in Dialogflow console Only used

    by the developer, not referenced in your code
  18. Action name String passed to the webhook that indicates what

    action is being performed Can share the same action name across different Intents Dialogflow specific use of “Action” compared to Google Assistant “Conversation Action”
  19. Input and Output Contexts You can require a context to

    be available before an Intent is enabled Intent can set context to enable other Intents Implement flow control between some Intents
  20. Parameters Values that we are trying to capture from the

    user phrases Can specify a parameter name and a type of value Values can be optional Values can be a list of fixed values
  21. Text response Spoken when the Intent is complete Embed entity

    values if desired - “I found a recipe for $protein and $vegetable!” Webhook can dynamically adjust this response
  22. None
  23. Developer entities

  24. Specify follow up questions if a user doesn’t specify certain

    values Read out in random order to make it more natural Prompts
  25. Fulfillment

  26. Test Console Test your agent by entering text or voice

    requests. Voice testing is available in Chrome and Firefox browsers.
  27. None
  28. Actions Console console.actions.google.com

  29. g.co/actionswebsim

  30. Action naming and policies https://developers.google.com/actions/distribute/general-policies Policies about what Conversation Actions

    are allowed See the guidelines for more information Invocation names must be unique since it is a global name Direct Actions not widely available to build and deploy, register for interest
  31. From ‘app’ to ‘experience’...

  32. Preserve and reinforce your persona by engaging the user as

    a separate entity from the Google Assistant. Own it. Hey! This is ___ Welcome to ___ Ready to play ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
  33. <speak> <!-- Must be at the start of the string

    --> <say-as interpret-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <say-as interpret-as="cardinal">10</say-as>. <say-as interpret-as="ordinal">10</say-as> <say-as interpret-as="characters">10</say-as>. <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> Reinforce it with SSML: Speech Synthesis Markup Language https://developers.google.com/actions/reference/ssml “S S M L” [3 second pause] [audio file plays] “Ten” “Tenth” “One Oh” World Wide Web [two sentences]
  34. function welcome (app) { return isPreviousUser(app.getUser().userId).then((userHasVisited) => { if (!userHasVisited)

    { app.ask(`Welcome to Number Genie!...`, NO_INPUT_PROMPTS); } else { app.ask(`Hey you're back...`, NO_INPUT_PROMPTS); } }); } Reinforce it with persistence https://developers.google.com/actions/assistant/best-practices
  35. Support different surface capabilities https://developers.google.com/actions/assistant/surface-capabilities AUDIO_OUTPUT SCREEN_OUTPUT

  36. function simpleResponse (app) { app.ask({ speech: 'Howdy! I can tell

    you fun facts about ' + 'almost any number, like 42. What do you have in mind?', displayText: 'Howdy! I can tell you fun facts about ' + 'almost any number. What do you have in mind?' }); } Support speech and display text https://developers.google.com/actions/assistant/responses Chat text should be a subset of audio
  37. function suggestionChips (app) { app.ask(app.buildRichResponse() .addSimpleResponse({ speech: 'Howdy! I can

    tell you fun facts about ' + 'almost any number like 0, 42, or 100. What number do you have ' + 'in mind?', displayText: 'Howdy! I can tell you fun facts about almost any ' + 'number. What number do you have in mind?' }) .addSuggestions(['0', '42', '100', 'Never mind']) ); } Guide the user (suggestion chips) https://developers.google.com/actions/assistant/responses
  38. function basicCard (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Math and prime numbers it

    is!') .addBasicCard( app.buildBasicCard(`42 is an even composite number. It ` + `is composed of three distinct prime numbers multiplied together. It ` + `has a total of eight divisors. 42 is an abundant number, because the ` + `sum of its proper divisors 54 is greater than itself. To count from ` + `1 to 42 would take you about twenty-one…`) .setTitle('Math & prime numbers') .addButton('Read more') .setImage('https://example.google.com/42.png', 'Image alternate text') ) ); } Display basic cards https://developers.google.com/actions/assistant/responses
  39. Lists and carousels for selection https://developers.google.com/actions/assistant/responses Used for easy selection

    <10 items Used for comparison <30 items
  40. Top 3 Design Tips

  41. Create a persona. 1

  42. Persona is conveyed through: Tone Word and phrase choices Functional

    design Style Technique Voice ...and based on: Your user population and their needs The imagery & qualities associated with your brand
  43. Think outside the box. Literally. 2

  44. code code code code code code code code code code

    code “ dialog string” code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code code code STRUCTURE + CODE
  45. code code code code code code code code code code

    code “ dialog string” code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code “dialog string” code code code code code code code code code code code code code code code code code code code code code code code code code STRUCTURE + CODE
  46. DIALOG + STRUCTURE

  47. In conversations, there are no “errors”. 3

  48. When a so-called “error” occurs in a conversation, it should

    be treated simply as a new turn in the dialog, only with different conditions.
  49. More info The Conversational UI and Why It Matters developers.google.com/actions/design/

    Codelabs codelabs.developers.google.com
  50. Daniele Bonaldo @danybony_ danybony danielebonaldo.com Thank you!