Extending the Google Assistant with Actions on Google

The Google Assistant is the conversational user interface that helps you get things done in your world. Actions on Google let you build on this assistance, while your integrations can help you engage users through Google Home on Pixel, Android and many other devices that connect with Google Assistant. In this session, we'll share the latest innovations behind the Google Assistant and how you can leverage those technologies and best practices for Voice User Interface design to build your own custom extensions to Google Assistant.

Transcript

1. Extending the Google Assistant with Actions on Google Guillaume Laforge, Developer Advocate, GCP Brad Abrams, Product Manager, Google Assistant
4. A conversation between you and Google that helps you get more done in your world
7. Actions on Google How YOU can extend the Google Assistant Connect with users wherever the Google Assistant is available Help users get things done with your service Innovate with Conversational Interface
8. Proprietary & Confidential 8
9. Ok Google, let me talk to Cloud NEXT 2017 Hi, I’ll be your guide to Google Cloud Next, I can help you explore topics or pick a session to attend. What would you like to know? When is the next App Engine talk? Sure! Here’s Cloud NEXT 2017. Enter Earcon The next session about App Engine is “Building Data Driven applications with App Engine” in room 4004 on Tuesday at 4. Is there another topic you’re interested in? Exit Earcon . . .
10. Agenda Design What to build Develop How to build it Deploy How to deliver it
11. Design What to build
12. Why conversation?
13. Confidential and proprietary How do we deliver on Sci-Fi’s promise?
14. Confidential and proprietary How do we get there?
15. Confidential and proprietary A conversation is a contract between two participants with a mutual investment in the outcome
16. Confidential and proprietary Conversations are hard to codify
18. Spoken Language 100,000 yrs Writing 5,000 yrs Print 575 yrs Typing 150 yrs TTS 80 yrs Computers 70 yrs Speech recogn. 50 yrs Word processor 40 yrs IVRs, virtual assistants & smartphones 25 yrs Dictation software 20 yrs Chat bots & A.I. assistants <7yrs Internet 30 yrs
19. 1. List out your core brand attributes (what words define the experience you’re shooting for?) 2. Correlate to attributes that will define your functional design principles (how will those manifest in the design?) 3. Define some attributes that you’d want to infuse into the voice, style of writing, and personality of the dialog (what personality traits match your strategy?) 4. Style guide & “bio sketch” (practical application and maintain consistency for longevity of your experience) knowledgeable helpful encouraging data rich recommending proactive geeky eager motivating Create your persona
20. Example style guide INSTEAD OF... IS MORE LIKELY TO SAY... I found Up for that? Does that sound good? Maybe later While you’re at it... what’s going on I did not receive a response if you feel you have reached this message in error please select from one of the following X options to help us serve you better for questions related to... you have entered that was an invalid… we require that you... please try again for faster answers we’re sorry, we are unable to… I did not understand MIGHT SAY THINGS LIKE... so you can keep up to date on, I’ll look it up right now Sure, that’s coming up Right around the corner from… That session’s full, but… You might like lets need can’t because more about help right now one sec stay allows require unable to due to additional regarding assist currently please hold remain WOULD NEVER SAY...
21. Write your sample dialogs
22. Sample dialogs 1. Canonical “Happy path” 2. First time experience 3. Tapered experience (Return user) 4. Repair 5. Personality questions g.co/dev/ActionsChecklist
23. Develop How to build it
24. Life of a conversation “Ok Google, talk to Cloud NEXT 2017” Invoke “Cloud NEXT” action “Hi! Welcome to Cloud Next...” Speech to Text “The next Machine Learning Session is…” “I want to hear more about Machine Learning” Text to Speech “Sure, here’s Cloud NEXT 2017” Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile Text to Speech
25. Cross-platform tool for building advanced conversational interfaces ListSessions (topics.ComputeEngine); “What GCE breakout sessions are there?”
26. Demo — API.AI
27. A “serverless platform for building event-based microservices”. Function-as-a-service approach Great fit for event-oriented architectures, supporting 3 kind of triggers: • Cloud Storage updates • Cloud Pub/Sub messages • Direct HTTP calls Cloud Functions
28. Cloud Functions Completely serverless & fully managed service ⇒ don’t worry about the ops! Automatic scaling and super-fast ⇒ grows with the success of your project ⇒ cost-effective Open and familiar ⇒ JavaScript / Node.js
29. Architecture in development
30. Architecture in development ngrok
31. Fast feedback loop: Ngrok + Functions emulator Google Cloud Functions emulator Ngrok secure internet tunnels to localhost LIVE RELOADING DEBUG IN CHROME
32. Demo — Cloud Functions & local development
33. server.js NPM dependencies • actions-on-google • node-fetch Export a function with the ApiAiAssistant handling the requests const Assistant = require('actions-on-google').ApiAiAssistant; const fetch = require('node-fetch'); function listTopicsIntent(assistant) { fetch('https://cloudnext.withgoogle.com/api/v1/categories') .then(response => response.text()) .then(text => { let data = JSON.parse(text.split('\n')[1]); let topics = data.categories .find(cat => cat.name === "Topics") .children.map(topic => topic.name).join(', ')); assistant.ask(`The topics covered are: ${topics}. What do you want to learn?'`); }); } exports.agent = function(request, response) { var assistant = new Assistant({ request, response }); assistant.handleRequest(listTopicsIntent); };
35. Deploy How to deliver it
36. Deploying Cloud Functions In production gcloud beta functions deploy agent \ --trigger-http \ --stage-bucket gs://gcp-next-2017-agent/ Locally functions deploy agent --trigger-http
37. Demo — deploying a Cloud Function
38. Review & approval Web-based portal • Triggering Information • Merchandising and information Approvals • Automatic and manual policy checks • Turn around in about 1 week
39. Discovery Discovery patterns • Guaranteed invocation ◦ “Talk to Cloud Next 2017” • Discovery Patterns ◦ “What’s happening at Next?” Google Home app
40. Demo — deploying action on Google
42. Key takeaways 1. You can (and should!) extend the Google Assistant with your custom action 2. Talking with humans is challenging, but API.AI makes it approachable 3. GCP offers a powerful platform for hosting business logic
43. Other Sessions Covering Conversational Application Development at NEXT BD223: Your next app might be a bot! Building conversational user experiences with API.AI BD225: Extending the Google Assistant with Actions on Google IO225: Best practices for building AI assistant in the enterprise What: How to build a conversational application with API.AI including tips and tricks from other developers Who: • Dan Aharon, Google • Cornelius Munz, Bosch • Petr Netusil, ING Belgium What: How to create an Action that extends Google Assistant and is hosted on GCP Who: • Brad Abrams, Google • Guillaume Laforge, Google What: How to use serverless architecture and Cloud Functions to build conversational applications Who: • Alan Ho, Google • Amir Shevat, Slack
44. g.co/dev/ActionsDesign
45. Questions & answers
46. Thank you