Google Assistant API - Ok Google... Do something

These are the slides for my talk at the GDG Android Hamburg Meetup from February 2017

Transcript

1. Google Assistant API Ok Google, … Do something! Daniel Hartwich - Android Developer - XING AG - GDG Hamburg Android Meetup 02.02.2017
2. Google Assistant API - Agenda 1. Introduction 2. Google Assistant 3. Build your own bot a. Voice Interaction API b. Optimizing Content for the Assistant c. Google Assistant API using API.AI d. Live Demo - A smart Meetup Notifier Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
4. Google Assistant - Presented by Google at Google I/O 2016 - 20% of queries are voice queries in the US - Using natural language processing engine - Contextual awareness (place, previous queries, time)
5. Google Assistant - Should be there for the users when they need it
6. Google Assistant - Google’s Definition Ask it anything.
7. Google Assistant - Google’s Definition Tell it to do things.
8. Google Assistant - Google’s Definition All your devices. One Assistant
9. Google Assistant - Google’s Definition Safe, secure and in your control.
10. Google Assistant - Google’s Definition Your Assistant works with your favorite stuff, too
11. Build your own Bot Daniel Hartwich - Android Developer - XING AG - Android Developer Meetup 02.02.2017
12. Voice Interaction API - FlowChart
13. Build your own Bot - Voice Interaction API - Voice API was introduced in 2015 - Enables conversations where apps can ask back questions if more information is required by an app - Confirm actions (Place phone call, send SMS)
14. Build your own Bot - Voice Interaction API - Voice Interaction is special kind of Android Activity - VoiceActivity → Complete an action - NormalActivity → Start an action, be completed by touch - Example: Android DIAL intent
15. Build your own Bot - Voice Interaction API Declare Voice Intent Filter in Manifest <activity android:name="org.example.MyVoiceActivity"> <intent-filter> <action android:name="org.example.MY_ACTION_INTENT" /> <category android:name="android.intent.category.DEFAULT" /> <category android:name="android.intent.category.VOICE" /> </intent-filter> </activity>
16. Build your own Bot - Voice Interaction API Handle the voice interaction - The app should decide what’s an appropriate action with the given input - Ask for more input - Confirm an upcoming action - User wants to book a taxi, with all necessary information given - Needs to confirm the time and price - After confirming the action can be finished and the request can be send
17. Build your own Bot - Voice Interaction API class MyVoiceActivity extends Activity { class Confirm extends VoiceInteractor.ConfirmationRequest { public Confirm(String ttsPrompt, String visualPrompt) { VoiceInteractor.Prompt prompt = new VoiceInteractor.Prompt( new String[] {ttsPrompt}, visualPrompt); super(prompt, null); } @Override public void onConfirmationResult( boolean confirmed, Bundle null) { if (confirmed) { doAction(); } finish(); } }; @Override public void onResume() { if (isVoiceInteraction()) { String ttsPrompt = getConfirmationTts(); String visualPrompt = getConfirmationDisplayText(); getVoiceInteractor().sendRequest(new Confirm(ttsPrompt, visualPrompt)); } else { finish(); } } }
18. Build your own Bot - Voice Interaction API Approving interactions without confirmation - There might be cases where it is not necessary to ask for confirmation - Good examples might be: Opening a website (if no more information is needed) Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { // Interaction started by the users voice doAction(); } finish(); }
19. Build your own Bot - Voice Interaction API Finishing the Voice Interaction - If your interaction is completed call finish() - If you want to keep your users in the app, start a new activity with Intent.FLAG_ACTIVITY_NEW_TASK Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { doAction(); } // Start my main non-voice Activity Intent intent = new Intent(this, MyMainActivity.class); intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK); startActivity(intent); finish(); } }
20. Build your own Bot - Voice Interaction API More information: - Codelab: http://io2015codelabs.appspot.com/codelabs/voice-interaction#1 - https://developers.google.com/voice-actions/interaction/ - https://developers.google.com/voice-actions/interaction/voice-interactions
21. Optimize Content for Assistant - aka Now on Tap -
22. Optimize Content for Assistant - aka Now on Tap - since 6.0 Marshmallow - show Deeplinks to relevant apps that are contextually relevant - Some things to look after as a ‘source’ app
23. Sharing content with the assistant - Follow accessibility best practices and you will be fine - Possibility to share additional content (e.g. name of the currently playing album) - Some listeners to register for providing context
24. Sharing content with the assistant
25. Sharing content with the assistant Example: Providing music app data @Override public void onProvideAssistContent(AssistContent assistContent) { super.onProvideAssistContent(assistContent); String structuredJson = new JSONObject() .put("@type", "MusicRecording") .put("@id", "https://example.com/music/recording") .put("name", "Album Title") .toString(); assistContent.setStructuredData(structuredJson); }
26. Sharing content with the assistant - Exclude sensitive infos using FLAG_SECURE
27. Sharing content - Destination app - Assistant takes advantage of deeplinking - FireBase App Indexing also helps with being linked - You can also implement your own Assistant
28. Sharing content with the assistant More information: - https://developer.android.com/training/articles/assistant.html - https://developer.android.com/guide/topics/ui/accessibility/apps.html - https://developer.android.com/reference/android/app/Activity.html#onProvideAssistContent(a ndroid.app.assist.AssistContent)
29. Google Assistant API using API.AI
30. Google Assistant API 1. Design a. Conversation basics b. Crafting a conversation c. Best practices 2. Develop a. Components of Conversation Action b. Conversation API c. Actions SDK d. API.AI 3. Example Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
31. Design - Conversation basics - Problem of recognizing spoken input - mostly solved - New challenge: Building a user experience that is similar to a natural human conversation - Turn Taking - Threading - Leveraging the inherent efficiency of speech (read between lines) - Anticipating variable user behaviour
32. Design - Crafting a conversation - Pick the right use cases - Create a persona - Write dialogs - Test it out - Build and iterate
33. Design - Best practices - Be cooperative… Like your users - Quality (only true things) - Quantity (don’t say too much or too less) - Relevance - Manner (be brief, to the point, no ambiguity
34. Design - Best practices - Unlocking the Power of Spoken Language - Communicate what was understood - Offer examples that show what’s possible - Avoid stating the obvious - Give users credit
35. Develop - Components of Conversation Action - Conversation Actions help you fulfill user requests - Request → Process Request → Determine best action → Invoke Conversation Action if relevant
36. Develop - Components of Conversation Action
37. Develop - Conversation API - Conversation API defines a request and response format to communicate with Google Assistant
38. Develop - Action SDK - Helper to parse requests and construct responses - Comes in a NodeJS client library
39. - NodeJS client library - This library provides convenience methods to parse requests and construct responses that adhere to the Conversation API. You use this library in your fulfillment code. - Web Simulator - a web-based, virtual Google Home device to simulate your actions on - gactions CLI - a command-line interface to test actions and deploy action packages - Action Package definition - A JSON-based definition that describes a group of actions, how they are invoked, what fulfillment endpoints to call, and other important metadata. You provide this file, along with other assets, when you submit your action for approval. Develop - Action SDK - What’s inside?
40. Develop - Action SDK - API.AI - Action SDK has everything we need - Better developer experience - Makes building actions easier - API.AI NLU - GUI Interface - Conversation building features
41. Develop - Action SDK - API.AI
42. Develop - API.AI - Example
43. Develop - API.AI - Example - API.AI Agent → Conversation Action - Meetup Helper Agent - Intents → Welcome Intent /Fallback Intent /Reco Intent - Entities (values) - Prebuilt System entities (time, number, address) - Developer entities
44. Develop - API.AI - Example - Integrations → Google Assistant - Training Section → Teach your assistant - Fulfillment → Send the responses to your service via webhook - Deployment of your assistant - https://developers.google.com/actions/develop/apiai/
45. Develop - API.AI - Example
46. Questions? goo.gl/slides/cdaf73