Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google Assistant API - Ok Google... Do something

Google Assistant API - Ok Google... Do something

These are the slides for my talk at the GDG Android Hamburg Meetup from February 2017

Daniel Hartwich

February 02, 2017

More Decks by Daniel Hartwich

Other Decks in Technology


  1. Google Assistant API Ok Google, … Do something! Daniel Hartwich

    - Android Developer - XING AG - GDG Hamburg Android Meetup 02.02.2017
  2. Google Assistant API - Agenda 1. Introduction 2. Google Assistant

    3. Build your own bot a. Voice Interaction API b. Optimizing Content for the Assistant c. Google Assistant API using API.AI d. Live Demo - A smart Meetup Notifier Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
  3. Google Assistant - Presented by Google at Google I/O 2016

    - 20% of queries are voice queries in the US - Using natural language processing engine - Contextual awareness (place, previous queries, time)
  4. Build your own Bot Daniel Hartwich - Android Developer -

    XING AG - Android Developer Meetup 02.02.2017
  5. Build your own Bot - Voice Interaction API - Voice

    API was introduced in 2015 - Enables conversations where apps can ask back questions if more information is required by an app - Confirm actions (Place phone call, send SMS)
  6. Build your own Bot - Voice Interaction API - Voice

    Interaction is special kind of Android Activity - VoiceActivity → Complete an action - NormalActivity → Start an action, be completed by touch - Example: Android DIAL intent
  7. Build your own Bot - Voice Interaction API Declare Voice

    Intent Filter in Manifest <activity android:name="org.example.MyVoiceActivity"> <intent-filter> <action android:name="org.example.MY_ACTION_INTENT" /> <category android:name="android.intent.category.DEFAULT" /> <category android:name="android.intent.category.VOICE" /> </intent-filter> </activity>
  8. Build your own Bot - Voice Interaction API Handle the

    voice interaction - The app should decide what’s an appropriate action with the given input - Ask for more input - Confirm an upcoming action - User wants to book a taxi, with all necessary information given - Needs to confirm the time and price - After confirming the action can be finished and the request can be send
  9. Build your own Bot - Voice Interaction API class MyVoiceActivity

    extends Activity { class Confirm extends VoiceInteractor.ConfirmationRequest { public Confirm(String ttsPrompt, String visualPrompt) { VoiceInteractor.Prompt prompt = new VoiceInteractor.Prompt( new String[] {ttsPrompt}, visualPrompt); super(prompt, null); } @Override public void onConfirmationResult( boolean confirmed, Bundle null) { if (confirmed) { doAction(); } finish(); } }; @Override public void onResume() { if (isVoiceInteraction()) { String ttsPrompt = getConfirmationTts(); String visualPrompt = getConfirmationDisplayText(); getVoiceInteractor().sendRequest(new Confirm(ttsPrompt, visualPrompt)); } else { finish(); } } }
  10. Build your own Bot - Voice Interaction API Approving interactions

    without confirmation - There might be cases where it is not necessary to ask for confirmation - Good examples might be: Opening a website (if no more information is needed) Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { // Interaction started by the users voice doAction(); } finish(); }
  11. Build your own Bot - Voice Interaction API Finishing the

    Voice Interaction - If your interaction is completed call finish() - If you want to keep your users in the app, start a new activity with Intent.FLAG_ACTIVITY_NEW_TASK Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { doAction(); } // Start my main non-voice Activity Intent intent = new Intent(this, MyMainActivity.class); intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK); startActivity(intent); finish(); } }
  12. Build your own Bot - Voice Interaction API More information:

    - Codelab: http://io2015codelabs.appspot.com/codelabs/voice-interaction#1 - https://developers.google.com/voice-actions/interaction/ - https://developers.google.com/voice-actions/interaction/voice-interactions
  13. Optimize Content for Assistant - aka Now on Tap -

    since 6.0 Marshmallow - show Deeplinks to relevant apps that are contextually relevant - Some things to look after as a ‘source’ app
  14. Sharing content with the assistant - Follow accessibility best practices

    and you will be fine - Possibility to share additional content (e.g. name of the currently playing album) - Some listeners to register for providing context
  15. Sharing content with the assistant Example: Providing music app data

    @Override public void onProvideAssistContent(AssistContent assistContent) { super.onProvideAssistContent(assistContent); String structuredJson = new JSONObject() .put("@type", "MusicRecording") .put("@id", "https://example.com/music/recording") .put("name", "Album Title") .toString(); assistContent.setStructuredData(structuredJson); }
  16. Sharing content - Destination app - Assistant takes advantage of

    deeplinking - FireBase App Indexing also helps with being linked - You can also implement your own Assistant
  17. Sharing content with the assistant More information: - https://developer.android.com/training/articles/assistant.html -

    https://developer.android.com/guide/topics/ui/accessibility/apps.html - https://developer.android.com/reference/android/app/Activity.html#onProvideAssistContent(a ndroid.app.assist.AssistContent)
  18. Google Assistant API 1. Design a. Conversation basics b. Crafting

    a conversation c. Best practices 2. Develop a. Components of Conversation Action b. Conversation API c. Actions SDK d. API.AI 3. Example Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
  19. Design - Conversation basics - Problem of recognizing spoken input

    - mostly solved - New challenge: Building a user experience that is similar to a natural human conversation - Turn Taking - Threading - Leveraging the inherent efficiency of speech (read between lines) - Anticipating variable user behaviour
  20. Design - Crafting a conversation - Pick the right use

    cases - Create a persona - Write dialogs - Test it out - Build and iterate
  21. Design - Best practices - Be cooperative… Like your users

    - Quality (only true things) - Quantity (don’t say too much or too less) - Relevance - Manner (be brief, to the point, no ambiguity
  22. Design - Best practices - Unlocking the Power of Spoken

    Language - Communicate what was understood - Offer examples that show what’s possible - Avoid stating the obvious - Give users credit
  23. Develop - Components of Conversation Action - Conversation Actions help

    you fulfill user requests - Request → Process Request → Determine best action → Invoke Conversation Action if relevant
  24. Develop - Conversation API - Conversation API defines a request

    and response format to communicate with Google Assistant
  25. Develop - Action SDK - Helper to parse requests and

    construct responses - Comes in a NodeJS client library
  26. - NodeJS client library - This library provides convenience methods

    to parse requests and construct responses that adhere to the Conversation API. You use this library in your fulfillment code. - Web Simulator - a web-based, virtual Google Home device to simulate your actions on - gactions CLI - a command-line interface to test actions and deploy action packages - Action Package definition - A JSON-based definition that describes a group of actions, how they are invoked, what fulfillment endpoints to call, and other important metadata. You provide this file, along with other assets, when you submit your action for approval. Develop - Action SDK - What’s inside?
  27. Develop - Action SDK - API.AI - Action SDK has

    everything we need - Better developer experience - Makes building actions easier - API.AI NLU - GUI Interface - Conversation building features
  28. Develop - API.AI - Example - API.AI Agent → Conversation

    Action - Meetup Helper Agent - Intents → Welcome Intent /Fallback Intent /Reco Intent - Entities (values) - Prebuilt System entities (time, number, address) - Developer entities
  29. Develop - API.AI - Example - Integrations → Google Assistant

    - Training Section → Teach your assistant - Fulfillment → Send the responses to your service via webhook - Deployment of your assistant - https://developers.google.com/actions/develop/apiai/