Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google Assistant API - Ok Google... Do something

Google Assistant API - Ok Google... Do something

These are the slides for my talk at the GDG Android Hamburg Meetup from February 2017


Daniel Hartwich

February 02, 2017


  1. Google Assistant API Ok Google, … Do something! Daniel Hartwich

    - Android Developer - XING AG - GDG Hamburg Android Meetup 02.02.2017
  2. Google Assistant API - Agenda 1. Introduction 2. Google Assistant

    3. Build your own bot a. Voice Interaction API b. Optimizing Content for the Assistant c. Google Assistant API using API.AI d. Live Demo - A smart Meetup Notifier Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
  3. None
  4. Google Assistant - Presented by Google at Google I/O 2016

    - 20% of queries are voice queries in the US - Using natural language processing engine - Contextual awareness (place, previous queries, time)
  5. Google Assistant - Should be there for the users when

    they need it
  6. Google Assistant - Google’s Definition Ask it anything.

  7. Google Assistant - Google’s Definition Tell it to do things.

  8. Google Assistant - Google’s Definition All your devices. One Assistant

  9. Google Assistant - Google’s Definition Safe, secure and in your

  10. Google Assistant - Google’s Definition Your Assistant works with your

    favorite stuff, too
  11. Build your own Bot Daniel Hartwich - Android Developer -

    XING AG - Android Developer Meetup 02.02.2017
  12. Voice Interaction API - FlowChart

  13. Build your own Bot - Voice Interaction API - Voice

    API was introduced in 2015 - Enables conversations where apps can ask back questions if more information is required by an app - Confirm actions (Place phone call, send SMS)
  14. Build your own Bot - Voice Interaction API - Voice

    Interaction is special kind of Android Activity - VoiceActivity → Complete an action - NormalActivity → Start an action, be completed by touch - Example: Android DIAL intent
  15. Build your own Bot - Voice Interaction API Declare Voice

    Intent Filter in Manifest <activity android:name="org.example.MyVoiceActivity"> <intent-filter> <action android:name="org.example.MY_ACTION_INTENT" /> <category android:name="android.intent.category.DEFAULT" /> <category android:name="android.intent.category.VOICE" /> </intent-filter> </activity>
  16. Build your own Bot - Voice Interaction API Handle the

    voice interaction - The app should decide what’s an appropriate action with the given input - Ask for more input - Confirm an upcoming action - User wants to book a taxi, with all necessary information given - Needs to confirm the time and price - After confirming the action can be finished and the request can be send
  17. Build your own Bot - Voice Interaction API class MyVoiceActivity

    extends Activity { class Confirm extends VoiceInteractor.ConfirmationRequest { public Confirm(String ttsPrompt, String visualPrompt) { VoiceInteractor.Prompt prompt = new VoiceInteractor.Prompt( new String[] {ttsPrompt}, visualPrompt); super(prompt, null); } @Override public void onConfirmationResult( boolean confirmed, Bundle null) { if (confirmed) { doAction(); } finish(); } }; @Override public void onResume() { if (isVoiceInteraction()) { String ttsPrompt = getConfirmationTts(); String visualPrompt = getConfirmationDisplayText(); getVoiceInteractor().sendRequest(new Confirm(ttsPrompt, visualPrompt)); } else { finish(); } } }
  18. Build your own Bot - Voice Interaction API Approving interactions

    without confirmation - There might be cases where it is not necessary to ask for confirmation - Good examples might be: Opening a website (if no more information is needed) Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { // Interaction started by the users voice doAction(); } finish(); }
  19. Build your own Bot - Voice Interaction API Finishing the

    Voice Interaction - If your interaction is completed call finish() - If you want to keep your users in the app, start a new activity with Intent.FLAG_ACTIVITY_NEW_TASK Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { doAction(); } // Start my main non-voice Activity Intent intent = new Intent(this, MyMainActivity.class); intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK); startActivity(intent); finish(); } }
  20. Build your own Bot - Voice Interaction API More information:

    - Codelab: http://io2015codelabs.appspot.com/codelabs/voice-interaction#1 - https://developers.google.com/voice-actions/interaction/ - https://developers.google.com/voice-actions/interaction/voice-interactions
  21. Optimize Content for Assistant - aka Now on Tap -

  22. Optimize Content for Assistant - aka Now on Tap -

    since 6.0 Marshmallow - show Deeplinks to relevant apps that are contextually relevant - Some things to look after as a ‘source’ app
  23. Sharing content with the assistant - Follow accessibility best practices

    and you will be fine - Possibility to share additional content (e.g. name of the currently playing album) - Some listeners to register for providing context
  24. Sharing content with the assistant

  25. Sharing content with the assistant Example: Providing music app data

    @Override public void onProvideAssistContent(AssistContent assistContent) { super.onProvideAssistContent(assistContent); String structuredJson = new JSONObject() .put("@type", "MusicRecording") .put("@id", "https://example.com/music/recording") .put("name", "Album Title") .toString(); assistContent.setStructuredData(structuredJson); }
  26. Sharing content with the assistant - Exclude sensitive infos using

  27. Sharing content - Destination app - Assistant takes advantage of

    deeplinking - FireBase App Indexing also helps with being linked - You can also implement your own Assistant
  28. Sharing content with the assistant More information: - https://developer.android.com/training/articles/assistant.html -

    https://developer.android.com/guide/topics/ui/accessibility/apps.html - https://developer.android.com/reference/android/app/Activity.html#onProvideAssistContent(a ndroid.app.assist.AssistContent)
  29. Google Assistant API using API.AI

  30. Google Assistant API 1. Design a. Conversation basics b. Crafting

    a conversation c. Best practices 2. Develop a. Components of Conversation Action b. Conversation API c. Actions SDK d. API.AI 3. Example Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
  31. Design - Conversation basics - Problem of recognizing spoken input

    - mostly solved - New challenge: Building a user experience that is similar to a natural human conversation - Turn Taking - Threading - Leveraging the inherent efficiency of speech (read between lines) - Anticipating variable user behaviour
  32. Design - Crafting a conversation - Pick the right use

    cases - Create a persona - Write dialogs - Test it out - Build and iterate
  33. Design - Best practices - Be cooperative… Like your users

    - Quality (only true things) - Quantity (don’t say too much or too less) - Relevance - Manner (be brief, to the point, no ambiguity
  34. Design - Best practices - Unlocking the Power of Spoken

    Language - Communicate what was understood - Offer examples that show what’s possible - Avoid stating the obvious - Give users credit
  35. Develop - Components of Conversation Action - Conversation Actions help

    you fulfill user requests - Request → Process Request → Determine best action → Invoke Conversation Action if relevant
  36. Develop - Components of Conversation Action

  37. Develop - Conversation API - Conversation API defines a request

    and response format to communicate with Google Assistant
  38. Develop - Action SDK - Helper to parse requests and

    construct responses - Comes in a NodeJS client library
  39. - NodeJS client library - This library provides convenience methods

    to parse requests and construct responses that adhere to the Conversation API. You use this library in your fulfillment code. - Web Simulator - a web-based, virtual Google Home device to simulate your actions on - gactions CLI - a command-line interface to test actions and deploy action packages - Action Package definition - A JSON-based definition that describes a group of actions, how they are invoked, what fulfillment endpoints to call, and other important metadata. You provide this file, along with other assets, when you submit your action for approval. Develop - Action SDK - What’s inside?
  40. Develop - Action SDK - API.AI - Action SDK has

    everything we need - Better developer experience - Makes building actions easier - API.AI NLU - GUI Interface - Conversation building features
  41. Develop - Action SDK - API.AI

  42. Develop - API.AI - Example

  43. Develop - API.AI - Example - API.AI Agent → Conversation

    Action - Meetup Helper Agent - Intents → Welcome Intent /Fallback Intent /Reco Intent - Entities (values) - Prebuilt System entities (time, number, address) - Developer entities
  44. Develop - API.AI - Example - Integrations → Google Assistant

    - Training Section → Teach your assistant - Fulfillment → Send the responses to your service via webhook - Deployment of your assistant - https://developers.google.com/actions/develop/apiai/
  45. Develop - API.AI - Example

  46. Questions? goo.gl/slides/cdaf73