Google Assistant API - Agenda 1. Introduction 2. Google Assistant 3. Build your own bot a. Voice Interaction API b. Optimizing Content for the Assistant c. Google Assistant API using API.AI d. Live Demo - A smart Meetup Notifier Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
Google Assistant - Presented by Google at Google I/O 2016 - 20% of queries are voice queries in the US - Using natural language processing engine - Contextual awareness (place, previous queries, time)
Build your own Bot - Voice Interaction API - Voice API was introduced in 2015 - Enables conversations where apps can ask back questions if more information is required by an app - Confirm actions (Place phone call, send SMS)
Build your own Bot - Voice Interaction API - Voice Interaction is special kind of Android Activity - VoiceActivity → Complete an action - NormalActivity → Start an action, be completed by touch - Example: Android DIAL intent
Build your own Bot - Voice Interaction API Handle the voice interaction - The app should decide what’s an appropriate action with the given input - Ask for more input - Confirm an upcoming action - User wants to book a taxi, with all necessary information given - Needs to confirm the time and price - After confirming the action can be finished and the request can be send
Build your own Bot - Voice Interaction API Approving interactions without confirmation - There might be cases where it is not necessary to ask for confirmation - Good examples might be: Opening a website (if no more information is needed) Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { // Interaction started by the users voice doAction(); } finish(); }
Build your own Bot - Voice Interaction API Finishing the Voice Interaction - If your interaction is completed call finish() - If you want to keep your users in the app, start a new activity with Intent.FLAG_ACTIVITY_NEW_TASK Code: class MyVoiceActivity extends Activity { @Override public void onResume() { if (isVoiceInteractionRoot()) { doAction(); } // Start my main non-voice Activity Intent intent = new Intent(this, MyMainActivity.class); intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK); startActivity(intent); finish(); } }
Build your own Bot - Voice Interaction API More information: - Codelab: http://io2015codelabs.appspot.com/codelabs/voice-interaction#1 - https://developers.google.com/voice-actions/interaction/ - https://developers.google.com/voice-actions/interaction/voice-interactions
Optimize Content for Assistant - aka Now on Tap - since 6.0 Marshmallow - show Deeplinks to relevant apps that are contextually relevant - Some things to look after as a ‘source’ app
Sharing content with the assistant - Follow accessibility best practices and you will be fine - Possibility to share additional content (e.g. name of the currently playing album) - Some listeners to register for providing context
Sharing content - Destination app - Assistant takes advantage of deeplinking - FireBase App Indexing also helps with being linked - You can also implement your own Assistant
Sharing content with the assistant More information: - https://developer.android.com/training/articles/assistant.html - https://developer.android.com/guide/topics/ui/accessibility/apps.html - https://developer.android.com/reference/android/app/Activity.html#onProvideAssistContent(a ndroid.app.assist.AssistContent)
Google Assistant API 1. Design a. Conversation basics b. Crafting a conversation c. Best practices 2. Develop a. Components of Conversation Action b. Conversation API c. Actions SDK d. API.AI 3. Example Daniel Hartwich - Android Developer - XING AG - GDG Android Meetup 02.02.2017
Design - Conversation basics - Problem of recognizing spoken input - mostly solved - New challenge: Building a user experience that is similar to a natural human conversation - Turn Taking - Threading - Leveraging the inherent efficiency of speech (read between lines) - Anticipating variable user behaviour
Design - Best practices - Be cooperative… Like your users - Quality (only true things) - Quantity (don’t say too much or too less) - Relevance - Manner (be brief, to the point, no ambiguity
Design - Best practices - Unlocking the Power of Spoken Language - Communicate what was understood - Offer examples that show what’s possible - Avoid stating the obvious - Give users credit
Develop - Components of Conversation Action - Conversation Actions help you fulfill user requests - Request → Process Request → Determine best action → Invoke Conversation Action if relevant
- NodeJS client library - This library provides convenience methods to parse requests and construct responses that adhere to the Conversation API. You use this library in your fulfillment code. - Web Simulator - a web-based, virtual Google Home device to simulate your actions on - gactions CLI - a command-line interface to test actions and deploy action packages - Action Package definition - A JSON-based definition that describes a group of actions, how they are invoked, what fulfillment endpoints to call, and other important metadata. You provide this file, along with other assets, when you submit your action for approval. Develop - Action SDK - What’s inside?
Develop - Action SDK - API.AI - Action SDK has everything we need - Better developer experience - Makes building actions easier - API.AI NLU - GUI Interface - Conversation building features
Develop - API.AI - Example - Integrations → Google Assistant - Training Section → Teach your assistant - Fulfillment → Send the responses to your service via webhook - Deployment of your assistant - https://developers.google.com/actions/develop/apiai/
dhartwich1991
korodroid
orgachem
aiji42
eliasnogueira
safie_recruit
odasho
mii3king
iotengineer22
kentosuzuki
seike460
nenonaninu
miyakemito
addyosmani
philhawksworth
lara
jakevdp
jacobian
destraynor
jnunemaker
samanthasiow
morganepeng
jrom
zenorocha
bluesmoon