Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Actions with Google
Search
Bryan Herbst
February 04, 2017
Programming
0
1.2k
Actions with Google
Learn how to create custom actions for the Google Assistant
Bryan Herbst
February 04, 2017
Tweet
Share
More Decks by Bryan Herbst
See All by Bryan Herbst
Semantics in Jetpack Compose
bherbst
0
8.8k
Kotlin Multiplatform
bherbst
0
630
Android App Architecture & Navigation
bherbst
0
1.9k
What's in the Android Toolbox
bherbst
0
430
Gradle Deep Dive
bherbst
2
1.7k
Android Transitions
bherbst
0
350
Databinding in Android
bherbst
0
370
Activities Vs. Fragments
bherbst
0
290
Android Transitions
bherbst
0
53
Other Decks in Programming
See All in Programming
仮想ファイルシステムを導入して開発環境のストレージ課題を解消する
segadevtech
2
490
GraphQLの魅力を引き出すAndroidクライアント実装
morux2
3
280
Our Websites Need a Lifestyle Change, Not a Diet
ryantownsend
0
120
Hono・Prisma・AWSでGeoなAPI開発
nokonoko1203
5
650
Appleの新しいプライバシー要件対応: ノーコードアプリ プラットフォームの実践事例
nao_randd
1
590
Kotlin 2.0が与えるAndroid開発の進化
masayukisuda
1
250
ESLint Rule により事業, 技術ドメインに沿った制約と誓約を敷衍させるアプローチのすゝめ
shinyaigeek
1
2.9k
開発を加速する共有Swift Package実践
elmetal
PRO
0
380
マイグレーションコード自作して File-Based Routing に自動移行!! ~250 ページの歴史的経緯を添えて~
cut0
1
250
rbs-inlineを導入してYARDからRBSに移行する
euglena1215
1
230
Prolog入門
qnighy
3
880
null or undefined
susisu
22
6.1k
Featured
See All Featured
Six Lessons from altMBA
skipperchong
26
3.3k
In The Pink: A Labor of Love
frogandcode
139
22k
Designing Experiences People Love
moore
138
23k
VelocityConf: Rendering Performance Case Studies
addyosmani
322
23k
The Illustrated Children's Guide to Kubernetes
chrisshort
47
48k
5 minutes of I Can Smell Your CMS
philhawksworth
201
19k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
45
4.8k
Music & Morning Musume
bryan
46
6k
Building Your Own Lightsaber
phodgson
101
6k
4 Signs Your Business is Dying
shpigford
179
21k
Learning to Love Humans: Emotional Interface Design
aarron
270
40k
For a Future-Friendly Web
brad_frost
174
9.3k
Transcript
Actions with Google Bryan Herbst Android Engineer @ Target
The Google Assistant “Ok Google” Allo Google Home Android Auto
Android Wear
The Google Assistant Google Home
Design principles Voice interaction is different
What actions can I take?
What actions can I take?
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. !
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
“Guide me downtown.”
“Guide me downtown.” Downtown where? New York? Minneapolis?
“Guide me downtown.” Driving or walking? Starting from where?
Speech uses shortcuts Filling in the blanks requires context.
Go scriptless You are (not) a robot.
Freedom of speech Say “help” for more options
Freedom of speech Say “help” for more options
Anticipate errors Shortcuts can cause ambiguity
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “Yes.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
Anticipate errors Shortcuts can cause ambiguity
Be Flexible Shortcuts can cause ambiguity
Types of actions Conversation and direct
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
Conversation Actions " Less ambiguity # Limited trigger
Making Conversation Actions
Conversation Actions Format $ Your Action Input
Conversation Actions Format $ Your Action Input JSON Request
Conversation Actions Format $ Your Action Input JSON Request JSON
Response
Conversation Actions Format $ Your Action Input Action Response JSON
Request JSON Response
Conversation Actions Format Your Action JSON Request JSON Response Conversation
API
Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
Invocation Dialogs Fulfillment “Ok, where would you like to go?”
“Paris.”
Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
Invocation - name Unique
Invocation - name Unique | Easy to say
Name restrictions Two words (with exceptions)
Name restrictions Two words (with exceptions) | Avoid reserved words
Trigger Phrases • Let me talk to… • I want
to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
Deep links “Talk to service about topic”
Not deep links “Talk to service.”
Conversations You need to end them
API.AI Actions SDK
API.AI Actions SDK GUI
API.AI Actions SDK GUI Natural Language Understanding
API.AI Actions SDK GUI Natural Language Understanding Integrations
API.AI Actions SDK It’s all on you GUI Natural Language
Understanding Integrations
None
$ API.AI Agent Entities Actions
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Welcome Intent Talk to Clocky
Fallback Intent Tell Clocky to order me a pizza
Fallback Intent Tell Clocky to order me a pizza I
don’t know how to help with that.
Action Capture parameters
Action Capture parameters | Prompt for parameters
System Entitities • Number • Name • Location • Date/time
• Airport • And so much more!
Custom Entities • Food menu • Team names • Fantasy
names
Response These are logic-less
Demo time
Advanced Fulfillment For everything else, there’s webhooks
$ (Your action) Input Action Response JSON Request JSON Response
$ Your server
$ Your server API.AI Webhook
$ Your server API.AI Webhook (also JSON)
Hosting Host anywhere
Hosting Host anywhere | Node.js recommended
Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =
express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I
will set an alarm for" + time); }
SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response
uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
Context Context allows for natural conversations
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. !
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
Interaction 1 Interaction 2 Context Output Input
Output Context Contains all parameters from the action
Input Context Intents only trigger if context exists
Input Context Intents only trigger if context exists Allows for
branching flows
Context - Lifetime Duration (in requests)
Context - Lifetime Duration (in requests) | Ends with conversation
Demo time
Publishing
Actions Directory
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review