Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Actions with Google
Search
Bryan Herbst
February 04, 2017
Programming
0
1.2k
Actions with Google
Learn how to create custom actions for the Google Assistant
Bryan Herbst
February 04, 2017
Tweet
Share
More Decks by Bryan Herbst
See All by Bryan Herbst
Semantics in Jetpack Compose
bherbst
0
8.1k
Kotlin Multiplatform
bherbst
0
600
Android App Architecture & Navigation
bherbst
0
1.9k
What's in the Android Toolbox
bherbst
0
420
Gradle Deep Dive
bherbst
2
1.7k
Android Transitions
bherbst
0
340
Databinding in Android
bherbst
0
360
Activities Vs. Fragments
bherbst
0
280
Android Transitions
bherbst
0
52
Other Decks in Programming
See All in Programming
CSC307 Lecture 09
javiergs
PRO
1
500
CSC307 Lecture 06
javiergs
PRO
0
360
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
Exploring the Gradually Lost Technical Skills in the Cloud Native Era
hwchiu
2
3.9k
SDCon2024: Enabling DevOps and Team Topologies thru architecture: architecting for fast flow
cer
PRO
0
780
【Go言語】golangci-lintの使い方
tomo1227
0
280
今こそ始める、CDKコンストラクトライブラリ開発 ― 入門から実践まで
tmokmss
1
930
「2024年版 Kotlin サーバーサイドプログラミング実践開発」の補講 〜O/Rマッパー編〜
n_takehata
2
260
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
370
AWSでゲームサーバーを運用! Amazon GameLiftのお話
iriikeita
0
200
12年前の『型システム入門』翻訳の思い出話
mame
11
1.2k
HMSコンペ 11th Solution (team : kansai-kaggler)
t88
1
680
Featured
See All Featured
Atom: Resistance is Futile
akmur
261
25k
Testing 201, or: Great Expectations
jmmastey
33
6.9k
How to Think Like a Performance Engineer
csswizardry
4
590
GitHub's CSS Performance
jonrohan
1026
450k
Optimizing for Happiness
mojombo
373
69k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.9k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
26
2.1k
Stop Working from a Prison Cell
hatefulcrawdad
266
20k
Statistics for Hackers
jakevdp
792
220k
Designing on Purpose - Digital PM Summit 2013
jponch
113
6.6k
How to name files
jennybc
67
96k
5 minutes of I Can Smell Your CMS
philhawksworth
200
19k
Transcript
Actions with Google Bryan Herbst Android Engineer @ Target
The Google Assistant “Ok Google” Allo Google Home Android Auto
Android Wear
The Google Assistant Google Home
Design principles Voice interaction is different
What actions can I take?
What actions can I take?
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. !
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
“Guide me downtown.”
“Guide me downtown.” Downtown where? New York? Minneapolis?
“Guide me downtown.” Driving or walking? Starting from where?
Speech uses shortcuts Filling in the blanks requires context.
Go scriptless You are (not) a robot.
Freedom of speech Say “help” for more options
Freedom of speech Say “help” for more options
Anticipate errors Shortcuts can cause ambiguity
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “Yes.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
Anticipate errors Shortcuts can cause ambiguity
Be Flexible Shortcuts can cause ambiguity
Types of actions Conversation and direct
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
Conversation Actions " Less ambiguity # Limited trigger
Making Conversation Actions
Conversation Actions Format $ Your Action Input
Conversation Actions Format $ Your Action Input JSON Request
Conversation Actions Format $ Your Action Input JSON Request JSON
Response
Conversation Actions Format $ Your Action Input Action Response JSON
Request JSON Response
Conversation Actions Format Your Action JSON Request JSON Response Conversation
API
Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
Invocation Dialogs Fulfillment “Ok, where would you like to go?”
“Paris.”
Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
Invocation - name Unique
Invocation - name Unique | Easy to say
Name restrictions Two words (with exceptions)
Name restrictions Two words (with exceptions) | Avoid reserved words
Trigger Phrases • Let me talk to… • I want
to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
Deep links “Talk to service about topic”
Not deep links “Talk to service.”
Conversations You need to end them
API.AI Actions SDK
API.AI Actions SDK GUI
API.AI Actions SDK GUI Natural Language Understanding
API.AI Actions SDK GUI Natural Language Understanding Integrations
API.AI Actions SDK It’s all on you GUI Natural Language
Understanding Integrations
None
$ API.AI Agent Entities Actions
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Welcome Intent Talk to Clocky
Fallback Intent Tell Clocky to order me a pizza
Fallback Intent Tell Clocky to order me a pizza I
don’t know how to help with that.
Action Capture parameters
Action Capture parameters | Prompt for parameters
System Entitities • Number • Name • Location • Date/time
• Airport • And so much more!
Custom Entities • Food menu • Team names • Fantasy
names
Response These are logic-less
Demo time
Advanced Fulfillment For everything else, there’s webhooks
$ (Your action) Input Action Response JSON Request JSON Response
$ Your server
$ Your server API.AI Webhook
$ Your server API.AI Webhook (also JSON)
Hosting Host anywhere
Hosting Host anywhere | Node.js recommended
Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =
express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I
will set an alarm for" + time); }
SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response
uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
Context Context allows for natural conversations
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. !
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
Interaction 1 Interaction 2 Context Output Input
Output Context Contains all parameters from the action
Input Context Intents only trigger if context exists
Input Context Intents only trigger if context exists Allows for
branching flows
Context - Lifetime Duration (in requests)
Context - Lifetime Duration (in requests) | Ends with conversation
Demo time
Publishing
Actions Directory
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review