Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Actions with Google
Search
Bryan Herbst
February 04, 2017
Programming
0
1.4k
Actions with Google
Learn how to create custom actions for the Google Assistant
Bryan Herbst
February 04, 2017
Tweet
Share
More Decks by Bryan Herbst
See All by Bryan Herbst
Semantics in Jetpack Compose
bherbst
0
12k
Kotlin Multiplatform
bherbst
0
850
Android App Architecture & Navigation
bherbst
0
2k
What's in the Android Toolbox
bherbst
0
520
Gradle Deep Dive
bherbst
2
2k
Android Transitions
bherbst
0
500
Databinding in Android
bherbst
0
450
Activities Vs. Fragments
bherbst
0
350
Android Transitions
bherbst
0
62
Other Decks in Programming
See All in Programming
Flutterで分数(Fraction)を表示する方法
koukimiura
0
140
Writing Better Go: Lessons from 10 Code Reviews
konradreiche
3
6.1k
その面倒な作業、「Dart」にやらせませんか? Flutter開発者のための業務効率化
yordgenome03
1
140
理論と実務のギャップを超える
eycjur
0
170
「ちょっと古いから」って避けてた技術書、今だからこそ読もう
mottyzzz
12
7.1k
GC25 Recap: The Code You Reviewed is Not the Code You Built / #newt_gophercon_tour
mazrean
0
110
技術的負債の正体を知って向き合う
irof
0
230
Developer Joy - The New Paradigm
hollycummins
1
350
ALL CODE BASE ARE BELONG TO STUDY
uzulla
27
6.6k
テーブル定義書の構造化抽出して、生成AIでDWH分析を試してみた / devio2025tokyo
kasacchiful
0
270
バッチ処理を「状態の記録」から「事実の記録」へ
panda728
PRO
0
190
CSC305 Lecture 09
javiergs
PRO
0
310
Featured
See All Featured
KATA
mclloyd
PRO
32
15k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
36
6.1k
Faster Mobile Websites
deanohume
310
31k
Writing Fast Ruby
sferik
629
62k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Code Reviewing Like a Champion
maltzj
526
40k
The Cost Of JavaScript in 2023
addyosmani
55
9.1k
The Pragmatic Product Professional
lauravandoore
36
7k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
10
880
YesSQL, Process and Tooling at Scale
rocio
173
15k
RailsConf 2023
tenderlove
30
1.3k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Transcript
Actions with Google Bryan Herbst Android Engineer @ Target
The Google Assistant “Ok Google” Allo Google Home Android Auto
Android Wear
The Google Assistant Google Home
Design principles Voice interaction is different
What actions can I take?
What actions can I take?
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. !
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
“Guide me downtown.”
“Guide me downtown.” Downtown where? New York? Minneapolis?
“Guide me downtown.” Driving or walking? Starting from where?
Speech uses shortcuts Filling in the blanks requires context.
Go scriptless You are (not) a robot.
Freedom of speech Say “help” for more options
Freedom of speech Say “help” for more options
Anticipate errors Shortcuts can cause ambiguity
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “Yes.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
Anticipate errors Shortcuts can cause ambiguity
Be Flexible Shortcuts can cause ambiguity
Types of actions Conversation and direct
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
Conversation Actions " Less ambiguity # Limited trigger
Making Conversation Actions
Conversation Actions Format $ Your Action Input
Conversation Actions Format $ Your Action Input JSON Request
Conversation Actions Format $ Your Action Input JSON Request JSON
Response
Conversation Actions Format $ Your Action Input Action Response JSON
Request JSON Response
Conversation Actions Format Your Action JSON Request JSON Response Conversation
API
Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
Invocation Dialogs Fulfillment “Ok, where would you like to go?”
“Paris.”
Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
Invocation - name Unique
Invocation - name Unique | Easy to say
Name restrictions Two words (with exceptions)
Name restrictions Two words (with exceptions) | Avoid reserved words
Trigger Phrases • Let me talk to… • I want
to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
Deep links “Talk to service about topic”
Not deep links “Talk to service.”
Conversations You need to end them
API.AI Actions SDK
API.AI Actions SDK GUI
API.AI Actions SDK GUI Natural Language Understanding
API.AI Actions SDK GUI Natural Language Understanding Integrations
API.AI Actions SDK It’s all on you GUI Natural Language
Understanding Integrations
None
$ API.AI Agent Entities Actions
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Welcome Intent Talk to Clocky
Fallback Intent Tell Clocky to order me a pizza
Fallback Intent Tell Clocky to order me a pizza I
don’t know how to help with that.
Action Capture parameters
Action Capture parameters | Prompt for parameters
System Entitities • Number • Name • Location • Date/time
• Airport • And so much more!
Custom Entities • Food menu • Team names • Fantasy
names
Response These are logic-less
Demo time
Advanced Fulfillment For everything else, there’s webhooks
$ (Your action) Input Action Response JSON Request JSON Response
$ Your server
$ Your server API.AI Webhook
$ Your server API.AI Webhook (also JSON)
Hosting Host anywhere
Hosting Host anywhere | Node.js recommended
Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =
express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I
will set an alarm for" + time); }
SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response
uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
Context Context allows for natural conversations
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. !
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
Interaction 1 Interaction 2 Context Output Input
Output Context Contains all parameters from the action
Input Context Intents only trigger if context exists
Input Context Intents only trigger if context exists Allows for
branching flows
Context - Lifetime Duration (in requests)
Context - Lifetime Duration (in requests) | Ends with conversation
Demo time
Publishing
Actions Directory
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review