Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Actions with Google
Search
Bryan Herbst
February 04, 2017
Programming
0
1.3k
Actions with Google
Learn how to create custom actions for the Google Assistant
Bryan Herbst
February 04, 2017
Tweet
Share
More Decks by Bryan Herbst
See All by Bryan Herbst
Semantics in Jetpack Compose
bherbst
0
11k
Kotlin Multiplatform
bherbst
0
690
Android App Architecture & Navigation
bherbst
0
2k
What's in the Android Toolbox
bherbst
0
470
Gradle Deep Dive
bherbst
2
1.8k
Android Transitions
bherbst
0
400
Databinding in Android
bherbst
0
400
Activities Vs. Fragments
bherbst
0
320
Android Transitions
bherbst
0
56
Other Decks in Programming
See All in Programming
為你自己學 Python
eddie
0
510
Внедряем бюджетирование, или Как сделать хорошо?
lamodatech
0
920
.NETでOBS Studio操作してみたけど…… / Operating OBS Studio by .NET
skasweb
0
120
Go の GC の不得意な部分を克服したい
taiyow
3
1k
Оптимизируем производительность блока Казначейство
lamodatech
0
930
ゼロからの、レトロゲームエンジンの作り方
tokujiros
0
480
Findy Team+ Awardを受賞したかった!ベストプラクティス応募内容をふりかえり、開発生産性向上もふりかえる / Findy Team Plus Award BestPractice and DPE Retrospective 2024
honyanya
0
140
生成AIでGitHubソースコード取得して仕様書を作成
shukob
0
620
競技プログラミングへのお誘い@阪大BOOSTセミナー
kotamanegi
0
400
PHPUnitしか使ってこなかった 一般PHPerがPestに乗り換えた実録
mashirou1234
0
410
Асинхронность неизбежна: как мы проектировали сервис уведомлений
lamodatech
0
1.3k
Beyond ORM
77web
11
1.5k
Featured
See All Featured
The Language of Interfaces
destraynor
155
24k
Product Roadmaps are Hard
iamctodd
PRO
50
11k
Designing for Performance
lara
604
68k
Typedesign – Prime Four
hannesfritz
40
2.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
45
2.3k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.2k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
28
4.5k
Music & Morning Musume
bryan
46
6.3k
YesSQL, Process and Tooling at Scale
rocio
170
14k
Designing for humans not robots
tammielis
250
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
560
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
Actions with Google Bryan Herbst Android Engineer @ Target
The Google Assistant “Ok Google” Allo Google Home Android Auto
Android Wear
The Google Assistant Google Home
Design principles Voice interaction is different
What actions can I take?
What actions can I take?
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. !
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
“Guide me downtown.”
“Guide me downtown.” Downtown where? New York? Minneapolis?
“Guide me downtown.” Driving or walking? Starting from where?
Speech uses shortcuts Filling in the blanks requires context.
Go scriptless You are (not) a robot.
Freedom of speech Say “help” for more options
Freedom of speech Say “help” for more options
Anticipate errors Shortcuts can cause ambiguity
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “Yes.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
Anticipate errors Shortcuts can cause ambiguity
Be Flexible Shortcuts can cause ambiguity
Types of actions Conversation and direct
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
Conversation Actions " Less ambiguity # Limited trigger
Making Conversation Actions
Conversation Actions Format $ Your Action Input
Conversation Actions Format $ Your Action Input JSON Request
Conversation Actions Format $ Your Action Input JSON Request JSON
Response
Conversation Actions Format $ Your Action Input Action Response JSON
Request JSON Response
Conversation Actions Format Your Action JSON Request JSON Response Conversation
API
Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
Invocation Dialogs Fulfillment “Ok, where would you like to go?”
“Paris.”
Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
Invocation - name Unique
Invocation - name Unique | Easy to say
Name restrictions Two words (with exceptions)
Name restrictions Two words (with exceptions) | Avoid reserved words
Trigger Phrases • Let me talk to… • I want
to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
Deep links “Talk to service about topic”
Not deep links “Talk to service.”
Conversations You need to end them
API.AI Actions SDK
API.AI Actions SDK GUI
API.AI Actions SDK GUI Natural Language Understanding
API.AI Actions SDK GUI Natural Language Understanding Integrations
API.AI Actions SDK It’s all on you GUI Natural Language
Understanding Integrations
None
$ API.AI Agent Entities Actions
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Welcome Intent Talk to Clocky
Fallback Intent Tell Clocky to order me a pizza
Fallback Intent Tell Clocky to order me a pizza I
don’t know how to help with that.
Action Capture parameters
Action Capture parameters | Prompt for parameters
System Entitities • Number • Name • Location • Date/time
• Airport • And so much more!
Custom Entities • Food menu • Team names • Fantasy
names
Response These are logic-less
Demo time
Advanced Fulfillment For everything else, there’s webhooks
$ (Your action) Input Action Response JSON Request JSON Response
$ Your server
$ Your server API.AI Webhook
$ Your server API.AI Webhook (also JSON)
Hosting Host anywhere
Hosting Host anywhere | Node.js recommended
Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =
express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I
will set an alarm for" + time); }
SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response
uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
Context Context allows for natural conversations
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. !
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
Interaction 1 Interaction 2 Context Output Input
Output Context Contains all parameters from the action
Input Context Intents only trigger if context exists
Input Context Intents only trigger if context exists Allows for
branching flows
Context - Lifetime Duration (in requests)
Context - Lifetime Duration (in requests) | Ends with conversation
Demo time
Publishing
Actions Directory
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review