Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Actions with Google
Search
Bryan Herbst
February 04, 2017
Programming
0
1.3k
Actions with Google
Learn how to create custom actions for the Google Assistant
Bryan Herbst
February 04, 2017
Tweet
Share
More Decks by Bryan Herbst
See All by Bryan Herbst
Semantics in Jetpack Compose
bherbst
0
11k
Kotlin Multiplatform
bherbst
0
720
Android App Architecture & Navigation
bherbst
0
2k
What's in the Android Toolbox
bherbst
0
480
Gradle Deep Dive
bherbst
2
1.9k
Android Transitions
bherbst
0
430
Databinding in Android
bherbst
0
410
Activities Vs. Fragments
bherbst
0
330
Android Transitions
bherbst
0
57
Other Decks in Programming
See All in Programming
読もう! Android build ドキュメント
andpad
1
190
家族・子育て重視/沖縄在住を維持しながらエンジニアとしてのキャリアをどのように育てていくか?
ug
0
200
いまさら聞けない生成AI入門: 「生成AIを高速キャッチアップ」
soh9834
5
1.2k
Compose Navigation実装の見通しを良くする
hiroaki404
0
120
AHC 044 混合整数計画ソルバー解法
kiri8128
0
260
Denoでフロントエンド開発 2025年春版 / Frontend Development with Deno (Spring 2025)
petamoriken
1
1.2k
PHPer's Guide to Daemon Crafting Taming and Summoning
uzulla
2
530
Cloudflare Pagesのサイトを NotebookLMから読みやすくする Cloudflare Meet-up Tokyo Vol.7
xiombatsg
0
110
Return of the Full-Stack Developer
simas
PRO
1
300
もう一人で悩まない! 個の知見をチームの知見にする3つの習慣と工夫 / Into team knowledge.
honyanya
3
500
Go言語での実装を通して学ぶ、高速なベクトル検索を支えるクラスタリング技術/fukuokago-kmeans
monochromegane
1
120
PHPのガベージコレクションを深掘りしよう
rinchoku
0
220
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
4
430
Bash Introduction
62gerente
611
210k
Practical Orchestrator
shlominoach
186
10k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
134
33k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
GitHub's CSS Performance
jonrohan
1030
460k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
21k
Building Applications with DynamoDB
mza
94
6.3k
Designing for Performance
lara
605
69k
The Cult of Friendly URLs
andyhume
78
6.3k
Faster Mobile Websites
deanohume
306
31k
RailsConf 2023
tenderlove
29
1k
Transcript
Actions with Google Bryan Herbst Android Engineer @ Target
The Google Assistant “Ok Google” Allo Google Home Android Auto
Android Wear
The Google Assistant Google Home
Design principles Voice interaction is different
What actions can I take?
What actions can I take?
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. !
What should I have for dinner? What are you in
the mood for? You can say Italian, American, Mexican, or Chinese. ! Take turns.
“Guide me downtown.”
“Guide me downtown.” Downtown where? New York? Minneapolis?
“Guide me downtown.” Driving or walking? Starting from where?
Speech uses shortcuts Filling in the blanks requires context.
Go scriptless You are (not) a robot.
Freedom of speech Say “help” for more options
Freedom of speech Say “help” for more options
Anticipate errors Shortcuts can cause ambiguity
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “Yes.”
”Buy me a KitKat bar.” ”Okay, ordering a king sized
KitKat bar from Amazon. Is that correct?” “No, make it a regular sized KitKat bar.”
Anticipate errors Shortcuts can cause ambiguity
Be Flexible Shortcuts can cause ambiguity
Types of actions Conversation and direct
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability
Conversation Direct Trigger “Talk to SERVICE about TOPIC” Responsibilities You
handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now
Conversation Direct Trigger “Talk to SERVICE about TOPIC” “Do this
action” Responsibilities You handle interaction and fulfillment Google handles interaction, you handle fulfillment. Availability Now Soon
Conversation Actions " Less ambiguity # Limited trigger
Making Conversation Actions
Conversation Actions Format $ Your Action Input
Conversation Actions Format $ Your Action Input JSON Request
Conversation Actions Format $ Your Action Input JSON Request JSON
Response
Conversation Actions Format $ Your Action Input Action Response JSON
Request JSON Response
Conversation Actions Format Your Action JSON Request JSON Response Conversation
API
Invocation Dialogs Fulfillment “Ask AwesomeFlightApp to book me a flight.”
Invocation Dialogs Fulfillment “Ok, where would you like to go?”
“Paris.”
Invocation Dialogs Fulfillment “Okay, booking a flight to Paris!”
Invocation - name Unique
Invocation - name Unique | Easy to say
Name restrictions Two words (with exceptions)
Name restrictions Two words (with exceptions) | Avoid reserved words
Trigger Phrases • Let me talk to… • I want
to talk to... • Can I talk to... • Talk to... • Ask... • At... • Tell... • Tell me...
Deep links “Talk to service about topic”
Not deep links “Talk to service.”
Conversations You need to end them
API.AI Actions SDK
API.AI Actions SDK GUI
API.AI Actions SDK GUI Natural Language Understanding
API.AI Actions SDK GUI Natural Language Understanding Integrations
API.AI Actions SDK It’s all on you GUI Natural Language
Understanding Integrations
None
$ API.AI Agent Entities Actions
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Intent examples • Set an alarm for 6:00am • Wake
me up at 6:00am • I need an alarm at 6:00am • Set an alarm for March 1st at 6:00pm
Welcome Intent Talk to Clocky
Fallback Intent Tell Clocky to order me a pizza
Fallback Intent Tell Clocky to order me a pizza I
don’t know how to help with that.
Action Capture parameters
Action Capture parameters | Prompt for parameters
System Entitities • Number • Name • Location • Date/time
• Airport • And so much more!
Custom Entities • Food menu • Team names • Fantasy
names
Response These are logic-less
Demo time
Advanced Fulfillment For everything else, there’s webhooks
$ (Your action) Input Action Response JSON Request JSON Response
$ Your server
$ Your server API.AI Webhook
$ Your server API.AI Webhook (also JSON)
Hosting Host anywhere
Hosting Host anywhere | Node.js recommended
Simple webhook server let Assistant = require('actions-on-google').ApiAiAssistant; let app =
express(); app.post('/', function (req, res) { const assistant = new Assistant({request: req, response: res}); } }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Basic webhook let actionMap = new Map(); actionMap.set(”set_alarm", setAlarm); assistant.handleRequest(actionMap);
function setAlarm() { assistant.tell("Hello from the webhook!"); }
Webhook with arguments function withArguments() { let time= assistant.getArgument(”time"); assistant.tell(”I
will set an alarm for" + time); }
SSML- Speech Synthesis Markup Language function withSsml() { assistant.tell("This response
uses” + "<say-as interpret-as='characters'>SSML</say-as>.” + "Isn't it <break> awesome?"); }
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address)
User Identity • ID (resettable!) • Display name • Given
name • Family name • Coarse device location (zip code and city) • Precise device location (lat/lng, address) Requires permission
Context Context allows for natural conversations
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. !
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
What’s the weather like in Minneapolis? It is currently 10°
in Minneapolis. ! Context Location: Minneapolis Temperature: 10°
Interaction 1 Interaction 2 Context Output Input
Output Context Contains all parameters from the action
Input Context Intents only trigger if context exists
Input Context Intents only trigger if context exists Allows for
branching flows
Context - Lifetime Duration (in requests)
Context - Lifetime Duration (in requests) | Ends with conversation
Demo time
Publishing
Actions Directory
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review
Create a Google Cloud Project Register the Action Deploy via
API.AI Wait for review