IoT with the Google Assistant

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Voice UX

Slide 4

Slide 4 text

Voice UX ● Voice UX is special – Discoverability of app – Discoverability of features – Onboarding ● Avoid repetitions – Past behaviour of user

Slide 5

Slide 5 text

Voice UX ● You need to think about your app‘s persona ● Try to keep to user expectations of a dialog – Script it – Practise script with a partner

Slide 6

Slide 6 text

Voice UX ● Some recommendations – provide feedback – elicit answers – Give helpful comments – Repair broken conversations – Provide help – Avoid the usage of the same phrases over and over again

Slide 7

Slide 7 text

Voice UX ● Loads of UX recommendations by Google ● See the explanations about the IO Assistant app

Slide 8

Slide 8 text

Developing

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Assistant SDK ● If you want to add the Assistant to your device ● Enables you to control some aspects of the device

Slide 11

Slide 11 text

Assistant SDK ● Communicate with Google‘s backend using gRPC ● Change the trigger for the assistant – No direct support for a different hotword (use snowboy or pocketSphinx) ● On device support for different traits – Brightness, color... ● Either use a lib (Python) or the service directly ● For proper devices; you must (!) contact Google; it‘s for development only

Slide 12

Slide 12 text

Actions on Google – Smart Home https://en.wikipedia.org/wiki/Philips_Hue

Slide 13

Slide 13 text

Actions on Google – Smart Home ● If you want your users to control your device using the Google Assistant – „Turn on the light in the living room“

Slide 14

Slide 14 text

Actions on Google – Smart Home ● Consists of two parts: – OAuth2 server to authenticate users and issue a token – Some backend to ● Query the state of the device ● Trigger changes to the device ● User adds a device via Google Home app – Here the Oauth flow comes into play ● Digg into sample app – https://github.com/actions-on-google/smart-home-nodejs

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Actions https://assistant.google.com/services/a/uid/0000008011b5e5f2?hl=en-US

Slide 20

Slide 20 text

Actions ● If you want your users to use a voice app ● „Ask MVV, when‘s the next bus due at Hofmannstraße“ ● Use one of those options – Actions SDK – A service like Dialogflow

Slide 21

Slide 21 text

Actions ● If you want your users to use a voice app ● „Ask MVV, when‘s the next bus due at Hofmannstraße“ ● Use Dialogflow

Slide 22

Slide 22 text

Action SDK

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Dialogflow

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

App-Structure

Slide 27

Slide 27 text

Base concepts ● Intents – Predefined and custom ones – Special ones for smart home ● Entities ● Fulfillment ● Actions ● Context

Slide 28

Slide 28 text

Permissions ● For some things you need the user‘s consent – User name – Fine or coarse location – Push actions – Repeated actions

Slide 29

Slide 29 text

Visual Elements ● Only on devices with visual interface – Cards with images, text and links – List selectors – Carousel – Suggestion chips

Slide 30

Slide 30 text

Fulfillment ● It‘s just JSON going back and forth ● Use Cloud Functions / Lambda or whatever you like ● You can use ngrok to make your localhost server available

Slide 31

Slide 31 text

Demo

Slide 32

Slide 32 text

Cross Platform

Slide 33

Slide 33 text

Libs ● AssitantJS https://github.com/webcomputing/AssistantJS ● Jovo Framework https://github.com/jovotech/jovo-framework-nodejs

Slide 34

Slide 34 text

Limitations

Slide 35

Slide 35 text

Limitations of the Platform ● No implicit intents – Probably the main beef on G+ ● Limited options for visual UI elements ● SSML missing important features – Though some have been added recently ● Testing ● Account switching is a PITA, language switching as well ● No private actions (store apps, inhouse apps)

Slide 36

Slide 36 text

But ● Google is eagerly changing and adding stuff – what might be a limitation now might work tomorrow

Slide 37

Slide 37 text

There‘s more out there

Slide 38

Slide 38 text

Resources ● Talks at Google I/O https://events.google.com/io/schedule/?section=may-18&track=assistant https://events.google.com/io/schedule/?section=may-19&track=assistant ● Actions on Google Dev Home: https://developers.google.com/actions/ ● Actions on Google Console: https://console.actions.google.com ● Dialogflow Console https://console.dialogflow.com

Slide 39

Slide 39 text

No content