Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IoT with the Google Assistant

IoT with the Google Assistant

Introductory talk held at the Voice meets IoT Hackathon:


Wolfram Rittmeyer

April 13, 2018


  1. None
  2. None
  3. Voice UX

  4. Voice UX • Voice UX is special – Discoverability of

    app – Discoverability of features – Onboarding • Avoid repetitions – Past behaviour of user
  5. Voice UX • You need to think about your app‘s

    persona • Try to keep to user expectations of a dialog – Script it – Practise script with a partner
  6. Voice UX • Some recommendations – provide feedback – elicit

    answers – Give helpful comments – Repair broken conversations – Provide help – Avoid the usage of the same phrases over and over again
  7. Voice UX • Loads of UX recommendations by Google •

    See the explanations about the IO Assistant app
  8. Developing

  9. None
  10. Assistant SDK • If you want to add the Assistant

    to your device • Enables you to control some aspects of the device
  11. Assistant SDK • Communicate with Google‘s backend using gRPC •

    Change the trigger for the assistant – No direct support for a different hotword (use snowboy or pocketSphinx) • On device support for different traits – Brightness, color... • Either use a lib (Python) or the service directly • For proper devices; you must (!) contact Google; it‘s for development only
  12. Actions on Google – Smart Home https://en.wikipedia.org/wiki/Philips_Hue

  13. Actions on Google – Smart Home • If you want

    your users to control your device using the Google Assistant – „Turn on the light in the living room“
  14. Actions on Google – Smart Home • Consists of two

    parts: – OAuth2 server to authenticate users and issue a token – Some backend to • Query the state of the device • Trigger changes to the device • User adds a device via Google Home app – Here the Oauth flow comes into play • Digg into sample app – https://github.com/actions-on-google/smart-home-nodejs
  15. None
  16. None
  17. None
  18. None
  19. Actions https://assistant.google.com/services/a/uid/0000008011b5e5f2?hl=en-US

  20. Actions • If you want your users to use a

    voice app • „Ask MVV, when‘s the next bus due at Hofmannstraße“ • Use one of those options – Actions SDK – A service like Dialogflow
  21. Actions • If you want your users to use a

    voice app • „Ask MVV, when‘s the next bus due at Hofmannstraße“ • Use Dialogflow
  22. Action SDK

  23. None
  24. Dialogflow

  25. None
  26. App-Structure

  27. Base concepts • Intents – Predefined and custom ones –

    Special ones for smart home • Entities • Fulfillment • Actions • Context
  28. Permissions • For some things you need the user‘s consent

    – User name – Fine or coarse location – Push actions – Repeated actions
  29. Visual Elements • Only on devices with visual interface –

    Cards with images, text and links – List selectors – Carousel – Suggestion chips
  30. Fulfillment • It‘s just JSON going back and forth •

    Use Cloud Functions / Lambda or whatever you like • You can use ngrok to make your localhost server available
  31. Demo

  32. Cross Platform

  33. Libs • AssitantJS https://github.com/webcomputing/AssistantJS • Jovo Framework https://github.com/jovotech/jovo-framework-nodejs

  34. Limitations

  35. Limitations of the Platform • No implicit intents – Probably

    the main beef on G+ • Limited options for visual UI elements • SSML missing important features – Though some have been added recently • Testing • Account switching is a PITA, language switching as well • No private actions (store apps, inhouse apps)
  36. But • Google is eagerly changing and adding stuff –

    what might be a limitation now might work tomorrow
  37. There‘s more out there

  38. Resources • Talks at Google I/O https://events.google.com/io/schedule/?section=may-18&track=assistant https://events.google.com/io/schedule/?section=may-19&track=assistant • Actions

    on Google Dev Home: https://developers.google.com/actions/ • Actions on Google Console: https://console.actions.google.com • Dialogflow Console https://console.dialogflow.com
  39. None