Implementing a custom AI voice Assistant by streaming WebRTC to Dialogflow & Cloud Speech

8a235da15adae86851fa3216834198ed?s=47 Lee Boonstra
February 18, 2020

Implementing a custom AI voice Assistant by streaming WebRTC to Dialogflow & Cloud Speech

A best practice for streaming audio from a browser microphone to Dialogflow or Google Cloud STT by using websockets.

8a235da15adae86851fa3216834198ed?s=128

Lee Boonstra

February 18, 2020
Tweet

Transcript

  1. Implementing a custom AI voice Assistant by streaming WebRTC to

    Dialogflow & Cloud Speech Lee Boonstra Developer Advocate, Google Twitter: @ladysign
  2. Lee Boonstra Developer Advocate, Conversational AI Ex- JavaScript Technical Trainer

    Public Speaker (since 2013) Writer/Blogger for Techzine, .Net Magazine, Marketingfacts.nl, CustomerTalk.nl and Google Cloud Blog @ladysign http://www.leeboonstra.com
  3. What is an AI voice assistant? An intelligent software agent

    that can perform tasks for an individual based on commands or questions, through human speech and respond via synthesized voices.
  4. Why would you create your own voice AI? This is

    not a talk about the Google Assistant! It’s about streaming your own microphone to a back-end that can give smart answers with machine learning.
  5. • Public available • Runs on all Assistant powered devices

    • Native Google Assistant features • Invoke actions (Hey Google talk to <my app>) • Customer Terms & Conditions • Special technical requirements Google Assistant facts: PUBLIC Your business action Action from someone else Weather Action Recipes Action Nest Thermostat Alarm Clock You might want to build your own voice AI instead because of technical requirements, overkill, enterprise usage Lee Boonstra | @ladysign
  6. Use Cases for custom AI voice assistants Lee Boonstra |

    @ladysign
  7. Short Utterance vs. Streaming Turn on the lights Short utterance.

    -> Match to 1 intent. Long utterance. Many possible intent matches. Lee Boonstra | @ladysign
  8. The idea: Airport Self Service Kiosk Demo

  9. App Demo

  10. None
  11. A best practice for streaming audio from a browser microphone

    to Dialogflow or Google Cloud STT by using websockets.
  12. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Browser plays audio Architecture Lee Boonstra | @ladysign
  13. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Browser plays audio Architecture How to get a microphone audio stream which works across all browsers? How to make sure the audio stream can be handled as an ArrayBuffer in the back-end?
  14. RecordRTC RecordRTC is WebRTC JavaScript library for audio/video as well

    as screen activity recording. Lee Boonstra | @ladysign
  15. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Server Node.js Codebase Docker Container Browser plays audio Architecture How to stream from front-end to back-end? How to stream Bidirectional Binary Data? Lee Boonstra | @ladysign
  16. • Socket.io - Socket.IO enables real-time bidirectional event-based communication. •

    Socket.io-Stream - for binary stream transfers through Socket.io Lee Boonstra | @ladysign
  17. RecordRTC (web) Lee Boonstra | @ladysign

  18. Socket Stream (web) Lee Boonstra | @ladysign

  19. Socket Stream (node) Lee Boonstra | @ladysign

  20. Lee Boonstra | @ladysign Google Speech-to-Text enables developers to convert

    audio to text by applying powerful neural network models in an easy-to-use API. The API recognizes 120 languages and variants to support your global user base. You can enable voice command-and-control, transcribe audio from call centers, and more.
  21. STT (node) Lee Boonstra | @ladysign

  22. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Server Node.js Codebase Docker Container Cloud STT Voice to text Browser plays audio Architecture AudioBuffer? Encoding? SampleRate? Number of Channels? How to get text from the HTML5 browser microphone stream? Lee Boonstra | @ladysign
  23. STT Node Demo

  24. Lee Boonstra | @ladysign Development suite for building Conversational UIs.

    • Formerly known as API.AI ◦ (Sept 2016, acquired by Google) • Powered by Machine Learning: ◦ Natural Language Understanding (NLU) ◦ Intent Matching ◦ Conversation Training • Cross platform • Build faster with the Web UI • Scalable: Separate your conversation from code • Speech / Voice Integration • Multi-lingual bot support (20+ languages) • Direct integration with 15+ channels like Google Assistant, Slack, Twilio, Facebook...
  25. None
  26. None
  27. None
  28. None
  29. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Server Node.js Codebase Docker Container Cloud STT Voice to text Dialogflow Intent Matching Browser plays audio Architecture Translate lang to English Translate lang to English Dialogflow to detect intents. Translate text to base language, and translate back. Lee Boonstra | @ladysign
  30. Dialogflow (node) Lee Boonstra | @ladysign

  31. Lee Boonstra | @ladysign When you sound like a computer.

    People treat you like a computer. This is where Wavenet Technology comes in. • Voices sound natural and unique • Capture subtleties like pitch, pace, and all the pauses that convey meaning TTS - Making use of DeepMind's WaveNet Technology https://deepmind.com/blog/wavenet-generative-model-raw-audio/ Lee Boonstra | @ladysign
  32. TTS (node) Lee Boonstra | @ladysign

  33. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Server Node.js Codebase Docker Container Cloud TTS Spoken voice Cloud STT Voice to text Dialogflow Intent Matching Browser plays audio Architecture Translate lang to English Translate lang to English How to bring an AudioBuffer to the WebClient? How to make it autoplay in all browsers? Lee Boonstra | @ladysign
  34. AudioBufferSourceNode (web) Lee Boonstra | @ladysign

  35. User records voice with browser microphone Web Client Angular App

    with WebRTC Docker Container Server Node.js Codebase Docker Container Cloud TTS Spoken voice Cloud STT Voice to text Dialogflow Intent Matching Browser plays audio Architecture Translate lang to English Translate lang to English Need HTTPS? SSL Certificates? Lee Boonstra | @ladysign
  36. https://github.com/dialogflow/sel fservicekiosk-audio-streaming I open sourced my code Lee Boonstra |

    @ladysign
  37. Thanks! Lee Boonstra | @ladysign Github: https://github.com/dialogflow/selfservicekiosk-audio-streaming Slides: https://bit.ly/2OZN9yg

  38. Hey, why not a Single Page App? Anti Pattern I've

    seen solutions online where the microphone directly got streamed to the Dialogflow, without a server part. The REST calls were made directly in the web client with JavaScript. I would consider this as an anti-pattern. You will likely expose your service account / private key in your client-side code. Anyone, who is handy with ChromeDev tools, could steal your key, and make (paid) API calls via your account. It's a better approach to always let a server handle the Google Cloud authentication. This way the service account won't be exposed to the public.