Build your own voice assistant

When you say, “Hey Siri, play Spotify”, or “Alexa, turn on my kitchen lights” have you ever wondered how they work?

In this talk I’ll show you how to build your own local, offline voice assistant using open source software – all runnable on a Raspberry Pi. In the process you’ll learn about wake words, speech-to-text and text-to-speech, intent matching and handling, and we’ll write a few simple commands to get the weather, and start playing music on Spotify.

No prior knowledge of Docker, Pi’s or voice assistants is needed, just a curiosity into how they work.

Transcript

1. Build your own voice assistant Dale Humby daleh@spotify.com
3. techtarget.com
4. dowsingandreynolds.com
5. Photo by Moritz Kindler on Unsplash
6. denofgeek.com
7. Black Mirror, White Christmas
8. home-assistant.io Open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.
13. • Local • Offline • Highly customisable • Experiment with new ideas • Doesn't have to be cost effective • Learn Why build your own voice assistant?
14. Rhasspy is an open source, fully offline set of voice assistant services for many human languages rhasspy.readthedocs.io github.com/rhasspy/rhasspy
15. Each Rhasspy services can be swapped-out, to customise and improve the voice assistant
16. tenor.com/en-GB/view/this-is-fine-gif-24177057
17. Hardware • Raspberry Pi 4B (Pi 3 should work) • A good micro-USB power supply • Micro-SD card, for Raspberry Pi OS • HDMI cable and USB keyboard • Jabra 410 or 510 USB speaker (but any popular mic+speaker or headset should be fine) … or as a Virtual Machine on your laptop
18. Install Rhasspy services: rhasspy: image: rhasspy/rhasspy container_name: rhasspy restart: unless-stopped volumes: - "$HOME/rhasspy/:/profiles" - "/etc/localtime:/etc/localtime:ro" ports: - "12101:12101" devices: - "/dev/snd:/dev/snd" command: --user-profiles /profiles --profile en hub.docker.com/r/rhasspy/rhasspy docker-compose.yml
21. "Hey Google… what's the weather today?"
22. Wake word
23. Hey Google… what's the weather today?
24. Hey Google… what's the weather today?
25. Hey Google… what's the weather today?
26. Hey Google… what's the weather today?
27. Hey Google… what's the weather today?
29. Speech to text
30. Hey Google… what's the weather today? Mozilla Deep Speech • Open source • offline, on-device • speech-to-text • real time • Raspberry Pi 4 Voice Activity Detector (VAD) • Classifies audio • voiced or unvoiced • Silence detection
32. Intent recognition
34. Hey Mycroft… Turn on the light Raw utterance
35. [LightState] turn (on | off){state} [the] light Hey Mycroft… Turn on the light Raw utterance Intents grammar
36. [LightState] turn (on | off){state} [the] light { "text": "turn on the light", "intent": { "name": "LightState" }, "slots": { "state": "on" } } Hey Mycroft… Turn on the light Raw utterance Intents grammar Structured data - Intent - Slots
37. { "wakeword_id": "hey-mycroft-2", "raw_text": "turn on the light", "intent": { "name": "LightState", "confidence": 1 }, "slots": { "state": "on" } }
38. [GetTime] what time is it tell me the time whats the time [GetWeather] whats the weather (today | tomorrow){day} whats the weather on ($rhasspy/days){day} [LightState] states = (on | off) turn <states>{state} [the] light Intents grammar sentences.ini
39. Text to speech
41. Install Mycroft AI's mimic3 hub.docker.com/r/mycroftai/mimic3 mimic3: image: mycroftai/mimic3 container_name: mimic3 restart: unless-stopped ports: - "59125:59125" volumes: - "$HOME/mimic3:/home/mimic3/.local/share/mycroft/mimic3" docker-compose.yml
44. Intent handling
45. [GetWeather] whats the weather (today | tomorrow){day} whats the weather on ($rhasspy/days){day} { "text": "whats the weather on tuesday", "intent": { "name": "GetWeather" }, "slots": { "day": "tuesday" } } Hey Mycroft… What's the weather on Tuesday? Raw utterance Intents grammar Structured data - Intent - Slots
47. Install Node-RED node-red: image: nodered/node-red:latest restart: unless-stopped ports: - "1880:1880" environment: - TZ=Europe/Sweden volumes: - "$HOME/node-red:/data" - "/etc/localtime:/etc/localtime:ro" hub.docker.com/r/nodered/node-red docker-compose.yml
51. msg.payload = "The time is 11 42" return msg return msg.payload Hey Mycroft… What's the time? POST http://rhasspy:12101/api/text-to-speech
52. Hey Mycroft… What's the time? const dateTime = new Date() const options = { hour: 'numeric', minute: 'numeric', hour12: 'true', timeZone: 'Europe/Stockholm' } const formatted = Intl.DateTimeFormat('en-GB', options).format(dateTime) msg.payload = `The time is ${formatted}` return msg
53. Hey Mycroft… What's the weather tomorrow?
68. Photo by Max van den Oetelaar on Unsplash
69. The Future…?
70. Hardware
71. Satellites Base
72. Low power Satellite High power Base
73. Supposed to be 300kr Now… 1,000+ kr
74. ESP32-wroom ESP32-S3 https://esp32s3.com/tinys3.html
75. Wake word
77. Open Wake Word github.com/dscripka/openWakeWord
78. ASR
79. Whisper … approaches human level robustness and accuracy on English speech recognition. openai.com/research/whisper github.com/ggerganov/whisper.cpp github.com/guillaumekln/faster-whisper
80. Large Language Model
81. Fulfilment with GPT-3 ChatGPT … GPT-4 ChatGPT in an iOS Shortcut — Worlds Smartest HomeKit Voice Assistant
82. [riffusion] riffusion.com Stable Diffusion / Dall.e stablediffusionweb.com
83. Deep Neural Networks for automation? How far can we push
84. GPT-4 The quick brown fox jumps over the lazy dogs Deep Neural Networks for automation? How far can we push
85. GPT-4 The quick brown fox jumps over the lazy dogs Deep Neural Networks for automation? Stable Diffusion Astronaut riding a horse How far can we push
86. GPT-4 The quick brown fox jumps over the lazy dogs Deep Neural Networks for automation? Riffusion Lo-Fi jazz How far can we push
87. Events > Actions Dale walks into kitchen GPT-4 The quick brown fox jumps over the lazy dogs Deep Neural Networks for automation? How far can we push Riffusion Lo-Fi jazz
88. Events > Actions Dale walks into kitchen - Kitchen light on - Under counter light on - Kettle on GPT-4 The quick brown fox jumps over the lazy dogs Deep Neural Networks for automation? How far can we push Riffusion Lo-Fi jazz
89. Deep Neural Networks for automation? How far can we push
90. Deep Neural Networks for automation? How far can we push
91. Deep Neural Networks for automation? How far can we push
92. Deep Neural Networks for automation? How far can we push
93. Deep Neural Networks for automation? How far can we push
94. Deep Neural Networks for automation? How far can we push
95. Deep Neural Networks for automation? How far can we push
96. Deep Neural Networks for automation? How far can we push
98. Thank you! dale@humby.co.za dhum.by/github home-assistant.io rhasspy.readthedocs.io
99. tts sonantic, … • multi-turn conversation • more general knowledge