[Webinale] Voice: The New Frontend

Transcript

1. Voice: The New Frontier Frontend Nara Kasbergen (@xiehan) #webinale Thursday,

   December 10, 2020

2. What is NPR? a nationwide network of public radio stations

3. Why voice? Then: Now:

4. None

5. What we're covering today 1. Vision of the future 2.

   How we got here 3. Where we are today 4. How to build for voice today 5. Lessons learned

6. Vision of the future 1

7. None
8. None
9. None
10. None

11. Prediction: In the future, we will all be voice developers.

12. How we got here 2

13. Natural Language Processing definition: the ability of a computer program

    to understand human language as it is spoken. NLP is a component of artificial intelligence (AI). (source) circa 1960 first major advances in the 1980s

14. Machine Learning (ML) definition: Machine learning is an application of

    artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it to learn for themselves. (source) circa 1950s first major advances in the 1980s

15. https:/ /hackernoon.com/moores-law-is-alive-and-well-adc010ea7a63

16. 1994 IBM Simon is the world's first smart phone 1987

    2009 Microsoft begins work on Cortana 2010 Siri launches as a standalone iOS app, later acquired by Apple 2012 Google Now Apple's Knowledge Navigator

17. 2015 Amazon Alexa Skills Kit launches (June) 2014 2016 Google

    Assistant (May) + Google Home (November) 2017 Cortana Skills Kit (May) + Samsung Bixby (August) 2018 Apple HomePod (February) Amazon Echo launches November 6

18. Rise of multimodal devices

19. Where we are today 3

20. The major players

21. The state at the end of 2019

22. Predictions for 2021

23. The request/response flow Your code request response

24. The request/response flow Your code request response P.S. all the

    NLP and ML happens here

25. Voice for the web Web Speech API: • Speech recognition

    • Speech synthesis (TTS) W3C Community specification was published in 2012 SpeechRecognition interface currently only supported in Chrome, experimental feature Uses Google's servers to convert speech to text (requires Internet connection)

26. Mozilla CommonVoice "Common Voice is Mozilla's initiative to help teach

    machines how real people speak." (source) * In maintenance mode since August 2020 Publicly open dataset Upload recordings of your voice Help reduce bias in Natural Language Processing (NLP) & ML

27. Mozilla Scout Mycroft Open source alternatives

28. The enterprise voice space NLP/ML/AI Consumer applications

29. How to build for voice today 4

30. disclaimer it's not about the code

31. The stack • node.js • serverless (AWS Lambda or Google

    Cloud Functions) • lightweight database (e.g. Amazon DynamoDB) • CI server of choice (e.g. Travis, Jenkins, etc.) • unit testing framework of choice (e.g. Jest) • TypeScript…?

32. Use the official SDKs Alexa node.js SDK: github.com/alexa/alexa-skills-kit-sdk-for-nodejs Actions on

    Google node.js SDK: github.com/actions-on-google/actions-on-google-nodejs

33. Glossary Alexa Google / Dialogflow skill action / agent invocation

    name intent slot entity sample utterance training phrase

34. “Alexa, ask NPR to play the latest news”

35. “Alexa, ask NPR to play the latest news” invocation name

36. “Alexa, ask NPR what's playing”

37. “Alexa, ask NPR what's playing” “Alexa, ask NPR what am

    I listening to?”

38. “Alexa, ask NPR what's playing” “Alexa, ask NPR what am

    I listening to?” intent sample utterance sample utterance

39. “Alexa, ask NPR to play Planet Money” “Alexa, ask NPR

    to play Hidden Brain”

40. “Alexa, ask NPR to play Planet Money” “Alexa, ask NPR

    to play Hidden Brain” intent slot (or entity)

41. Basic code architecture JSON event with intent name and optional

    slot(s) Handler function mapped to that intent Use SDK to produce a response with speech, audio, etc.

42. Alexa "Hello World" skill

43. Alexa "Hello World" skill

44. Alexa "Hello World" skill

45. Alexa "Hello World" skill

46. Google "Hello World" action

47. Google "Hello World" action

48. Next, add more features Simple • Launch requests • Add

    intents • Wait for response • Handle slots • Add images to cards • Play simple audio • Use SSML Medium • User login (OAuth2) • Persistent data • State management • Contexts (Google) • Dialog management (Alexa) • Advanced audio (Alexa) Advanced • Customize display on visual devices • Monetization / transactions • Request user permissions • Notifications • Internationalization

49. Publishing a skill/action • By default, skills in development are

    only available to you • Certification process similar to mobile app store submissions ◦ ~48-hour turnaround on average ◦ Feedback is unpredictable! ◦ Respect existing brands • Can share a "beta" version with co-workers, friends, etc. ◦ Great for QA as well as hobby projects

50. Get started without code Alexa: • Alexa Skill Blueprints Google:

    • Actions on Google Templates

51. Lessons learned 5

52. The challenges • The code is not hard • What

    is hard: ◦ Learning about the platform limitations ◦ Managing stakeholder expectations ◦ Understanding & changing user behaviors ◦ QA • It's not just an engineering challenge!

53. Prediction: In the future, we will all be voice developers.

54. Thank you! Keep in touch: @xiehan nara@nara.codes