[Webinale] Voice: The New Frontend

Transcript

1. Voice: The New
   Frontier
   Frontend Nara Kasbergen (@xiehan)
   #webinale
   Thursday, December 10, 2020
   

   View Slide

2. What is NPR?
   a
   nationwide
   network of
   public radio
   stations
   

   View Slide

3. Why voice?
   Then: Now:
   

   View Slide

4. View Slide

5. What we're covering today
   1. Vision of the future
   2. How we got here
   3. Where we are today
   4. How to build for voice today
   5. Lessons learned
   

   View Slide

6. Vision of the future
   1
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. Prediction:
    In the future, we
    will all be voice
    developers.
    

    View Slide

12. How we got here
    2
    

    View Slide

13. Natural Language Processing
    definition:
    the ability of a computer program to understand human
    language as it is spoken. NLP is a component of artificial
    intelligence (AI). (source)
    circa 1960
    first major advances in the 1980s
    

    View Slide

14. Machine Learning (ML)
    definition:
    Machine learning is an application of artificial intelligence (AI)
    that provides systems the ability to automatically learn and
    improve from experience without being explicitly programmed.
    Machine learning focuses on the development of computer
    programs that can access data and use it to learn for
    themselves. (source)
    circa 1950s
    first major advances in the 1980s
    

    View Slide

15. https:/
    /hackernoon.com/moores-law-is-alive-and-well-adc010ea7a63
    

    View Slide

16. 1994
    IBM Simon is
    the world's first
    smart phone
    1987 2009
    Microsoft
    begins work on
    Cortana
    2010
    Siri launches as a
    standalone iOS
    app, later acquired
    by Apple
    2012
    Google Now
    Apple's
    Knowledge
    Navigator
    

    View Slide

17. 2015
    Amazon Alexa
    Skills Kit launches
    (June)
    2014 2016
    Google
    Assistant (May) +
    Google Home
    (November)
    2017
    Cortana Skills Kit
    (May) +
    Samsung Bixby
    (August)
    2018
    Apple HomePod
    (February)
    Amazon Echo
    launches
    November 6
    

    View Slide

18. Rise of multimodal devices
    

    View Slide

19. Where we are today
    3
    

    View Slide

20. The major players
    

    View Slide

21. The state at the end of 2019
    

    View Slide

22. Predictions for 2021
    

    View Slide

23. The request/response flow
    Your
    code
    request
    response
    

    View Slide

24. The request/response flow
    Your
    code
    request
    response
    P.S. all the NLP and ML happens here
    

    View Slide

25. Voice for the web
    Web Speech API:
    ● Speech recognition
    ● Speech synthesis (TTS)
    W3C Community specification was published in 2012
    SpeechRecognition interface currently only supported in
    Chrome, experimental feature
    Uses Google's servers to convert speech to text (requires
    Internet connection)
    

    View Slide

26. Mozilla CommonVoice
    "Common Voice is Mozilla's initiative to help teach machines
    how real people speak." (source)
    * In maintenance mode since August 2020
    Publicly open dataset
    Upload recordings of your voice
    Help reduce bias in Natural Language Processing (NLP) & ML
    

    View Slide

27. Mozilla Scout Mycroft
    Open source alternatives
    

    View Slide

28. The enterprise voice space
    NLP/ML/AI Consumer applications
    

    View Slide

29. How to build for voice
    today
    4
    

    View Slide

30. disclaimer
    it's not about the code
    

    View Slide

31. The stack
    ● node.js
    ● serverless (AWS Lambda or Google Cloud Functions)
    ● lightweight database (e.g. Amazon DynamoDB)
    ● CI server of choice (e.g. Travis, Jenkins, etc.)
    ● unit testing framework of choice (e.g. Jest)
    ● TypeScript…?
    

    View Slide

32. Use the official SDKs
    Alexa node.js SDK:
    github.com/alexa/alexa-skills-kit-sdk-for-nodejs
    Actions on Google node.js SDK:
    github.com/actions-on-google/actions-on-google-nodejs
    

    View Slide

33. Glossary Alexa Google / Dialogflow
    skill action / agent
    invocation name
    intent
    slot entity
    sample utterance training phrase
    

    View Slide

34. “Alexa, ask NPR
    to play the latest news”
    

    View Slide

35. “Alexa, ask NPR
    to play the latest news”
    invocation name
    

    View Slide

36. “Alexa, ask NPR
    what's playing”
    

    View Slide

37. “Alexa, ask NPR
    what's playing”
    “Alexa, ask NPR
    what am I listening to?”
    

    View Slide

38. “Alexa, ask NPR
    what's playing”
    “Alexa, ask NPR
    what am I listening to?”
    intent
    sample utterance
    sample utterance
    

    View Slide

39. “Alexa, ask NPR
    to play Planet Money”
    “Alexa, ask NPR
    to play Hidden Brain”
    

    View Slide

40. “Alexa, ask NPR
    to play Planet Money”
    “Alexa, ask NPR
    to play Hidden Brain”
    intent slot
    (or entity)
    

    View Slide

41. Basic code architecture
    JSON event with intent name and optional slot(s)
    Handler function mapped to that intent
    Use SDK to produce a response with speech, audio, etc.
    

    View Slide

42. Alexa "Hello World" skill
    

    View Slide

43. Alexa "Hello World" skill
    

    View Slide

44. Alexa "Hello World" skill
    

    View Slide

45. Alexa "Hello World" skill
    

    View Slide

46. Google "Hello World" action
    

    View Slide

47. Google "Hello World" action
    

    View Slide

48. Next, add more features
    Simple
    ● Launch requests
    ● Add intents
    ● Wait for response
    ● Handle slots
    ● Add images to cards
    ● Play simple audio
    ● Use SSML
    Medium
    ● User login (OAuth2)
    ● Persistent data
    ● State management
    ● Contexts (Google)
    ● Dialog management
    (Alexa)
    ● Advanced audio
    (Alexa)
    Advanced
    ● Customize display
    on visual devices
    ● Monetization /
    transactions
    ● Request user
    permissions
    ● Notifications
    ● Internationalization
    

    View Slide

49. Publishing a skill/action
    ● By default, skills in development are only available to you
    ● Certification process similar to mobile app store submissions
    ○ ~48-hour turnaround on average
    ○ Feedback is unpredictable!
    ○ Respect existing brands
    ● Can share a "beta" version with co-workers, friends, etc.
    ○ Great for QA as well as hobby projects
    

    View Slide

50. Get started without code
    Alexa:
    ● Alexa Skill Blueprints
    Google:
    ● Actions on Google Templates
    

    View Slide

51. Lessons learned
    5
    

    View Slide

52. The challenges
    ● The code is not hard
    ● What is hard:
    ○ Learning about the platform limitations
    ○ Managing stakeholder expectations
    ○ Understanding & changing user behaviors
    ○ QA
    ● It's not just an engineering challenge!
    

    View Slide

53. Prediction:
    In the future, we
    will all be voice
    developers.
    

    View Slide

54. Thank you!
    Keep in touch:
    @xiehan
    nara@nara.codes
    

    View Slide