Pro Yearly is on sale from $80 to $50! »

Lessons learned using Natural Language Understanding in the Real World

705df02f597ff36f900251c749e879cf?s=47 Gillian Armstrong
September 06, 2018
12

Lessons learned using Natural Language Understanding in the Real World

In Liberty IT we’ve been building real-world chatbots for a Fortune 100 company and have learned a lot of lessons. Let us talk you through the basics of Natural Language Understanding and how it works. We’ll cover why it’s challenging, how we’ve used it and give pointers on how to get the best from your NLU chatbot.

705df02f597ff36f900251c749e879cf?s=128

Gillian Armstrong

September 06, 2018
Tweet

Transcript

  1. Why Is Natural Language Understanding So Hard To Do In

    Practice? Gillian Armstrong Susan O’Brien Convercon 2018
  2. Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Emerging Technologies Susan

    O’Brien @susanobrien88 Principal Software Engineer Liberty IT Lessons learned using Natural Language Understanding in the Real World
  3. What is Natural Language Understanding? "the comprehension by computers of

    the structure and meaning of human language, allowing users to interact with the computer using natural sentences.” - Gartner
  4. The Web is about Telling the user what to do

  5. The Web is about Telling the user what to do

    Conversational UI is about Listening to what the user wants to do
  6. Let me show you…

  7. Employee Digital Assistant https://www.workgrid.com/

  8. Claims Virtual Agent

  9. How does Natural Language Understanding work?

  10. DICTIONARY GRAMMAR

  11. None
  12. over /ˈəʊvə/ preposition

  13. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall.
  14. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR

  15. The bird flew over the wall The ball flew over

    the wall The flag flew over the wall
  16. The bird flew over the wall, circling ominously as if

    waiting to see if I would lose my precarious hold and fall to my death. He loaded the ball into the cannon and lit the fuse. The ball flew over the wall and soared off into the distance. ‘Watch out’, he called, tossing the flag. The flag flew over the wall. I sprung forward just in time to catch it.
  17. Sometimes balls do fly over things by themselves

  18. Demo using Amazon Lex Bot Follow along or try out

    later at https://github.com/virtualgill/breakfast-bot
  19. None
  20. Utterance

  21. MAKE TEA Intent Utterance MAKE COFFEE MAKE TOAST

  22. Get me a coffee Bring me a cup of coffee

    Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast Intent Utterance
  23. Get me a coffee Bring me a cup of coffee

    Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast Intent Fulfillment Utterance
  24. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast
  25. None
  26. MAKE TEA MAKE COFFEE MAKE TOAST

  27. Natural Language Understanding, however clever, doesn’t really involve understanding natural

    language.
  28. MAKE TEA MAKE COFFEE MAKE TOAST

  29. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast
  30. TEA COFFEE TOAST

  31. TEA COFFEE TOAST

  32. DRINKS TOAST

  33. MAKE HOT DRINK MAKE TOAST I’d like some {DRINK} Make

    me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} Get me a cup of {DRINK} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast Coffee Tea … {DRINK}
  34. MAKE BREAKFAST I’d like some {DRINK} I’d like some {DRINK}

    and {FOOD} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} and {FOOD} Get me a cup of {DRINK} and some {FOOD} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} and {FOOD} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Get me some {FOOD} I would like some {FOOD} I'd like {FOOD} Coffee Tea … {DRINK} Toast … {FOOD}
  35. Some very quick tips

  36. Make sure you do your analysis up front to really

    understand what your intents should be. It may not be what you initially think!
  37. Combine Intents and use Slots where it makes sense. Remember

    slots are training data too. Include the right data in your slot data, and give good examples of where it would appear in your utterances.
  38. Find the right amount of utterances – not too many

    or too few. This is highly dependent on your context. The more intents you have, the more vital it is that the utterances in it are similar to each other, and different enough from other intents.
  39. Don’t make up your conversational data. Get it from your

    users!
  40. Don’t just add in missed utterances without thought. It may

    modify other intents. Also – are you sure you know what the utterance should match? Does it need a new intent?
  41. Get a great set of tests – and run every

    time you make a change. Retraining changes your bot’s whole world. It may not know what it did before.
  42. First Run: After Utterance Refinement:

  43. None
  44. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Don’t forget there’s more than just technology
  45. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  46. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  47. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  48. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Gillian Armstrong @virtualgill
  49. Got questions? Let’s chat! @virtualgill @susanobrien88 Demo Code at https://github.com/virtualgill/breakfast-bot

  50. Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies Susan

    O’Brien @susanobrien88 Principal Software Engineer Liberty IT Lessons learned using Natural Language Understanding in the Real World