Lessons learned using Natural Language Understanding in the Real World

In Liberty IT we’ve been building real-world chatbots for a Fortune 100 company and have learned a lot of lessons. Let us talk you through the basics of Natural Language Understanding and how it works. We’ll cover why it’s challenging, how we’ve used it and give pointers on how to get the best from your NLU chatbot.

Transcript

1. Why Is Natural Language Understanding So Hard To Do In Practice? Gillian Armstrong Susan O’Brien Convercon 2018
2. Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Emerging Technologies Susan O’Brien @susanobrien88 Principal Software Engineer Liberty IT Lessons learned using Natural Language Understanding in the Real World
3. What is Natural Language Understanding? "the comprehension by computers of the structure and meaning of human language, allowing users to interact with the computer using natural sentences.” - Gartner
4. The Web is about Telling the user what to do
5. The Web is about Telling the user what to do Conversational UI is about Listening to what the user wants to do
6. Let me show you…
7. Employee Digital Assistant https://www.workgrid.com/
8. Claims Virtual Agent
9. How does Natural Language Understanding work?
10. DICTIONARY GRAMMAR
12. over /ˈəʊvə/ preposition
13. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall.
14. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR
15. The bird flew over the wall The ball flew over the wall The flag flew over the wall
16. The bird flew over the wall, circling ominously as if waiting to see if I would lose my precarious hold and fall to my death. He loaded the ball into the cannon and lit the fuse. The ball flew over the wall and soared off into the distance. ‘Watch out’, he called, tossing the flag. The flag flew over the wall. I sprung forward just in time to catch it.
17. Sometimes balls do fly over things by themselves
18. Demo using Amazon Lex Bot Follow along or try out later at https://github.com/virtualgill/breakfast-bot
20. Utterance
21. MAKE TEA Intent Utterance MAKE COFFEE MAKE TOAST
22. Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast Intent Utterance
23. Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast Intent Fulfillment Utterance
24. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast
26. MAKE TEA MAKE COFFEE MAKE TOAST
27. Natural Language Understanding, however clever, doesn’t really involve understanding natural language.
28. MAKE TEA MAKE COFFEE MAKE TOAST
29. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast
30. TEA COFFEE TOAST
31. TEA COFFEE TOAST
32. DRINKS TOAST
33. MAKE HOT DRINK MAKE TOAST I’d like some {DRINK} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} Get me a cup of {DRINK} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast Coffee Tea … {DRINK}
34. MAKE BREAKFAST I’d like some {DRINK} I’d like some {DRINK} and {FOOD} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} and {FOOD} Get me a cup of {DRINK} and some {FOOD} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} and {FOOD} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Get me some {FOOD} I would like some {FOOD} I'd like {FOOD} Coffee Tea … {DRINK} Toast … {FOOD}
35. Some very quick tips
36. Make sure you do your analysis up front to really understand what your intents should be. It may not be what you initially think!
37. Combine Intents and use Slots where it makes sense. Remember slots are training data too. Include the right data in your slot data, and give good examples of where it would appear in your utterances.
38. Find the right amount of utterances – not too many or too few. This is highly dependent on your context. The more intents you have, the more vital it is that the utterances in it are similar to each other, and different enough from other intents.
39. Don’t make up your conversational data. Get it from your users!
40. Don’t just add in missed utterances without thought. It may modify other intents. Also – are you sure you know what the utterance should match? Does it need a new intent?
41. Get a great set of tests – and run every time you make a change. Retraining changes your bot’s whole world. It may not know what it did before.
42. First Run: After Utterance Refinement:
44. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Don’t forget there’s more than just technology
45. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
46. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
47. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
48. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Gillian Armstrong @virtualgill
49. Got questions? Let’s chat! @virtualgill @susanobrien88 Demo Code at https://github.com/virtualgill/breakfast-bot
50. Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies Susan O’Brien @susanobrien88 Principal Software Engineer Liberty IT Lessons learned using Natural Language Understanding in the Real World