Natural Language (mis)Understanding

As a developer I spend a lot of time being cross that a computer did what asked it to, and not what I meant it to. With all the AI services out there allowing Natural Language Processing and Understanding, surely the time has come when computers can finally really understand us? Let’s take a look at how easy it is to build a chatbot using an NLU service, and how easy it is to break it if you haven’t thought through your interaction model design and training data carefully. We may need other humans for a while yet…

Transcript

1. Natural Language (mis)Understanding Gillian Armstrong
2. 2 Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies Liberty IT / Workgrid Software
3. Will a chatbot take my job? Does your job involve talking to people? Not if I can help it Yes Do you end up saying the same thing over and over again? Yes No Did anyone actually ask you? Yes, that’s totally my whole job High Risk of co-workers replacing you with small shell script Unlikely to be a chatbot that takes your job Not any time soon Well, no, but… Gillian Armstrong @virtualgill
4. What is Natural Language Understanding?? Gillian Armstrong @virtualgill
5. Gillian Armstrong @virtualgill
6. Gillian Armstrong @virtualgill
7. DICTIONARY GRAMMAR Gillian Armstrong @virtualgill
8. over /ˈəʊvə/ preposition Gillian Armstrong @virtualgill
9. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
10. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
11. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
12. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
13. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
14. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
15. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
16. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
17. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
18. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
19. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
20. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
21. Gillian Armstrong @virtualgill
22. CAT DOG ZEBRA Gillian Armstrong @virtualgill
23. • He drove over the bridge • The house is just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
24. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR Gillian Armstrong @virtualgill
25. The bird flew over the wall The ball flew over the wall The flag flew over the wall Gillian Armstrong @virtualgill
26. The bird flew over the wall The ball flew over the wall The flag flew over the wall Gillian Armstrong @virtualgill
27. The bird flew over the wall, circling ominously as if waiting to see if I would lose my precarious hold and fall to my death. He loaded the ball into the cannon and lit the fuse. The ball flew over the wall and soared off into the distance. ‘Watch out’, he called, tossing the flag. The flag flew over the wall. I sprung forward just in time to catch it. Gillian Armstrong @virtualgill
28. Sometimes balls do fly over things by themselves Gillian Armstrong @virtualgill
29. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR Gillian Armstrong @virtualgill
30. Gillian Armstrong @virtualgill
31. Current State NLU Gillian Armstrong @virtualgill
32. Gillian Armstrong @virtualgill
33. Utterance Gillian Armstrong @virtualgill
34. MAKE TEA Intent Utterance MAKE COFFEE MAKE TOAST Gillian Armstrong @virtualgill
35. Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like a wee brew Make me a pot of tea Bring me a cup of tea Bring me tea I really need a cuppa Make me some tea Bring me toast I would like toast and jam Toast me some bread Get me some toast I would like some toast Intent Utterance Gillian Armstrong @virtualgill
36. Get me a coffee Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like a wee brew Make me a pot of tea Bring me a cup of tea Bring me tea I really need a cuppa Make me some tea Bring me toast I would like toast and jam Toast me some bread Get me some toast I would like some toast Intent Fulfillment Utterance Gillian Armstrong @virtualgill
37. Demo using Amazon Lex Bot Code at https://github.com/virtualgill/breakfast-bot Gillian Armstrong @virtualgill
38. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast V1 Gillian Armstrong @virtualgill
39. V1 Gillian Armstrong @virtualgill
40. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast V1 Gillian Armstrong @virtualgill
41. Gillian Armstrong @virtualgill
42. Gillian Armstrong @virtualgill
43. V1 Gillian Armstrong @virtualgill
44. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V2 Gillian Armstrong @virtualgill
45. V2 Gillian Armstrong @virtualgill
46. V2 Gillian Armstrong @virtualgill
47. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Fetch me a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V3 Gillian Armstrong @virtualgill
48. V3 Gillian Armstrong @virtualgill
49. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Fetch me a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V3 Gillian Armstrong @virtualgill
50. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Fetch me a cuppa I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V4 Gillian Armstrong @virtualgill
51. V4 Gillian Armstrong @virtualgill
52. Natural Language Understanding, however clever, doesn’t really involve understanding natural language. Gillian Armstrong @virtualgill
53. Gillian Armstrong @virtualgill
54. Gillian Armstrong @virtualgill
55. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Fetch me a cuppa I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V4 Gillian Armstrong @virtualgill
56. TEA COFFEE TOAST Gillian Armstrong @virtualgill
57. TEA COFFEE TOAST Gillian Armstrong @virtualgill
58. DRINKS TOAST Gillian Armstrong @virtualgill
59. MAKE HOT DRINK MAKE TOAST I’d like some {DRINK} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} Get me a cup of {DRINK} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V5 Gillian Armstrong @virtualgill
60. MAKE BREAKFAST I’d like some {DRINK} I’d like some {DRINK} and {FOOD} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} and {FOOD} Get me a cup of {DRINK} and some {FOOD} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} and {FOOD} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Get me some {FOOD} I would like some {FOOD} I'd like {FOOD} V6 Gillian Armstrong @virtualgill
61. Some very quick tips Gillian Armstrong @virtualgill
62. Make sure you do your analysis up front to really understand what your intents should be. It may not be what you initially think! Gillian Armstrong @virtualgill
63. You wouldn’t make up other training data for ML – don’t make up your conversational data either. Get it from your users. Gillian Armstrong @virtualgill
64. Decide if you want to use a service or roll your own. If you have a small closed domain with an unusual lexicon you may get better results with you own. Gillian Armstrong @virtualgill
65. Combine Intents and use Entities with synonyms and metonyms where needed. Remember entities are training data too. Include the right data in your entity data, and give good examples of where it would appear in your utterances. Gillian Armstrong @virtualgill
66. Find the right amount of utterances – not too many or too few. This is highly dependent on your context. The more intents you have, the more vital it is that the utterances in it are similar to each other, and different enough from other intents. Gillian Armstrong @virtualgill
67. Don’t just add in missed utterances without thought. It may modify other intents. Also – are you sure you know what the utterance should match? Does it need a new intent? Gillian Armstrong @virtualgill
68. Get a great set of tests – and run every time you make a change. Retraining changes your bot’s whole world. It may not know what it did before. Gillian Armstrong @virtualgill
69. How do I compensate for the Technology not being as quite there yet? Gillian Armstrong @virtualgill
70. At the moment it’s a lack of understanding of conversational interaction design that is killing chatbots, much more than poor NLU technology. Gillian Armstrong @virtualgill
71. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
72. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
73. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
74. Gillian Armstrong @virtualgill
75. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Gillian Armstrong @virtualgill
76. Got questions? Let’s chat! @virtualgill Demo Code at https://github.com/virtualgill/breakfast-bot
77. 77 Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies Liberty IT / Workgrid Software