Natural Language (mis)Understanding

Natural Language (mis)Understanding

As a developer I spend a lot of time being cross that a computer did what asked it to, and not what I meant it to. With all the AI services out there allowing Natural Language Processing and Understanding, surely the time has come when computers can finally really understand us? Let’s take a look at how easy it is to build a chatbot using an NLU service, and how easy it is to break it if you haven’t thought through your interaction model design and training data carefully. We may need other humans for a while yet…

705df02f597ff36f900251c749e879cf?s=128

Gillian Armstrong

April 22, 2018
Tweet

Transcript

  1. Natural Language (mis)Understanding Gillian Armstrong

  2. 2 Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies

    Liberty IT / Workgrid Software
  3. Will a chatbot take my job? Does your job involve

    talking to people? Not if I can help it Yes Do you end up saying the same thing over and over again? Yes No Did anyone actually ask you? Yes, that’s totally my whole job High Risk of co-workers replacing you with small shell script Unlikely to be a chatbot that takes your job Not any time soon Well, no, but… Gillian Armstrong @virtualgill
  4. What is Natural Language Understanding?? Gillian Armstrong @virtualgill

  5. Gillian Armstrong @virtualgill

  6. Gillian Armstrong @virtualgill

  7. DICTIONARY GRAMMAR Gillian Armstrong @virtualgill

  8. over /ˈəʊvə/ preposition Gillian Armstrong @virtualgill

  9. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  10. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  11. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  12. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  13. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  14. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  15. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  16. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  17. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  18. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  19. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  20. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  21. Gillian Armstrong @virtualgill

  22. CAT DOG ZEBRA Gillian Armstrong @virtualgill

  23. • He drove over the bridge • The house is

    just over the hill • There were ants all over me • New shops were opening up all over the town • She climbed over the wall • I’m all over it • I’m so over it • The film was finally over • She has a lot of power over him • He’s not overly friendly • I was over the credit limit • They were all over me to get the latest gossip • The fence fell over • The picture hangs over the piano • I was over a team of 25 people • We went over the answers • We are going to have to start over • He practised the song over and over • Who would choose ice cream over cake? • She went over and above • He handed over the keys • The airplane flew over the city • He handed over responsibility for the project • Look over here • He looked over the evidence • She’s somewhere over there • We grew wiser over time • He worried over the decision to be made • We had over 20 choices • His explanation went totally over my head • I was in over my head • She threw it over the wall • To hide it, we hung the picture over the hole in the wall. Gillian Armstrong @virtualgill
  24. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR

    Gillian Armstrong @virtualgill
  25. The bird flew over the wall The ball flew over

    the wall The flag flew over the wall Gillian Armstrong @virtualgill
  26. The bird flew over the wall The ball flew over

    the wall The flag flew over the wall Gillian Armstrong @virtualgill
  27. The bird flew over the wall, circling ominously as if

    waiting to see if I would lose my precarious hold and fall to my death. He loaded the ball into the cannon and lit the fuse. The ball flew over the wall and soared off into the distance. ‘Watch out’, he called, tossing the flag. The flag flew over the wall. I sprung forward just in time to catch it. Gillian Armstrong @virtualgill
  28. Sometimes balls do fly over things by themselves Gillian Armstrong

    @virtualgill
  29. Understanding Meaning is more than just understanding Words DICTIONARY GRAMMAR

    Gillian Armstrong @virtualgill
  30. Gillian Armstrong @virtualgill

  31. Current State NLU Gillian Armstrong @virtualgill

  32. Gillian Armstrong @virtualgill

  33. Utterance Gillian Armstrong @virtualgill

  34. MAKE TEA Intent Utterance MAKE COFFEE MAKE TOAST Gillian Armstrong

    @virtualgill
  35. Get me a coffee Bring me a cup of coffee

    Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like a wee brew Make me a pot of tea Bring me a cup of tea Bring me tea I really need a cuppa Make me some tea Bring me toast I would like toast and jam Toast me some bread Get me some toast I would like some toast Intent Utterance Gillian Armstrong @virtualgill
  36. Get me a coffee Bring me a cup of coffee

    Brew me up some coffee Get me a cup of coffee Make me a coffee I’d like a wee brew Make me a pot of tea Bring me a cup of tea Bring me tea I really need a cuppa Make me some tea Bring me toast I would like toast and jam Toast me some bread Get me some toast I would like some toast Intent Fulfillment Utterance Gillian Armstrong @virtualgill
  37. Demo using Amazon Lex Bot Code at https://github.com/virtualgill/breakfast-bot Gillian Armstrong

    @virtualgill
  38. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast V1 Gillian Armstrong @virtualgill
  39. V1 Gillian Armstrong @virtualgill

  40. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast V1 Gillian Armstrong @virtualgill
  41. Gillian Armstrong @virtualgill

  42. Gillian Armstrong @virtualgill

  43. V1 Gillian Armstrong @virtualgill

  44. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V2 Gillian Armstrong @virtualgill
  45. V2 Gillian Armstrong @virtualgill

  46. V2 Gillian Armstrong @virtualgill

  47. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Fetch me a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V3 Gillian Armstrong @virtualgill
  48. V3 Gillian Armstrong @virtualgill

  49. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea I need a cuppa Fetch me a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V3 Gillian Armstrong @virtualgill
  50. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Fetch me a cuppa I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V4 Gillian Armstrong @virtualgill
  51. V4 Gillian Armstrong @virtualgill

  52. Natural Language Understanding, however clever, doesn’t really involve understanding natural

    language. Gillian Armstrong @virtualgill
  53. Gillian Armstrong @virtualgill

  54. Gillian Armstrong @virtualgill

  55. MAKE TEA MAKE COFFEE MAKE TOAST I’d like some tea

    Make me a pot of tea Bring me a cup of tea Bring me tea Get me a cup of tea Make me some tea I would like some tea I would like tea Fetch me a cuppa I need a cuppa Bring me a cup of coffee Brew me up some coffee Get me a cup of coffee Make me a coffee Get me a coffee Get me a coffeee Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V4 Gillian Armstrong @virtualgill
  56. TEA COFFEE TOAST Gillian Armstrong @virtualgill

  57. TEA COFFEE TOAST Gillian Armstrong @virtualgill

  58. DRINKS TOAST Gillian Armstrong @virtualgill

  59. MAKE HOT DRINK MAKE TOAST I’d like some {DRINK} Make

    me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} Get me a cup of {DRINK} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Bring me toast I would like toast Toast me some bread Get me some toast I would like some toast I'd like toast V5 Gillian Armstrong @virtualgill
  60. MAKE BREAKFAST I’d like some {DRINK} I’d like some {DRINK}

    and {FOOD} Make me a pot of {DRINK} Bring me a cup of {DRINK} Bring me {DRINK} and {FOOD} Get me a cup of {DRINK} and some {FOOD} Make me some {DRINK} I would like some {DRINK} I would like {DRINK} and {FOOD} Fetch me a {DRINK} I need a {DRINK} Brew me up some {DRINK} Get me some {FOOD} I would like some {FOOD} I'd like {FOOD} V6 Gillian Armstrong @virtualgill
  61. Some very quick tips Gillian Armstrong @virtualgill

  62. Make sure you do your analysis up front to really

    understand what your intents should be. It may not be what you initially think! Gillian Armstrong @virtualgill
  63. You wouldn’t make up other training data for ML –

    don’t make up your conversational data either. Get it from your users. Gillian Armstrong @virtualgill
  64. Decide if you want to use a service or roll

    your own. If you have a small closed domain with an unusual lexicon you may get better results with you own. Gillian Armstrong @virtualgill
  65. Combine Intents and use Entities with synonyms and metonyms where

    needed. Remember entities are training data too. Include the right data in your entity data, and give good examples of where it would appear in your utterances. Gillian Armstrong @virtualgill
  66. Find the right amount of utterances – not too many

    or too few. This is highly dependent on your context. The more intents you have, the more vital it is that the utterances in it are similar to each other, and different enough from other intents. Gillian Armstrong @virtualgill
  67. Don’t just add in missed utterances without thought. It may

    modify other intents. Also – are you sure you know what the utterance should match? Does it need a new intent? Gillian Armstrong @virtualgill
  68. Get a great set of tests – and run every

    time you make a change. Retraining changes your bot’s whole world. It may not know what it did before. Gillian Armstrong @virtualgill
  69. How do I compensate for the Technology not being as

    quite there yet? Gillian Armstrong @virtualgill
  70. At the moment it’s a lack of understanding of conversational

    interaction design that is killing chatbots, much more than poor NLU technology. Gillian Armstrong @virtualgill
  71. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  72. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  73. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics
  74. Gillian Armstrong @virtualgill

  75. HUMANS TECHNOLOGY HUMAN-COMPUTER INTERACTION psychology emotion mental models mental processing

    expectation & affordance memory & attention social AI product design copywriting script writing storytelling conversational design voice design personality design creativity imagination language analysis conversational & voice ux data science ML NLP NLU AI cloud services software engineering soundscape design cognitive & socio-linguistics Gillian Armstrong @virtualgill
  76. Got questions? Let’s chat! @virtualgill Demo Code at https://github.com/virtualgill/breakfast-bot

  77. 77 Liberty Information Technology Gillian Armstrong @virtualgill Technologist, Cognitive Technologies

    Liberty IT / Workgrid Software