How AI is changing User Experience design

AI is making new ways of interacting with technology possible, but understanding the new paradigms of human-computer interaction and user experience is critical to take full advantage of them.

1. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. How AI is changing User Experience design Gillian Armstrong Senior Specialist Builder Solutions Architect AWS
2. © 2022, Amazon Web Services, Inc. or its affiliates.
3. © 2022, Amazon Web Services, Inc. or its affiliates.
4. © 2022, Amazon Web Services, Inc. or its affiliates.
5. © 2022, Amazon Web Services, Inc. or its affiliates.
6. © 2022, Amazon Web Services, Inc. or its affiliates.
7. © 2022, Amazon Web Services, Inc. or its affiliates.
8. © 2022, Amazon Web Services, Inc. or its affiliates.
9. © 2022, Amazon Web Services, Inc. or its affiliates.
10. © 2022, Amazon Web Services, Inc. or its affiliates. Amazon Lex Alexa Conversational AI on Amazon / AWS
11. © 2022, Amazon Web Services, Inc. or its affiliates. How conversational AI works NLU NLP INTENTS I n t e r a c t i o n M o d e l SLOTS ASR Fulfilment CUX/VUX TTS Machine Learning
12. © 2022, Amazon Web Services, Inc. or its affiliates. Utterance (“what the user said”) Automatic Speech Recognition (Speech to Text) How conversational AI works NLU NLP INTENTS I n t e r a c t i o n M o d e l SLOTS ASR Mapping of Utterance to Intent (“what the user meant”) Extraction of Slots (variables like time, place, number, etc) Fulfilment Intent maps to Fulfillment (“what we need to do”) CUX/VUX TTS Machine Learning
13. © 2022, Amazon Web Services, Inc. or its affiliates. Utterance (“what the user said”) Automatic Speech Recognition (Speech to Text) How conversational AI works NLU NLP INTENTS I n t e r a c t i o n M o d e l SLOTS ASR Mapping of Utterance to Intent (“what the user meant”) Extraction of Slots (variables like time, place, number, etc) Fulfilment Intent maps to Fulfillment (“what we need to do”) CUX/VUX TTS Machine Learning
14. © 2022, Amazon Web Services, Inc. or its affiliates. Utterance (“what the user said”) Automatic Speech Recognition (Speech to Text) How conversational AI works NLU NLP INTENTS I n t e r a c t i o n M o d e l SLOTS ASR Mapping of Utterance to Intent (“what the user meant”) Extraction of Slots (variables like time, place, number, etc) Fulfilment Intent maps to Fulfillment (“what we need to do”) CUX/VUX Conversational / Voice User Experience (“how it sounds and the words it says”) TTS Text to Speech Machine Learning
15. © 2022, Amazon Web Services, Inc. or its affiliates. The new user experience 1. Mental Models 2. Explainability and Trust 3. Feedback and Control
16. © 2022, Amazon Web Services, Inc. or its affiliates. The new user experience 1. Mental Models
17. © 2022, Amazon Web Services, Inc. or its affiliates. Traditional web UX
18. © 2022, Amazon Web Services, Inc. or its affiliates. What about a chatbot?
19. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX For information about your orders press 1, to learn about Prime benefits press 2, to add a new address press 3………..
20. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX For information about your orders say orders, to learn about Prime benefits say prime, to add a new address say address……
21. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX For information about your orders say orders, to learn about Prime benefits say prime, to add a new address say address……
22. © 2022, Amazon Web Services, Inc. or its affiliates. Model your design on a human, not a menus, forms or websites People are the original conversational interface The new paradigm in UX
23. © 2022, Amazon Web Services, Inc. or its affiliates. The Web is about Telling the user what to do Conversational UI is about Listening to what the user wants to do The new paradigm in UX
24. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX I need to add a new address
25. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX I’ve moved house
26. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX Can I get packages delivered to my office too?
27. © 2022, Amazon Web Services, Inc. or its affiliates. Utterance (“what the user said”) Automatic Speech Recognition (Speech to Text) How conversational AI works NLU NLP INTENTS I n t e r a c t i o n M o d e l SLOTS ASR Mapping of Utterance to Intent (“what the user meant”) Extraction of Slots (variables like time, place, number, etc) Fulfilment Intent maps to Fulfillment (“what we need to do”) CUX/VUX Conversational / Voice User Experience (“how it sounds and the words it says”) TTS Text to Speech Machine Learning
28. © 2022, Amazon Web Services, Inc. or its affiliates. How Lex Interaction Model Works INTENT 1 • Example utterance • Example utterance • Example utterance INTENT 2 • Example utterance • Example utterance • Example utterance
29. © 2022, Amazon Web Services, Inc. or its affiliates. How Lex Interaction Model Works INTENT 1 • Example utterance • Example utterance • Example utterance INTENT 2 • Example utterance • Example utterance • Example utterance YourOrders • Where is my order? • When will my order be delivered? • Has my order been dispatched? YourAddresses • I need to add a new address • I’ve moved house • Can I get packages delivered to my office too?
30. © 2022, Amazon Web Services, Inc. or its affiliates. How Lex Interaction Model Works Input Expected Output Where is my order? YourOrders When will my order be delivered? YourOrders Has my order been dispatched? YourOrders I need to add a new address YourAddresses I’ve moved house YourAddresses Can I get packages delivered to my office too? YourAddresses Utterance (“what the user said”) Intent (“what the user meant”)
31. © 2022, Amazon Web Services, Inc. or its affiliates. How Lex Interaction Model Works Input Expected Output Where is my order number {orderId}? YourOrders When will my order be delivered? YourOrders Has my order {orderId} been dispatched? YourOrders I need to add a new address YourAddresses I’ve moved house YourAddresses Can I get packages delivered to my office too? YourAddresses Utterance (“what the user said”) Intent (“what the user meant”) Slot (variables like time, place, number, etc.) orderId ABC1234 HDNF112 A234ASD Slot Values (here we would likely just define this as AlphaNumberic or a regex)
32. © 2022, Amazon Web Services, Inc. or its affiliates. The new user experience 1. Mental Models 2. Explainability and Trust
33. © 2022, Amazon Web Services, Inc. or its affiliates. Web UX
34. © 2022, Amazon Web Services, Inc. or its affiliates. Confidence Score Action based on click location Prediction i.e. guess - (although hopefully a good one!) Confidence: 100% Confidence: ??%
35. © 2022, Amazon Web Services, Inc. or its affiliates. Machine learning Confidence Score A number between 0 and 1 that represents the likelihood that the output of a particular prediction from a ML model is correct Precision For a given test set what percentage of the items predicted as a particular answer were correct Recall For a given test set what percentage of the items that should have been a particular answer were correctly identified
36. © 2022, Amazon Web Services, Inc. or its affiliates. Precision vs Recall Phrase for add address Intent Phrase for something else Predicted as add address Intent Good precision Good recall
37. © 2022, Amazon Web Services, Inc. or its affiliates. Precision vs Recall e.g. • When to launch my rocket • Recommendations • Email spam filtering e.g. • Alert you of potential fraud on your card • Critical medical test • High risk content moderation Good precision Good recall When you really don’t want to get it wrong When you really don’t want to miss it
38. © 2022, Amazon Web Services, Inc. or its affiliates. Using Confidence Score e.g. • When to launch my rocket • Recommendations • Email spam filtering e.g. • Alert you of potential fraud on your card • Critical medical test • High risk content moderation Only use high confidence answers Use lower confidence answers When you really don’t want to get it wrong When you really don’t want to miss it
39. © 2022, Amazon Web Services, Inc. or its affiliates. Using Confidence Score Only use high confidence answers Ok to use lower confidence answers Cancelling your order Our lines are open 9am to 5pm every day Difficult to reverse actions should be confirmed first I’ll be angry if you get this wrong… The wrong answer is a less annoying than have to repeat if it was correct… Change the address on my last order What times can I call?
40. © 2022, Amazon Web Services, Inc. or its affiliates. Web UX
41. © 2022, Amazon Web Services, Inc. or its affiliates. Affordance – a title and breadcrumbs show me what page I am on I can see all the fields I need to fill out I have some prompts on how to fill the fields in
42. © 2022, Amazon Web Services, Inc. or its affiliates. What about a chatbot? What’s your phone number? I need to add a new address
43. © 2022, Amazon Web Services, Inc. or its affiliates. What about a chatbot? I need to add a new address Did it understand? What’s your phone number?
44. © 2022, Amazon Web Services, Inc. or its affiliates. What about a chatbot? To add a new address I’ll first need to get your phone number I need to add a new address
45. © 2022, Amazon Web Services, Inc. or its affiliates.
46. © 2022, Amazon Web Services, Inc. or its affiliates. You did this wrong!!!!
47. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX oh, umm, let me check which one I’ll put on this What’s your phone number?
48. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX oh, umm, let me check which one I’ll put on this That’s an incorrect phone number
49. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX The customer has never said the wrong thing – the chatbot just hasn’t understood
50. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX oh, umm, let me check which one I’ll put on this No problem. This will be the number we would contact you on if there was an issue with the delivery to that address.
51. © 2022, Amazon Web Services, Inc. or its affiliates. Conversational UX oh, umm, let me check which one I’ll put on this I’m sorry - I didn’t get the phone number. Can you repeat the phone number for the new address?
52. © 2022, Amazon Web Services, Inc. or its affiliates. The new user experience 1. Mental Models 2. Explainability and Trust 3. Feedback and Control
53. © 2022, Amazon Web Services, Inc. or its affiliates. Always keep your user in control Is your customer clear they are talking to a bot?
54. © 2022, Amazon Web Services, Inc. or its affiliates. Always keep your user in control I’d really prefer to talk to a human Does your bot understand how to give a customer other options? Is your customer clear they are talking to a bot?
55. © 2022, Amazon Web Services, Inc. or its affiliates. Always make sure you know what is happening too… Do you know how well your bot is working? Do you know what your customers are asking? Could you explain why your bot answered a certain way? Getting feedback both directly and through monitoring is critical to good ML system
56. © 2022, Amazon Web Services, Inc. or its affiliates. The new user experience 1. Mental Models 2. Explainability and Trust 3. Feedback and Control
57. © 2022, Amazon Web Services, Inc. or its affiliates. Take-aways • We need to break away from our traditional patterns – start thinking about how to make the computer understand the human rather than the other way around • An understanding on Machine Learning is required to build great User Experiences that use AI • The future is exciting!
58. © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Gillian Armstrong GillianQ@amazon.co.uk @virtualgill