ABN_AMRO-Automated_insights__uncovering_insights_from_customer_data_with_the_use_of_OpenAI..pptx.pdf

Transcript

1. ”Please Cheat At Your Job” – Inspired by Please Cheat At Your Job by Cassie Kozyrkov
2. Introduction 3-9-2025 | 2 Irin Otto Data Scientist Nelly Dua Data Scientist
3. Create opportunities from unknown data
4. Agenda ABN channels (human vs digital) Introduction Chatbot Anna Conversation structure: endpoints Results Evaluating Pros and Cons Security 3-9-2025 | 4 Automated Insights: unknown category
5. Interaction at scale 3-9-2025 | 5 1.500.000 Conversations via CIVR 600.000 Live Chats 7.000.000 Inbound calls 1.800.000 Outbound calls 235.000 Conversations via Video Banking 88.000 Social media messages 2.000.0000 Conversations via chatbot Anna 1.500.000 Automated summaries 40.000 Conversations via chatbot Rikkie
6. Chatbot Anna • Intent Classification • Scripted and Generative Dialogues • Ability to rephrase and digress to a different topic, or ask for a human agent Future plan: • Intent classification with GenAI • LLMaj for GenAI answers • Scale up the generative dialogues • Proactive and personalized conversations 3-9-2025 | 6
7. Endpoints 3-9-2025 | 7 Endpoint (OKR level) Definition Escalated The user question reached a handover (Un)resolved User that reached the usefulness question Unknown Rest case: we do not know if this is positive or negative
8. Create opportunities from unknown data
9. Automated Insights Process 3-9-2025 | 9 Useful Conversation Unknown Endpoint Not a Useful Conversation Possible endpoint outcomes Labelling OpenAI integration Prompting Monitoring Finetuning Approach
10. Results 3-9-2025 | 10
11. Evaluating the Pros and Cons Pros: 1. Accelerated labelling process 2. Consistent labelling results 3. Enhanced efficiency 4. Frees human labellers to focus on higher-level tasks 5. Easily adaptable for optimization 3-9-2025 | 11
12. Evaluating the Pros and Cons Cons: 1. Associated costs with OpenAI 2. Environmental considerations 3. Potential for bias 4. Garbage in, garbage out Personal Struggles: 1. OpenAI Access / Databricks 2. Ambiguity in labelling 3. Token limit 3-9-2025 | 12
13. Security • Handled PII data • Fully private data, not shared with anyone • Outputs used for optimization 3-9-2025 | 13
14. Agenda recap 3-9-2025 | 14 ABN channels (human vs digital) Introduction Chatbot Anna Conversation structure: endpoints Results Security Evaluating Pros and Cons Automated Insights: unknown category
15. Create opportunities from unknown data
16. Action steps • Try to find your blind spots/audit your unknowns • Label a small sample • Use AI for insights • Turn unknowns into learnings • Test and improve before scaling 3-9-2025 | 16
17. Connect with us on LinkedIn 3-9-2025 | 17 Irin Otto Nelly Dua
19. Title First level body text ∙ Second level – Third level • Fourth level 3-9-2025 Insert footer via the 'insert' tab | 19
20. Sample – two columns with comment box Lorem ipsum ∙ Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor ∙ Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus ∙ Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim ∙ Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum ∙ Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor ∙ Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus ∙ Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim ∙ Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor 03/09/2025 Insert footer via the 'insert' tab | 20
21. Sample – two columns with comment box 03/09/2025 Insert footer via the 'insert' tab | 21 Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Aenean commodo ligula eget dolor. Doughnut chart Column chart
22. Sample – two tables Table Company 1 Company 2 Product A ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product B ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product C ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product D ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product E ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Table Company 1 Company 2 Company 3 Product A 56 88 92 Product B 44 32 15 Product C 90 120 78 Product D 123 67 93 Product E 71 66 59 Product F 35 29 21 Product G 16 23 19 Product H 82 40 33 3-9-2025 Insert footer via the 'insert' tab | 22
23. Sample – two tables Table Company 1 Company 2 Product A ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product B ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product C ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product D ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx Product E ∙ Xxx ∙ Xxx ∙ Xxx ∙ Xxx 3-9-2025 Insert footer via the 'insert' tab | 23 Table Company 1 Company 2 Company 3 Product A 56 88 92 Product B 44 32 15 Product C 90 120 78 Product D 123 67 93 Product E 71 66 59 Product F 35 29 21 Product G 16 23 19 Product H 82 40 33
24. Sample – charts (5 parts, max) 3-9-2025 Insert footer via the 'insert' tab | 24
25. Sample – charts (5 parts, max) 3-9-2025 Insert footer via the 'insert' tab | 25
26. Sample – chart and world map 3-9-2025 Insert footer via the 'insert' tab | 26 Combined chart World map
27. Sample – four textboxes Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. 3-9-2025 Insert footer via the 'insert' tab | 27
28. Sample – four textboxes Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Lorem ipsum Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. 3-9-2025 Insert footer via the 'insert' tab | 28
29. Sample – organisation chart 3-9-2025 Insert footer via the 'insert' tab | 29 Lorem ipsum dolor aliquid ex ea commodi Lorem ipsum dolor aliquid ex ea commodi Lorem ipsum dolor aliquid ex ea commodi Lorem ipsum dolor aliquid ex ea commodi Lorem ipsum dolor aliquid ex ea commodi Lorem ipsum dolor aliquid ex ea commodi