A tester's journey in the world of machine learning

Machine learning (ML) has become the epicenter of software in today’s era. Tremendous efforts are put into developing ML-based systems. But what about ensuring the quality of such applications? What role can testers play in this context?

Usually, testing takes a backseat here and testers struggle to find their role and contributions they could make. The perception remains that “It’s machine learning, it cannot be tested”. But is this really the case? If not, how can we break this myth? How can testers make their space in the ML world?

In this presentation, Shivani explores answers to such questions and many more by sharing her experience of testing ML-based applications without having any prior experience of it. She’ll reflect on her motivation, challenges faced, learnings and techniques to test such systems. Get to know how a tester's skill set like product knowledge, attention to details, thinking out of the box and customer focus can be game changer here.

Transcript

1. A tester’s journey in the world of machine learning @shivani_gaba
2. @shivani_gaba Who am I?
3. @shivani_gaba
5. @shivani_gaba
6. @shivani_gaba
7. @shivani_gaba Amazon Recruitment tool Google object recognition Source – The Verge, theladders Microsoft Tay Chatbot
9. @shivani_gaba Personalised article recommendations
11. @shivani_gaba How do I test this
12. @shivani_gaba
13. Asked my friend – Google! @shivani_gaba
14. I don’t get it! Recall Precision PR-AUC Confusion matrix @shivani_gaba
15. @shivani_gaba
16. @shivani_gaba
17. Quality Assurance Data Science End User Perspective Domain Expert @shivani_gaba
18. Challenge status quo Being a Question Asker Think out of the box @shivani_gaba Apply product knowledge Understand the structure & building blocks
19. @shivani_gaba
20. Enabling machines to learn from data & algorithms without being explicitly programmed @shivani_gaba Machine learning
21. Supervised learning Unsupervised learning ? Model Shivani Labeled data Model Input data Shivani Shivani Shivani Shivani
22. The model learns from past labelled input data of what articles user clicked. @shivani_gaba Supervised learning (for article recommendation) And then predicts for future recommendations that are likely to be clicked.
23. How do model learn? @shivani_gaba
24. Step 1: Collect DATA Step 2: Select model FEATURES Step 3: Train MODEL @shivani_gaba Learning process
25. Step 1: Collect DATA Step 2: Select model FEATURES Step 3: Train MODEL @shivani_gaba Learning process
26. ………………………….….. Source 1 Source 2 Source 3 Source n Collected historical data @shivani_gaba
27. The higher the quantity of data, the better the training
28. = Principle of SISO
29. The higher the quantity & QUALITY of data, the better the training
30. Missing values Inconsistent data types Duplicates Domain knowledge Security Data privacy Biases (completeness) Outlier data ...... @shivani_gaba
31. User Id User Gender Article id Article age User Clicked? 1 Male 20 100000 Yes 2 Male 30 30 Yes 2 Male 30 30 Yes 0 Male 40 20 No 4 Male 70 Null No 5 Female 98 -2 Yes 5 Female 86 -3 No Duplicates Missing values Invalid data Outlier Bias Domain knowledge
32. User Id User Gender Article id Article age … User Clicked? 1 Male 20 100000 …. Yes 2 Male 30 30 …. Yes 2 Male 30 30 …. Yes 0 Male 40 20 …. No 4 Male 70 Null …. No 5 Male 98 -2 …. Yes 5 Female 86 -3 …. No … …. …. …. …. ….
33. @shivani_gaba Negative age Outlier age
34. @shivani_gaba
35. Step 2: Select model FEATURES Step 3: Train MODEL @shivani_gaba Step 1: Collect DATA Learning process
36. @shivani_gaba - Review & enhance features - Standardize rules of features - Challenge the current features - Document features Feature engineering Features are the independent variables/attributes fed to the models.
37. User Id User Gender Article id Article age User Clicked? 1 Male 20 10 Yes 2 Female 30 30 Yes 3 Non binary 55 30 Yes … … … … …
38. User Id User Gender Article id Article age Total article views Total article clicks User Clicked? 1 Male 20 10 90000 600 Yes 2 Female 30 30 80000 200 Yes 3 Non binary 55 30 30000 500 Yes … … … … … … … à 2 views à 1 click
39. User Id User Gender Article id Article age Total article views Total article clicks User Clicked? 1 Male 20 10 3000 600 Yes 2 Female 30 30 1000 200 Yes 3 Non binary 55 30 2000 500 Yes … … … … … … …
40. User Id User Gender Article id Article age Total article views Total article clicks Article clicks/ views User Clicked? 1 Male 20 10 3000 600 0.50 Yes 2 Female 30 30 1000 200 0.20 Yes 3 Non binary 55 30 2000 500 0.40 Yes … … … … … … … …
41. User Id User Gender Article id Article age Total article views Total article clicks Article clicks/ views User Clicked? 1 Male 20 10 3000 600 0.50 Yes 2 Female 30 30 1000 200 0.20 Yes 3 Non binary 55 30 2000 500 0.40 Yes … … … … … … … … Model Features Target(label)
42. Step 2: Select model FEATURES Step 3: Train the MODEL @shivani_gaba Step 1: Collect DATA
43. User Id User Gender Article id Article age Total article impressions Total article clicks Article click/view ratio User Clicked? 1 Male A4 10 1000 500 0.50 Yes 2 Female A3 30 2000 600 0.30 Yes 2 Female A5 30 3000 600 0.50 Yes 3 Male A4 20 4000 120 0.03 No 4 Male A2 50 5000 300 0.06 No 5 Non binary A4 18 700 80 0.11 Yes 5 Non binary A6 2 5000 540 0.10 Yes 6 Female A4 3 3400 340 0.10 No 7 Male A7 6 800 80 0.10 Yes
44. User Id User Gender Article id Article age Total article impressions Total article clicks Article click/view ratio User Clicked? 1 Male A4 10 1000 500 0.50 Yes 2 Female A3 30 2000 600 0.30 Yes 2 Female A5 30 3000 600 0.50 Yes 3 Male A4 20 4000 120 0.03 No 4 Male A2 50 5000 300 0.06 No 5 Non binary A4 18 700 80 0.11 Yes 5 Non binary A6 2 5000 540 0.10 Yes 6 Female A4 3 3400 340 0.10 No 7 Male A7 6 800 80 0.10 Yes Training set Testing set
45. Training set Testing set Model Algorithm Evaluation Prediction
46. @shivani_gaba Age least important feature Feature importance User industry Total article clicks Article age Total article views User gender …………………………………
47. Step 1: Collected DATA Step 2: Selected model FEATURES Step 3: Trained MODEL @shivani_gaba
48. @shivani_gaba What next?
49. User makes requests Predict relevance Return results @shivani_gaba Delivery process
50. Already seen articles ………….. …….… ..…..… 10,000 users Top X articles Top X articles Average age # of overlap articles Diverse sources Diverse types Old version New version Out of the box test idea
51. Recap
52. Lookback
54. Developers bringing up test ideas
55. @shivani_gaba Onboarded other testers
56. What about me?
57. Is there a need for testers to be involved in testing AI/ML based project in today’s era?
58. My new friend- ChatGPT
59. The era we live in! @shivani_gaba
60. @shivani_gaba
61. Thank you
62. Q&A