Making Classification Error and Bias Transparent and Understandable

Transcript

1. MAKING
 CLASSIFICATION ERROR & BIAS
 TRANSPARENT & UNDERSTANDABLE
2. https://www.excavating.ai/
3. https://royalsociety.org/science-events-and-lectures/2018/07/you-and-ai-equality/
4. https://royalsociety.org/science-events-and-lectures/2018/07/you-and-ai-equality/
6. Bias can be
 desirable… …if it represents
 reality …if it makes
 better classifiers
7. We must scrutinise
 classification errors
8. STEP 1 DEFINE 
 ERROR & BIAS Disclaimer: 
 Does not discuss the (human) sources of bias.
9. ERROR What belongs to Class A It is directional: objects from Class A are misclassified into Class B. Belongs to Class A is classified as Class B. Classified as Class B ERROR Classified as Class A A -> B ERROR ? -> A ERROR A -> ?
10. BIAS (1) Errors systematically occur for one class. Belong to Class A Classified as 
 Class A BIAS
11. BIAS (2) Errors systematically differ between classes. Belong to Class A Belong to 
 Class B Classified as 
 Class A Classified
 as Class B BIASED
 AGAINST
12. Female 
 belonging to Class A BIASED
 AGAINST Male 
 belonging to Class A Classified as 
 Class A Classified as 
 Class A BIAS (3) Errors systematically differ between groups.
13. BIAS (4) Groups are under-represented. Real Proportion BIASED
 AGAINST Male 
 belonging to Class A Female 
 belonging to Class A
14. BIAS (5) Real Proportion Belong to
 Class A Belong to 
 Class B BIASED
 AGAINST Classes are under-represented.
15. STEP 2 MEASURE 
 ERROR & BIAS Warning: 
 Contains math.
16. Error Rate = Number of Error On What ?! ON WHAT?! [1] Hoffrage, Krauss, Martignon, and Gigerenzer (2015) 
 [2] Khan, Breslav, Glueck, and Hornbæk (2015) [3] Beauxis-Aussalet, van Doorn, and Hardman (2018) THAT’S VERY IMPORTANT… BEWARE OF ERROR RATES We may forget what denominators represent.
 (experts or non-experts)
17. Precision = Number of Objects Correctly Classified as X Total Number of Objects Classified as X Belongs to Class A Classified as Class B ERROR BEWARE OF PRECISION If class proportions vary, precision varies.
18. Precision = TP TP + FP = Classified as Class B TP FP BEWARE OF PRECISION If class proportions vary, precision varies.
19. Precision = TP TP + FP = Classified as Class B Belong to Class A TP FP BEWARE OF PRECISION If class proportions vary, precision varies.
20. Precision = TP TP + FP = Classified as Class B Belong to Class A TP FP BEWARE OF PRECISION If class proportions vary, precision varies.
21. Precision = TP TP + FP = Belong to Class B Belong to Class A FP TP Classified as Class B BEWARE OF PRECISION If class proportions vary, precision varies.
22. Precision = TP TP + FP = Belong to Class A FP TP Classified as Class B FP Belong to Class B BEWARE OF PRECISION If class proportions vary, precision varies.
23. Precision = TP TP + FP = Belong to Class B Belong to Class A FP TP Classified as Class B BEWARE OF PRECISION If class proportions vary, precision varies.
24. Precision = TP TP + FP = Belong to Class B FP TP Classified as Class B Belong to Class A TP BEWARE OF PRECISION If class proportions vary, precision varies.
25. TP Rate = TP TP + FN = Belong to Class B TP Classified as Class B Belong to Class A TP FN FN BEWARE OF CLASS PROPORTIONS Some error rates remain unchanged by class proportions.
 (e.g., those of ROC curves)
26. TP Rate = TP TP + FN = TP Classified as Class B Belong to Class A FN Belong to Class B BEWARE OF CLASS PROPORTIONS Some error rates remain unchanged by class proportions.
 (e.g., those of ROC curves)
27. Accuracy = Number of Correct Classifications Total Number of Classifications = + + Example: We achieve 98% accuracy, but one class is entirely misclassified 
 (a small class ≤ 2% of all objects). BEWARE OF ACCURACY It fails to indicate the errors for specific classes or groups.
 It varies with class proportions. Biases cannot be identified.
28. STEP 3 TUNE 
 ERROR & BIAS 
 No math, yet.
29. TUNING CLASSIFICATION ERRORS Test sets contain examples of correct classifications, to measure the errors. Examples of Happy Faces Examples of 
 Sad Faces MISCLASSIFIED
30. Happiness Score TUNING CLASSIFICATION ERRORS Classifiers have tuning parameters to balance errors between classes.
31. Happiness Score TUNING CLASSIFICATION ERRORS Classifiers have tuning parameters to balance errors between classes.
32. Happiness Score TUNING CLASSIFICATION ERRORS Classifiers have tuning parameters to balance errors between classes.
33. TUNING CLASSIFICATION ERRORS sad happy Happiness Score Classifiers have tuning parameters to balance errors between classes.
34. sad happy Happiness Score TUNING CLASSIFICATION ERRORS Classifiers have tuning parameters to balance errors between classes.
35. sad happy Happiness Score TUNING CLASSIFICATION ERRORS Classifiers have tuning parameters to balance errors between classes.
36. sad happy Happiness Score TUNING CLASSIFICATION ERRORS Class proportions may vary in real life.
37. sad happy Happiness Score TUNING CLASSIFICATION ERRORS Class proportions may vary in real life.
38. damaged undamaged Donut Score TUNING CLASSIFICATION ERRORS Tuning decisions depend on use cases.
39. damaged undamaged Donut Score TUNING CLASSIFICATION ERRORS Tuning decisions depend on use cases.
40. STEP 3 TUNE VISUALISE 
 ERROR & BIAS
41. Cells are tedious to locate & compare [1] Beauxis-Aussalet, van Doorn, and Hardman (2018) Columns & rows are confused for on another Columns or rows are omitted even by experts CONFUSION MATRIX
42. [1] Beauxis-Aussalet, van Doorn, and Hardman (2018) Rates on both axes are unusual Lines are like axes too Class sizes are unknown Rate formulas are hard to remember Denominators are not specified Labels are ambiguous (TP, FP)
43. Classified as Class B Belong to Class A Belong to Class B
44. ERRO R OK ERROR OK
46. Belong to Class A Belong to Class B Classified as Class B Classified as Class A
47. 100 100 200 300 400 0 Number of classified items Actual Class Predicted Class
48. 100 100 200 300 400 0 Number of classified items Actual Class Predicted Class
49. [1] Beauxis-Aussalet, van Doorn, and Hardman (2019)
50. CLASSEE VISUALIZATION
51. Classified as Negative | Classified as Positive −1000 −500 0 500 1000 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.2 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.3 0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 Threshold Number of Objects Legend TP FN FP TN CLASSEE VISUALIZATION
52. Classified as Negative | Classified as Positive −1000 −500 0 500 1000 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.2 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.3 0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63 0.64 Threshold Number of Objects Legend TP FN FP TN It is more intuitive. It helps to cope with the terminology
 (TP, FN, FP, TN) CLASSEE VISUALIZATION
53. R CODE github.com/emma-ba/Classification_Error_Toolkit/
54. DATA FORMATING github.com/emma-ba/Classification_Error_Toolkit/
55. MORE TEMPLATES github.com/emma-ba/Classification_Error_Toolkit/
56. CLASSEE FOR MULTICLASS
57. CLASSEE FOR MULTICLASS
58. CLASSEE FOR MULTICLASS
59. CLASSEE FOR MULTICLASS
60. Sign B Sign C Sign A Sign D Sign N Sign M Sign L Sign K Sign J Sign I Sign H Sign G Sign F Sign E Sign O Sign Q Sign R Sign P Sign S Sign AC Sign AB Sign AA Sign Z Sign Y Sign X Sign W Sign V Sign U Sign T Sign AD Sign AF Sign AG Sign AH Sign AI Sign AE Sign AK Sign AJ Sign AQ Sign AP Sign AO Sign AN Sign AM Sign AL 100% 200% 300% 400% Detected Missed Legend: -300% -200% -100% -400 -200 200 400 600 800 Sign B Sign C Sign A Sign D Sign N Sign M Sign L Sign K Sign J Sign I Sign H Sign G Sign F Sign E Sign O Sign Q Sign R Sign P Sign S Sign AC Sign AB Sign AA Sign Z Sign Y Sign X Sign W Sign V Sign U Sign T Sign AD Sign AF Sign AG Sign AH Sign AI Sign AE Sign AK Sign AJ Sign AQ Sign AP Sign AO Sign AN Sign AM Sign AL Legend: Numbers Proportions Black and red blocks represent the same objects. 
 It is unusual. Legends are difficult to understand. A learning step is required. CLASSEE FOR MULTICLASS
61. R CODE github.com/emma-ba/Classification_Error_Toolkit/
62. D3 CODE http://classee.project.cwi.nl/components/
63. D3 TOOL http://classee.project.cwi.nl/workspaces/
64. D3 TOOL http://classee.project.cwi.nl/workspaces/
65. D3 TOOL http://classee.project.cwi.nl/workspaces/
66. D3 TOOL http://classee.project.cwi.nl/workspaces/
67. D3 TOOL http://classee.project.cwi.nl/workspaces/
68. D3 TOOL http://classee.project.cwi.nl/workspaces/
69. STEP 4 ERROR & BIAS 
 IN PRACTICE Warning: 
 Contains math.
70. Let’s have the AI do
 your blood test… Time Classes of cells over time Number of Cells
71. Time Classes of cells over time Number of Cells Class A increases… Class B increases too… Error from Class A to Class B increase too… Does Class B increase due to errors from Class A ? Within the things classified as Class B 
 how many belong to Class A ?
72. How many errors
 in the end-results? Time Classes of cells over time Number of Cells
73. Reclassification Method [1] Tenenbein (1970) 
 [2] Hay (1989) 
 [3] Beauxis-Aussalet and Hardman (2017) Misclassification Method PRECISION ROC OLD MATH FROM THE 70’S
74. Reclassification Method PRECISION OLD MATH FROM THE 70’S Not applicable if class proportions differ 
 between test and target sets Time Classes of cells over time Number of Cells
75. OLD MATH FROM THE 70’S High variance means that the test set is too small. The methods remain valid. c) Segment d) Ohscal 50 1 C1 C 50 1 C1 C2 C3 C4 C5 C6 C7 50 1 C1 C2 C3 C4 C5 C6 C7 C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Misclassification Method C0 C1 C C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Classifier Results 1000 1400 Reclassification Method 1000 1400 Misclassification Method 1000 1400 Classifier Results 1000 1200 1000 1200 Classifier Results Mis 100 200 300 400 500 600 700 c) Segment d) Ohscal 50 1 C1 C2 C3 C4 C5 C6 C7 C6 C7 50 1 C1 C2 C3 C4 C5 C6 C7 C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Misclassification Method C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Reclassification Method C6 C7 C8 C9 ults 1000 1400 Reclassification Method sification Method 1000 1200 1000 1200 1000 1200 Reclass. Meth. Classifier Results Misclass. Meth. c) Segment d) Ohscal 50 1 C1 C2 C3 C4 C5 C6 C7 50 1 C1 C2 C3 C4 C5 C6 C7 50 1 C1 C2 C3 C4 C5 C6 C7 C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Misclassification Method C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Reclassification Method C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Classifier Results 1000 1400 Reclassification Method 1000 1400 Misclassification Method 1000 1400 Classifier Results 1000 1200 1000 1200 1000 1200 Reclass. Meth. Classifier Results Misclass. Meth. 100 200 300 400 500 600 700 c) Segment d) Ohscal 50 100 C1 C2 C3 C4 C5 C6 C7 C7 50 100 C1 C2 C3 C4 C5 C6 C7 C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Misclassification Method C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 Reclassification Method C7 C8 C9 000 1400 Reclassification Method ation Method 1000 1200 1000 1200 1000 1200 Reclass. Meth. Classifier Results Misclass. Meth. [1] Beauxis-Aussalet and Hardman (2017)
76. STEP 5 VARIANCE 
 THE MISSING PIECE Warning: 
 Contains math.
77. Errors may differ between classes or groups, but is it systematic? ? BIASED AGAINST WE NEED VARIANCE TO ASSESS BIAS
78. [1] Cochran (1977) Smaller classes have larger variance. BEWARE OF CLASS SIZES
79. Target Set 
 belonging to Class A Classified as 
 Class A Classified as 
 Class A Test Set 
 belonging to Class A BEWARE OF CLASS SIZES IN END-RESULTS Class sizes in both test & target sets impact the variance of error estimates.
80. Population 
 belonging to 
 Class A Samples 
 belonging to 
 Class A BEWARE OF CLASS SIZES IN END-RESULTS Class sizes in both test & target sets impact the variance of error estimates.
81. [1] Beauxis-Aussalet and Hardman (2017) SAMPLE 
 TO 
 SAMPLE BEWARE OF CLASS SIZES IN END-RESULTS Class sizes in both test & target sets impact the variance of error estimates.
82. COURSE COMING UP! Get the details in our next newsletters (scroll down our website). https://digitalsocietyschool.org/courses/
83. RECAP 
 This talk is finishing soon. A MINIMAL CHECKLIST
84. Do classification results systematically differ from the truth? Do errors from Class A to Class B systematically occur? Do they significantly differ between classes and groups? Can they significantly vary among target sets? Can they significantly vary among target sets? Can the test set provide reliable error estimates? Is the test set representative of all classes and groups? Inherent Bias: Error Direction: Error Magnitudes: Class Proportions: Group Proportions: Error Variance: Sample Validity:
85. Bias & Error
 in End-Results Classification 
 Errors Ground-Truth Quality Classification 
 Errors Data Quality Ground-Truth Quality Classification 
 Errors Data Quality Ground-Truth Quality Data Quality Classification 
 Errors Conditions
 of Application
86. THE END EMMA BEAUXIS-AUSSALET e.m.a.l.beauxis@hva.nl @emma_beauxis
 github.com/emma-ba/Classification_Error_Toolkit/