Information Retrieval and Text Mining 2020 - Text Classification Evaluation

University of Stavanger, DAT640, 2020 fall

Transcript

1. Text Classifica on Evalua on [DAT640] Informa on Retrieval and Text Mining Krisz an Balog University of Stavanger September 1, 2020 CC BY 4.0
2. Evalua on • Measuring the performance of a classifier ◦ Comparing the predicted label y against the true label y for each document in some set dataset • Based on the number of records (documents) correctly and incorrectly predicted by the model • Counts are tabulated in a table called the confusion matrix • Compute various performance measures based on this matrix 2 / 34
3. Evalua ng binary classifica on 3 / 34
4. Confusion matrix Predicted class negative positive Actual negative true negatives (TN) false positives (FP) class positive false negatives (FN) true positives (TP) • False positives = Type I error (“raising a false alarm”) • False negatives = Type II error (“failing to raise an alarm”) 4 / 34
5. Question Which of Type I and Type II error is worse? 5 / 34
6. Type I vs. Type II errors1 1Source: https://www.analyticsindiamag.com/understanding-type-i-and-type-ii-errors/ 6 / 34
7. Example Id Actual Predicted 1 + - 2 + + 3 - - 4 + + 5 + - 6 + + 7 - - 8 - + 9 + - 10 + - predicted - + actual - + 7 / 34
8. Example Id Actual Predicted 1 + - 2 + + 3 - - 4 + + 5 + - 6 + + 7 - - 8 - + 9 + - 10 + - predicted - + actual - 2 + 8 / 34
9. Example Id Actual Predicted 1 + - 2 + + 3 - - 4 + + 5 + - 6 + + 7 - - 8 - + 9 + - 10 + - predicted - + actual - 2 1 + 9 / 34
10. Example Id Actual Predicted 1 + - 2 + + 3 - - 4 + + 5 + - 6 + + 7 - - 8 - + 9 + - 10 + - predicted - + actual - 2 1 + 4 3 10 / 34
11. Evalua on measures • Summarizing performance in a single number • Accuracy Fraction of correctly classified items out of all items ACC = TP + TN TP + TN + FP + FN • Error rate Fraction of incorrectly classified items out of all items ERR = FP + FN FP + FN + TP + TN predicted - + actual - TN FP + FN TP 11 / 34
12. Evalua on measures (2) • Precision Fraction of items correctly identified as positive out of the total items identified as positive P = TP TP + FP • Recall (also called Sensitivity or True Positive Rate) Fraction of items correctly identified as positive out of the total actual positives R = TP TP + FN predicted - + actual - TN FP + FN TP 12 / 34
13. Evalua on measures (3) • F1-score The harmonic mean of precision and recall F1 = 2 · P · R P + R predicted - + actual - TN FP + FN TP 13 / 34
14. Evalua on measures (4) • False Positive Rate (Type I Error) Fraction of items wrongly identified as positive out of the total actual negatives FPR = FP FP + TN • False Negative Rate (Type II Error) Fraction of items wrongly identified as negative out of the total actual positives FNR = FN FN + TP predicted - + actual - TN FP + FN TP 14 / 34
15. Example predicted - + actual - TN=2 FP=1 + FN=4 TP=3 ACC = TP + TN TP + TN + FP + FN = 5 10 = 0.5 P = TP TP + FP = 3 4 = 0.75 R = TP TP + FN = 3 7 = 0.429 F1 = 2 · P · R P + R = 2 · 3/4 · 3/7 3/4 + 3/7 = 0.545 15 / 34
16. Evalua ng mul class classifica on 16 / 34
17. Mul class classifica on • Imagine that you need to automatically sort news stories according to their topical categories comp.graphics rec.autos sci.crypt comp.os.ms-windows.misc rec.motorcycles sci.electronics comp.sys.ibm.pc.hardware rec.sport.baseball sci.med comp.sys.mac.hardware rec.sport.hockey sci.space comp.windows.x misc.forsale talk.politics.misc talk.religion.misc talk.politics.guns alt.atheism talk.politics.mideast soc.religion.christian Table: Categories in the 20-Newsgroups dataset 17 / 34
18. Mul class classifica on • Many classification algorithms are originally designed for binary classification • Two main strategies for applying binary classification approaches to the multiclass case ◦ One-against-rest ◦ One-against-one • Both apply a voting scheme to combine predictions ◦ A tie-breaking procedure is needed (not detailed here) 18 / 34
19. One-against-rest • Assume there are k possible target classes (y1, . . . , yk) • Train a classifier for each target class yi (i ∈ [1..k]) ◦ Instances that belong to yi are positive examples ◦ All other instances yj , j = i are negative examples • Combining predictions ◦ If an instance is classified positive, the positive class gets a vote ◦ If an instance is classified negative, all classes except for the positive class receive a vote 19 / 34
20. Example • 4 classes (y1, y2, y3, y4) • Classifying a given test instance (dots indicate the votes cast): y1 + • y1 - • y1 - • y1 - • y2 - y2 + y2 - • y2 - • y3 - y3 - • y3 + y3 - • y4 - y4 - • y4 - • y4 + Pred. + Pred. - Pred. - Pred. - • Sum votes received: (y1,••••), (y2,••), (y3,••), (y4,••) 20 / 34
21. One-against-one • Assume there are k possible target classes (y1, . . . , yk) • Construct a binary classifier for each pair of classes (yi, yj) ◦ k·(k−1) 2 binary classifiers in total • Combining predictions ◦ The predicted class receives a vote in each pairwise comparison 21 / 34
22. Example • 4 classes (y1, y2, y3, y4) • Classifying a given test instance (dots indicate the votes cast): y1 + • y1 + • y1 + y2 - y3 - y4 - • Pred. + Pred. + Pred. - y2 + • y2 + y3 + • y3 - y4 - • y4 - Pred. + Pred. - Pred. + • Sum votes received: (y1,••), (y2,•), (y3,•), (y4,••) 22 / 34
23. Question How to evaluate multiclass classification? Which of the evaluation measures from binary classification can be applied? 23 / 34
24. Evalua ng mul class classifica on • Accuracy can still be computed as ACC = #correctly classified instances #total number of instances • For other metrics ◦ View it as a set of k binary classification problems (k is the number of classes) ◦ Create confusion matrix for each class by evaluating “one against the rest” ◦ Average over all classes 24 / 34
25. Confusion matrix Predicted 1 2 3 . . . k Actual 1 24 0 2 0 2 0 10 1 1 3 1 0 9 0 . . . k 2 0 1 30 25 / 34
26. Binary confusion matrices, one-against-rest Predicted 1 2 3 . . . k Actual 1 24 0 2 0 2 0 10 1 1 3 1 0 9 0 . . . k 2 0 1 30 For the sake of this illustration, we assume that the cells which are not shown are all zeros. ⇒ Predicted 1 ¬1 Act. 1 TP=24 FN=3 ¬1 FP=2 TN=52 Predicted 2 ¬2 Act. 2 TP=10 FN=2 ¬2 FP=0 TN=69 . . . 26 / 34
27. Averaging over classes • Averaging can be performed on the instance level or on the class level • Micro-averaging aggregates the results of individual instances across all classes ◦ All instances are treated equal • Macro-averaging computes the measure independently for each class and then take the average ◦ All classes are treated equal 27 / 34
28. Micro-averaging • Precision Pµ = k i=1 TPi k i=1 (TPi + FPi) • Recall Rµ = k i=1 TPi k i=1 (TPi + FNi) • F1-score F1µ = 2 · Pµ · Rµ Pµ + Rµ predicted i ¬i actual i TPi FNi ¬i FPi TNi 28 / 34
29. Macro-averaging • Precision PM = k i=1 TPi TPi+FPi k • Recall RM = k i=1 TPi TPi+FNi k • F1-score F1M = k i=1 2·Pi·Ri Pi+Ri k ◦ where Pi and Ri are Precision and Recall, respectively, for class i predicted i ¬i actual i TPi FNi ¬i FPi TNi 29 / 34
30. Model development 30 / 34
31. Question How can we evaluate the performance of the model during development? 31 / 34
32. Using a valida on set • Idea: hold out part of the training data for testing into a validation set • Single train/validation split ◦ Split the training data into X% training split and 100 − X% validation split (an 80/20 split is common) 32 / 34
33. Using a valida on set2 • k-fold cross-validation ◦ Partition the training data randomly into k folds ◦ Use k − 1 folds for training and test on the kth fold; repeat k times (each fold is used for testing exactly once) ◦ k is typically 5 or 10 ◦ Extreme: k is the number of data points, to maximize the number of training material available (called “leave-one-out” evaluation) 2Image source: http://ethen8181.github.io/machine-learning/model_selection/model_selection.html 33 / 34
34. Reading • Text Data Management and Analysis (Zhai&Massung), Chapter 15 ◦ Section 15.6 34 / 34