JGS594 Lecture 07

Software Engineering for Machine Learning
Performance Measurements
(202202)

Transcript

1. jgs SER 594 Software Engineering for Machine Learning Lecture 07: Performance Measurement Dr. Javier Gonzalez-Sanchez javiergs@asu.edu javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment
2. jgs Previously …
3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3 jgs ND4J Input
4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4 jgs DL4J | Our Model
5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5 jgs DL4J | Training
6. jgs Confusion Matrix
7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7 jgs Definition A summary of prediction results on a classification problem
8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8 jgs Definition A summary of prediction results on a classification problem TP FP FN TN positive 0 negative 1 positive 0 negative 1
9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9 jgs Definition TP FP FN TN positive 1 negative 0 positive 1 negative 0 FP Type 1 Error It is FALSE Computer said TRUE FP Type 2 Error It is TRUE Computer said FALSE
10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10 jgs Definition What about not binary classifiers? e.g. emotion recognition TP FP FN TN positive 😀 😡 . 🙁 negative positive 😀 😡. 🙁 negative FN TN FP TN TN
11. jgs Accuracy
12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12 jgs Definition § overall measure of how much the model is correctly predicting on the entire set of data § Addition of the elements in the main diagonal divide by the sum of all the entries of the confusion matrix at the denominator.
13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13 jgs Accuracy Accuracy = TP + TN / TP + TN + FP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Accuracy = TP + TN / TP + TN + FP + FN + X
15. jgs Precision and Recall
16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16 jgs Accuracy / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN
18. jgs F1-score
19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19 jgs Definition § The harmonic mean of precision and recall. § Mixture of: How much we can trust the model when predict a Positive (Precision), and The ability of the model to find all Positive units (Recall)
20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20 jgs Harmonic mean a = 7 b = 3 / / if same units (big) Arithmetic mean = 7+3 / 2 = 5 / / if diverse units (small) Geometric mean = sqrt (7*3) = 4.58 / / ratios of diverse units (smaller) Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2
21. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21 jgs F1-score Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall TP FP FN TN positive 1 negative 0 positive 1 negative 0
22. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall
23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23 jgs Evaluation
24. jgs Assignment
25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25 jgs Assignment Using our own implementation of a Neural Network (The code you analyzed before): § Can you create a method output that returns all outputs? It is OK can return a Java array. § Can you create a class Evaluation with its method eval? i.e., create a report similar to the one reviewed before
26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26 jgs Questions
27. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27 jgs Reference § Deeplearning4j Suite Overview https://deeplearning4j.konduit.ai § Source Code https://github.com/javiergs/Medium/blob/main/ NeuralNetwork/ExampleXORWithDL4J.java
28. jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez, Ph.D. javiergs@asu.edu Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.