JGS594 Lecture 07 - Speaker Deck

Slide 1

Slide 1 text

jgs SER 594 Software Engineering for Machine Learning Lecture 07: Performance Measurement Dr. Javier Gonzalez-Sanchez [email protected] javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment

Slide 2

Slide 2 text

jgs Previously …

Slide 3

Slide 3 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3 jgs ND4J Input

Slide 4

Slide 4 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4 jgs DL4J | Our Model

Slide 5

Slide 5 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5 jgs DL4J | Training

Slide 6

Slide 6 text

jgs Confusion Matrix

Slide 7

Slide 7 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7 jgs Definition A summary of prediction results on a classification problem

Slide 8

Slide 8 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8 jgs Definition A summary of prediction results on a classification problem TP FP FN TN positive 0 negative 1 positive 0 negative 1

Slide 9

Slide 9 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9 jgs Definition TP FP FN TN positive 1 negative 0 positive 1 negative 0 FP Type 1 Error It is FALSE Computer said TRUE FP Type 2 Error It is TRUE Computer said FALSE

Slide 10

Slide 10 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10 jgs Definition What about not binary classifiers? e.g. emotion recognition TP FP FN TN positive 😀 😡 . 🙁 negative positive 😀 😡. 🙁 negative FN TN FP TN TN

Slide 11

Slide 11 text

jgs Accuracy

Slide 12

Slide 12 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12 jgs Definition § overall measure of how much the model is correctly predicting on the entire set of data § Addition of the elements in the main diagonal divide by the sum of all the entries of the confusion matrix at the denominator.

Slide 13

Slide 13 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13 jgs Accuracy Accuracy = TP + TN / TP + TN + FP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0

Slide 14

Slide 14 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Accuracy = TP + TN / TP + TN + FP + FN + X

Slide 15

Slide 15 text

jgs Precision and Recall

Slide 16

Slide 16 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16 jgs Accuracy / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0

Slide 17

Slide 17 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN

Slide 18

Slide 18 text

jgs F1-score

Slide 19

Slide 19 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19 jgs Definition § The harmonic mean of precision and recall. § Mixture of: How much we can trust the model when predict a Positive (Precision), and The ability of the model to find all Positive units (Recall)

Slide 20

Slide 20 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20 jgs Harmonic mean a = 7 b = 3 / / if same units (big) Arithmetic mean = 7+3 / 2 = 5 / / if diverse units (small) Geometric mean = sqrt (7*3) = 4.58 / / ratios of diverse units (smaller) Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2

Slide 21

Slide 21 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21 jgs F1-score Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall TP FP FN TN positive 1 negative 0 positive 1 negative 0

Slide 22

Slide 22 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22 jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall

Slide 23

Slide 23 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23 jgs Evaluation

Slide 24

Slide 24 text

jgs Assignment

Slide 25

Slide 25 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25 jgs Assignment Using our own implementation of a Neural Network (The code you analyzed before): § Can you create a method output that returns all outputs? It is OK can return a Java array. § Can you create a class Evaluation with its method eval? i.e., create a report similar to the one reviewed before

Slide 26

Slide 26 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26 jgs Questions

Slide 27

Slide 27 text

Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27 jgs Reference § Deeplearning4j Suite Overview https://deeplearning4j.konduit.ai § Source Code https://github.com/javiergs/Medium/blob/main/ NeuralNetwork/ExampleXORWithDL4J.java

Slide 28

Slide 28 text

jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez, Ph.D. [email protected] Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.