JGS594 Lecture 07

Software Engineering for Machine Learning
Performance Measurements
(202202)

Transcript

1. jgs
   SER 594
   Software Engineering for
   Machine Learning
   Lecture 07: Performance Measurement
   Dr. Javier Gonzalez-Sanchez
   javiergs@asu.edu
   javiergs.engineering.asu.edu | javiergs.com
   PERALTA 230U
   Office Hours: By appointment
   

   View Slide

2. jgs
   Previously …
   

   View Slide

3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3
   jgs
   ND4J Input
   

   View Slide

4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4
   jgs
   DL4J | Our Model
   

   View Slide

5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5
   jgs
   DL4J | Training
   

   View Slide

6. jgs
   Confusion Matrix
   

   View Slide

7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7
   jgs
   Definition
   A summary of prediction results on a classification problem
   

   View Slide

8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8
   jgs
   Definition
   A summary of prediction results on a classification problem
   TP FP
   FN TN
   positive 0
   negative 1
   positive
   0
   negative
   1
   

   View Slide

9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9
   jgs
   Definition
   TP FP
   FN TN
   positive 1
   negative 0
   positive
   1
   negative
   0
   FP
   Type 1 Error
   It is FALSE
   Computer said TRUE
   FP
   Type 2 Error
   It is TRUE
   Computer said FALSE
   

   View Slide

10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10
    jgs
    Definition
    What about not binary classifiers?
    e.g. emotion recognition
    TP FP
    FN TN
    positive
    😀
    😡
    .
    🙁
    negative
    positive
    😀 😡. 🙁
    negative
    FN TN
    FP
    TN
    TN
    

    View Slide

11. jgs
    Accuracy
    

    View Slide

12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12
    jgs
    Definition
    § overall measure of how much the model is correctly predicting on the entire
    set of data
    § Addition of the elements in the main diagonal divide by the sum of all the
    entries of the confusion matrix at the denominator.
    

    View Slide

13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13
    jgs
    Accuracy
    Accuracy = TP + TN / TP + TN + FP + FN
    TP FP
    FN TN
    positive 1
    negative 0
    positive
    1
    negative
    0
    

    View Slide

14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14
    jgs
    Definition
    positive
    negative
    TP FP
    FN TN
    positive negative
    FN X
    FP
    X
    TN
    Accuracy = TP + TN / TP + TN + FP + FN + X
    

    View Slide

15. jgs
    Precision and Recall
    

    View Slide

16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16
    jgs
    Accuracy
    /
    / How much we can trust the model when predict a Positive
    Precision = TP / TP + FP
    /
    / Measure the ability of the model to find all Positive units
    Recall = TP / TP + FN
    TP FP
    FN TN
    positive 1
    negative 0
    positive
    1
    negative
    0
    

    View Slide

17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17
    jgs
    Definition
    positive
    negative
    TP FP
    FN TN
    positive negative
    FN X
    FP
    X
    TN
    /
    / How much we can trust the model when predict a Positive
    Precision = TP / TP + FP
    /
    / Measure the ability of the model to find all Positive units
    Recall = TP / TP + FN
    

    View Slide

18. jgs
    F1-score
    

    View Slide

19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19
    jgs
    Definition
    § The harmonic mean of precision and recall.
    § Mixture of:
    How much we can trust the model when predict a Positive (Precision), and
    The ability of the model to find all Positive units (Recall)
    

    View Slide

20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20
    jgs
    Harmonic mean
    a = 7
    b = 3
    /
    / if same units (big)
    Arithmetic mean = 7+3 / 2 = 5
    /
    / if diverse units (small)
    Geometric mean = sqrt (7*3) = 4.58
    /
    / ratios of diverse units (smaller)
    Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2
    

    View Slide

21. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21
    jgs
    F1-score
    Precision = TP / TP + FP /
    / predicted
    Recall = TP / TP + FN /
    / real
    F1-score = 2 * Precision * Recall / Precision + Recall
    TP FP
    FN TN
    positive 1
    negative 0
    positive
    1
    negative
    0
    

    View Slide

22. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22
    jgs
    Definition
    positive
    negative
    TP FP
    FN TN
    positive negative
    FN X
    FP
    X
    TN
    Precision = TP / TP + FP /
    / predicted
    Recall = TP / TP + FN /
    / real
    F1-score = 2 * Precision * Recall / Precision + Recall
    

    View Slide

23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23
    jgs
    Evaluation
    

    View Slide

24. jgs
    Assignment
    

    View Slide

25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25
    jgs
    Assignment
    Using our own implementation of a Neural Network (The code you analyzed
    before):
    § Can you create a method output that returns all outputs?
    It is OK can return a Java array.
    § Can you create a class Evaluation with its method eval?
    i.e., create a report similar to the one reviewed before
    

    View Slide

26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26
    jgs
    Questions
    

    View Slide

27. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27
    jgs
    Reference
    § Deeplearning4j Suite Overview
    https://deeplearning4j.konduit.ai
    § Source Code
    https://github.com/javiergs/Medium/blob/main/
    NeuralNetwork/ExampleXORWithDL4J.java
    

    View Slide

28. jgs
    SER 594 Software Engineering for Machine Learning
    Javier Gonzalez-Sanchez, Ph.D.
    javiergs@asu.edu
    Spring 2022
    Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University.
    They cannot be distributed or used for another purpose.
    

    View Slide