Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JGS594 Lecture 07

JGS594 Lecture 07

Software Engineering for Machine Learning
Performance Measurements
(202202)

B546a9b97d993392e4b22b74b99b91fe?s=128

Javier Gonzalez
PRO

February 02, 2022
Tweet

More Decks by Javier Gonzalez

Other Decks in Programming

Transcript

  1. jgs SER 594 Software Engineering for Machine Learning Lecture 07:

    Performance Measurement Dr. Javier Gonzalez-Sanchez javiergs@asu.edu javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment
  2. jgs Previously …

  3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3

    jgs ND4J Input
  4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4

    jgs DL4J | Our Model
  5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5

    jgs DL4J | Training
  6. jgs Confusion Matrix

  7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7

    jgs Definition A summary of prediction results on a classification problem
  8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8

    jgs Definition A summary of prediction results on a classification problem TP FP FN TN positive 0 negative 1 positive 0 negative 1
  9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9

    jgs Definition TP FP FN TN positive 1 negative 0 positive 1 negative 0 FP Type 1 Error It is FALSE Computer said TRUE FP Type 2 Error It is TRUE Computer said FALSE
  10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10

    jgs Definition What about not binary classifiers? e.g. emotion recognition TP FP FN TN positive 😀 😡 . 🙁 negative positive 😀 😡. 🙁 negative FN TN FP TN TN
  11. jgs Accuracy

  12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12

    jgs Definition § overall measure of how much the model is correctly predicting on the entire set of data § Addition of the elements in the main diagonal divide by the sum of all the entries of the confusion matrix at the denominator.
  13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13

    jgs Accuracy Accuracy = TP + TN / TP + TN + FP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
  14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Accuracy = TP + TN / TP + TN + FP + FN + X
  15. jgs Precision and Recall

  16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16

    jgs Accuracy / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
  17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN
  18. jgs F1-score

  19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19

    jgs Definition § The harmonic mean of precision and recall. § Mixture of: How much we can trust the model when predict a Positive (Precision), and The ability of the model to find all Positive units (Recall)
  20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20

    jgs Harmonic mean a = 7 b = 3 / / if same units (big) Arithmetic mean = 7+3 / 2 = 5 / / if diverse units (small) Geometric mean = sqrt (7*3) = 4.58 / / ratios of diverse units (smaller) Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2
  21. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21

    jgs F1-score Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall TP FP FN TN positive 1 negative 0 positive 1 negative 0
  22. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall
  23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23

    jgs Evaluation
  24. jgs Assignment

  25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25

    jgs Assignment Using our own implementation of a Neural Network (The code you analyzed before): § Can you create a method output that returns all outputs? It is OK can return a Java array. § Can you create a class Evaluation with its method eval? i.e., create a report similar to the one reviewed before
  26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26

    jgs Questions
  27. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27

    jgs Reference § Deeplearning4j Suite Overview https://deeplearning4j.konduit.ai § Source Code https://github.com/javiergs/Medium/blob/main/ NeuralNetwork/ExampleXORWithDL4J.java
  28. jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez,

    Ph.D. javiergs@asu.edu Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.