$30 off During Our Annual Pro Sale. View Details »

JGS594 Lecture 07

JGS594 Lecture 07

Software Engineering for Machine Learning
Performance Measurements
(202202)

Javier Gonzalez-Sanchez

February 02, 2022
Tweet

More Decks by Javier Gonzalez-Sanchez

Other Decks in Programming

Transcript

  1. jgs SER 594 Software Engineering for Machine Learning Lecture 07:

    Performance Measurement Dr. Javier Gonzalez-Sanchez [email protected] javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment
  2. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7

    jgs Definition A summary of prediction results on a classification problem
  3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8

    jgs Definition A summary of prediction results on a classification problem TP FP FN TN positive 0 negative 1 positive 0 negative 1
  4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9

    jgs Definition TP FP FN TN positive 1 negative 0 positive 1 negative 0 FP Type 1 Error It is FALSE Computer said TRUE FP Type 2 Error It is TRUE Computer said FALSE
  5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10

    jgs Definition What about not binary classifiers? e.g. emotion recognition TP FP FN TN positive 😀 😡 . 🙁 negative positive 😀 😡. 🙁 negative FN TN FP TN TN
  6. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12

    jgs Definition § overall measure of how much the model is correctly predicting on the entire set of data § Addition of the elements in the main diagonal divide by the sum of all the entries of the confusion matrix at the denominator.
  7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13

    jgs Accuracy Accuracy = TP + TN / TP + TN + FP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
  8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Accuracy = TP + TN / TP + TN + FP + FN + X
  9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16

    jgs Accuracy / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN TP FP FN TN positive 1 negative 0 positive 1 negative 0
  10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN
  11. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19

    jgs Definition § The harmonic mean of precision and recall. § Mixture of: How much we can trust the model when predict a Positive (Precision), and The ability of the model to find all Positive units (Recall)
  12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20

    jgs Harmonic mean a = 7 b = 3 / / if same units (big) Arithmetic mean = 7+3 / 2 = 5 / / if diverse units (small) Geometric mean = sqrt (7*3) = 4.58 / / ratios of diverse units (smaller) Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2
  13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21

    jgs F1-score Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall TP FP FN TN positive 1 negative 0 positive 1 negative 0
  14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall
  15. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25

    jgs Assignment Using our own implementation of a Neural Network (The code you analyzed before): § Can you create a method output that returns all outputs? It is OK can return a Java array. § Can you create a class Evaluation with its method eval? i.e., create a report similar to the one reviewed before
  16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27

    jgs Reference § Deeplearning4j Suite Overview https://deeplearning4j.konduit.ai § Source Code https://github.com/javiergs/Medium/blob/main/ NeuralNetwork/ExampleXORWithDL4J.java
  17. jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez,

    Ph.D. [email protected] Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.