710

# JGS594 Lecture 07

Software Engineering for Machine Learning
Performance Measurements
(202202)

## Javier Gonzalez-SanchezPRO

February 02, 2022

## Transcript

1. jgs
SER 594
Software Engineering for
Machine Learning
Lecture 07: Performance Measurement
Dr. Javier Gonzalez-Sanchez
[email protected]
javiergs.engineering.asu.edu | javiergs.com
PERALTA 230U
Office Hours: By appointment

2. jgs
Previously …

3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3
jgs
ND4J Input

4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4
jgs
DL4J | Our Model

5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5
jgs
DL4J | Training

6. jgs
Confusion Matrix

7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7
jgs
Definition
A summary of prediction results on a classification problem

8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8
jgs
Definition
A summary of prediction results on a classification problem
TP FP
FN TN
positive 0
negative 1
positive
0
negative
1

9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9
jgs
Definition
TP FP
FN TN
positive 1
negative 0
positive
1
negative
0
FP
Type 1 Error
It is FALSE
Computer said TRUE
FP
Type 2 Error
It is TRUE
Computer said FALSE

10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10
jgs
Definition
e.g. emotion recognition
TP FP
FN TN
positive
😀
😡
.
🙁
negative
positive
😀 😡. 🙁
negative
FN TN
FP
TN
TN

11. jgs
Accuracy

12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12
jgs
Definition
§ overall measure of how much the model is correctly predicting on the entire
set of data
§ Addition of the elements in the main diagonal divide by the sum of all the
entries of the confusion matrix at the denominator.

13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13
jgs
Accuracy
Accuracy = TP + TN / TP + TN + FP + FN
TP FP
FN TN
positive 1
negative 0
positive
1
negative
0

14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14
jgs
Definition
positive
negative
TP FP
FN TN
positive negative
FN X
FP
X
TN
Accuracy = TP + TN / TP + TN + FP + FN + X

15. jgs
Precision and Recall

16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16
jgs
Accuracy
/
/ How much we can trust the model when predict a Positive
Precision = TP / TP + FP
/
/ Measure the ability of the model to find all Positive units
Recall = TP / TP + FN
TP FP
FN TN
positive 1
negative 0
positive
1
negative
0

17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17
jgs
Definition
positive
negative
TP FP
FN TN
positive negative
FN X
FP
X
TN
/
/ How much we can trust the model when predict a Positive
Precision = TP / TP + FP
/
/ Measure the ability of the model to find all Positive units
Recall = TP / TP + FN

18. jgs
F1-score

19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19
jgs
Definition
§ The harmonic mean of precision and recall.
§ Mixture of:
How much we can trust the model when predict a Positive (Precision), and
The ability of the model to find all Positive units (Recall)

20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20
jgs
Harmonic mean
a = 7
b = 3
/
/ if same units (big)
Arithmetic mean = 7+3 / 2 = 5
/
/ if diverse units (small)
Geometric mean = sqrt (7*3) = 4.58
/
/ ratios of diverse units (smaller)
Harmonic mean = pow (sqrt (7*3)) / (7+3 / 2) = 21 / 5 = 4.2

21. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21
jgs
F1-score
Precision = TP / TP + FP /
/ predicted
Recall = TP / TP + FN /
/ real
F1-score = 2 * Precision * Recall / Precision + Recall
TP FP
FN TN
positive 1
negative 0
positive
1
negative
0

22. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 22
jgs
Definition
positive
negative
TP FP
FN TN
positive negative
FN X
FP
X
TN
Precision = TP / TP + FP /
/ predicted
Recall = TP / TP + FN /
/ real
F1-score = 2 * Precision * Recall / Precision + Recall

23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23
jgs
Evaluation

24. jgs
Assignment

25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25
jgs
Assignment
Using our own implementation of a Neural Network (The code you analyzed
before):
§ Can you create a method output that returns all outputs?
It is OK can return a Java array.
§ Can you create a class Evaluation with its method eval?
i.e., create a report similar to the one reviewed before

26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26
jgs
Questions

27. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27
jgs
Reference
§ Deeplearning4j Suite Overview
https://deeplearning4j.konduit.ai
§ Source Code
https://github.com/javiergs/Medium/blob/main/
NeuralNetwork/ExampleXORWithDL4J.java

28. jgs
SER 594 Software Engineering for Machine Learning
Javier Gonzalez-Sanchez, Ph.D.
[email protected]
Spring 2022
Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University.
They cannot be distributed or used for another purpose.