Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JGS594 Lecture 13

JGS594 Lecture 13

Software Engineering for Machine Learning
Midterm Review
(202203)

B546a9b97d993392e4b22b74b99b91fe?s=128

Javier Gonzalez
PRO

March 17, 2022
Tweet

More Decks by Javier Gonzalez

Other Decks in Programming

Transcript

  1. jgs SER 594 Software Engineering for Machine Learning Lecture 13:

    Midterm Review Dr. Javier Gonzalez-Sanchez javiergs@asu.edu javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment
  2. jgs Part 0

  3. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 3

    jgs 2. Machine Learning
  4. jgs Part 1

  5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5

    jgs W, X, Y https://medium.com/analytics-vidhya/neural-networks-in-a-nutshell-with-java-b4a635a2c4af
  6. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 6

    jgs Activation Functions
  7. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 7

    jgs Neurons and Layers https://medium.com/analytics-vidhya/neural-networks-in-a-nutshell-with-java-b4a635a2c4af
  8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8

    jgs Error/Cost Function
  9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9

    jgs Back propagation Did I mention something called learning rate? Review Details (Math) Here: https://medium.com/analytics-vidhya/neural-networks-in-a-nutshell-with-java-b4a635a2c4af
  10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10

    jgs Code https://github.com/javiergs/Medium/blob/main/NeuralNetwork/BasicNeuralNetwork.java
  11. jgs Part 2

  12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12

    jgs ML Frameworks and Libraries
  13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13

    jgs ND4J Input (coordinates, value)
  14. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 14

    jgs DL4J | Our Model * Dense layer – a layer that is deeply connected with its preceding layer which means the neurons of the layer are connected to every neuron of its preceding layer.
  15. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 15

    jgs DL4J | Our Model
  16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16

    jgs DL4J | Training
  17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17

    jgs DL4J | Testing
  18. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 18

    jgs DL4J | Our Model
  19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19

    jgs DL4J | Training
  20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20

    jgs A sneak peek at the Code
  21. jgs Javier Gonzalez-Sanchez | SER332 | Spring 2018 | 21

    … Go for a value < 0.05
  22. jgs Part 3

  23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23

    jgs Evaluation
  24. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 24

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Accuracy = TP + TN / TP + TN + FP + FN + X
  25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN / / How much we can trust the model when predict a Positive Precision = TP / TP + FP / / Measure the ability of the model to find all Positive units Recall = TP / TP + FN
  26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26

    jgs Definition positive negative TP FP FN TN positive negative FN X FP X TN Precision = TP / TP + FP / / predicted Recall = TP / TP + FN / / real F1-score = 2 * Precision * Recall / Precision + Recall
  27. jgs Part 4

  28. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 28

    jgs Load Data (28 x 28) x 70,000
  29. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 29

    jgs Code
  30. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 30

    jgs Model 1
  31. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 31

    jgs Evaluation (Number of classes)
  32. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 32

    jgs Weight Initialization | Xavier § A too-large initialization leads to exploding (partial derivatives) § A too-small initialization leads to vanishing (partial derivatives) Advice: § The mean of the activations should be zero. § The variance of the activations should stay the same across every layer. / / statistical measurement of / / the spread between numbers in a data set
  33. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 33

    jgs Activation Functions | RELU § ReLU –– Rectified linear activation function § Popular activation function for hidden layers
  34. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 34

    jgs Activation Functions | SoftMax § Sigmoid is independent § Most popular activation function for output layers handling multiple classes. § Probabilities.
  35. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 35

    jgs Error Function | Negative Log-Likelihood § the SoftMax function is used in tandem with the negative log-likelihood. § Likelihood of observed data y would be produced by parameter values w L(y, w) Likelihood can be in range 0 to 1. § Log facilitates the derivatives § The Log likelihood values are then in range -Infinite to 0. § Negative make it Infinite to 0 https://hea-www.harvard.edu/AstroStat/aas227_2016/lecture1_Robinson.pdf
  36. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 36

    jgs Updater | Nadam / /Original Choice / /Velocity Momentum coefficient / /Nesterovs / / Nesterov-accelerated Adaptive Moment Estimation ADAM + Nesterov Momentum
  37. jgs Part 5

  38. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 38

    jgs Epoch § An epoch is a term used in machine learning and indicates the number of passes of the entire training dataset the machine learning algorithm has completed.
  39. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 39

    jgs A Second Option RecordReader recordReader = new CSVRecordReader (0, ',’)); // skipNumLines, delimiter recordReader.initialize( new FileSplit( new ClassPathResource("iris.txt").getFile()) ); DataSetIterator iterator = new RecordReaderDataSetIterator( recordReader, // source 150, // rows 4, // inputs 3 // labels );
  40. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 40

    jgs A Second Option DataSet allData = iterator.next(); allData.shuffle(42); //System.currentTimeMillis() // 65% training and 35% testing SplitTestAndTrain testAndTrain = allData.splitTestAndTrain(0.65); DataSet trainingData = testAndTrain.getTrain(); DataSet testData = testAndTrain.getTest();
  41. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 41

    jgs Listeners // add this before calling fit() model.setListeners(new MyListener());
  42. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 42

    jgs MyListener
  43. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 43

    jgs Output …
  44. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 44

    jgs Output 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361 381 401 421 441 461 481 501 521 541 561 581 601 621 641 661 681 701 721 741 761 781 801 821 841 861 881 901 921 941 961 981
  45. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 45

    jgs Save the Model
  46. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 46

    jgs Load the Model Load the updater
  47. jgs Part 6

  48. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 48

    jgs Application
  49. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 49

    jgs Application / / int [] y = model.predict (input) / / y[0]
  50. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 50

    jgs Can we create a GUI? SEPAL LENGTH: SEPAL WIDTH: PETAL LENGTH: PETAL WIDTH: IRIS TYPE: Calculate
  51. jgs Part 7

  52. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 52

    jgs Convolution § Convolutions have been used for a long time in image processing to blur, sharpen images, enhance edges, and emboss. 1 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 4 2 2 3 4 3 4 3 4 Image matrix: Filter: Feature: (RI x CI) (RF x CF) (RI-RF+1 x CI-CF+1)
  53. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 53

    jgs Filters § There are a set of few filters that are used to perform a few tasks. blur sharp borders 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0 -1 0 -1 5 -1 0 -1 0 -1 0 1 -2 0 2 -1 0 1 -1 -2 -1 0 0 0 1 2 1 horizontal vertical
  54. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 54

    jgs Stride § The number of pixels which are shift over the input matrix. § When the stride is equaled to 1, then we move the filters to 1 pixel at a time and similarly, if the stride is equaled to 2, then we move the filters to 2 pixels at a time, etc. 11 21 31 41 51 12 22 32 42 52 13 23 33 43 53 14 24 34 44 54 15 25 35 45 54 61 62 63 64 65 0 1 2 3 4 16 26 36 46 55 10 20 30 40 50 66 60 5 6 1 1 1 1 1 1 1 1 1 99 117 135 279 * =
  55. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 55

    jgs Padding § The pixel in the corner will only get covers one time, but the middle pixels will get covered more than once. § Padding refers to the number of pixels added to an image when it is being processed 1 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  56. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 56

    jgs Pooling § Pooling is downscaling of the image obtained from the previous layers. § It can be compared to shrinking an image to reduce its pixel density § Options: Max-pooling, Average-pooling, Sum-pooling 11 21 31 12 22 32 13 23 33 0 2 3 10 20 30 1 2x2 Max pooling 33 11 31 13
  57. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 57

    jgs Code
  58. jgs Test Yourselves

  59. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 59

    jgs Let’s Work
  60. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 60

    jgs Questions
  61. jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez,

    Ph.D. javiergs@asu.edu Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.