JGS594 Lecture 09

Software Engineering for Machine Learning
Image Recognition II
(202202)

Transcript

1. jgs SER 594 Software Engineering for Machine Learning Lecture 09: Image Recognition with DL4J II Dr. Javier Gonzalez-Sanchez javiergs@asu.edu javiergs.engineering.asu.edu | javiergs.com PERALTA 230U Office Hours: By appointment
2. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 2 jgs Career Fair § Master’s and PhD Online Career Fair Tuesday, Feb 15, 2022 9 a.m.–4 p.m. § No lecture that day. § Faculty picnic with students Thursday, Feb 24, 2022 (SER Faculty, SCAI Director, Dean of Students). (there will be food) 11:30 am I will start the lecture 12:15 pm that day.
3. jgs Previously …
4. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 4 jgs MNIST dataset § Each number is stored as an anti-aliased image in black and white and is normalized to fit into a 28x28 pixel bounding box
5. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 5 jgs Model 1
6. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 6 jgs Results 🙁
7. jgs Model Case 2
8. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 8 jgs Model 2
9. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 9 jgs Weight Initialization | Xavier § A too-large initialization leads to exploding (partial derivatives) § A too-small initialization leads to vanishing (partial derivatives) Advice: § The mean of the activations should be zero. § The variance of the activations should stay the same across every layer. / / statistical measurement of / / the spread between numbers in a data set
10. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 10 jgs Activation Functions | RELU § ReLU –– Rectified linear activation function § Popular activation function for hidden layers
11. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 11 jgs Activation Functions | SoftMax § Sigmoid is independent § Most popular activation function for output layers handling multiple classes. § Probabilities.
12. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 12 jgs Error Function | Negative Log-Likelihood § the SoftMax function is used in tandem with the negative log-likelihood. § Likelihood of observed data y would be produced by parameter values w L(y, w) Likelihood can be in range 0 to 1. § Log facilitates the derivatives § The Log likelihood values are then in range -Infinite to 0. § Negative make it Infinite to 0 https://hea-www.harvard.edu/AstroStat/aas227_2016/lecture1_Robinson.pdf
13. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 13 jgs Results
14. jgs Model Case 3
15. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 15 jgs Model 3 (learningRate, momentum)
16. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 16 jgs Updater § Training mechanisms.
17. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 17 jgs Updater § Training mechanisms. § There are methods that can result in much faster network training compared to 'vanilla' gradient descent. You can set the updater using the .updater(Updater) configuration option. § E.g., momentum, RMSProp, adagrad, ADAM, NADAM, and others
18. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 18 jgs Updater § A limitation of gradient descent is that the progress of the search can slow down if the gradient becomes flat or large curvature. § Momentum can be added to gradient descent that incorporates some inertia to updates. / / quantity of motion of a moving body / / (product of its mass and velocity)
19. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 19 jgs Updater | Nesterov Momentum / /Original Choice / /Velocity Momentum coefficient / /Nesterovs
20. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 20 jgs Results
21. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 21 jgs Results Is this Good enough?
22. jgs Model Case 4
23. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 23 jgs Model 4
24. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 24 jgs Updater § A limitation of gradient descent is that the progress of the search can slow down if the gradient becomes flat or large curvature. § Momentum can be added to gradient descent that incorporates some inertia to updates. § Adaptive Movement Estimation (ADAM): Calculate learning rate for each input objective function and further smooths the search process by using an exponentially decreasing moving average of the gradient § Nesterov Momentum + ADAM Nesterov-accelerated Adaptive Moment Estimation
25. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 25 jgs Updater | Nadam / /Original Choice / /Velocity Momentum coefficient / /Nesterovs / / Nesterov-accelerated Adaptive Moment Estimation ADAM + Nesterov Momentum
26. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 26 jgs Results
27. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 27 jgs Results 👍
28. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 28 jgs Model 4 / / Adam() ?
29. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 32 jgs Questions
30. Javier Gonzalez-Sanchez | SER 594 | Spring 2022 | 33 jgs Reference § Deeplearning4j Suite Overview https://deeplearning4j.konduit.ai § Papers on Canvas § Source Code
31. jgs SER 594 Software Engineering for Machine Learning Javier Gonzalez-Sanchez, Ph.D. javiergs@asu.edu Spring 2022 Copyright. These slides can only be used as study material for the class CSE205 at Arizona State University. They cannot be distributed or used for another purpose.