Gradient Descent Easy

Easy/brief version of Gradient Descent from Artificial Intelligence Lecture

Transcript

1. Artificial Intelligence Gradient Descent soulchild
2. Gradient Descent Let computer find the minimum point in a given graph or equation
3. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2
4. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 Minimum point
5. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 Minimum point How does a computer know that this is a minimum point?
6. Gradient Descent Ans : By brute-forcing the derivative until a value equal or near to 0 is found y = x2 dy dx = 2x Then guess x by starting from, eg: -6 to 6
7. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2
8. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 dy dx = -12
9. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 dy dx = -12 dy dx = —8
10. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 dy dx = -12 dy dx = —8 dy dx = -4
11. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 dy dx = -12 dy dx = —8 dy dx = -4 dy dx = 0
12. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 dy dx = -12 dy dx = —8 dy dx = -4 dy dx = 0 Minimum point found, stop
13. Gradient Descent A better way to brute force
14. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 In short, it works like this
15. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 In short, it works like this
16. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 In short, it works like this
17. Gradient Descent y 0 10 20 30 40 x -6 -4 -2 0 2 4 6 y = x2 In short, it works like this
18. Gradient Descent How to set a good learning rate (α) ?
19. Gradient Descent When to stop searching ?
20. Gradient Descent When to stop searching ? Set a maximum number of iteration
21. Gradient Descent When to stop searching ? Set a maximum number of iteration dy dx < n When n can be 0.1, 0.01,.. etc
22. Gradient Descent Weakness of Gradient Descent 0 3 6 9 12 -8 -6 -4 -2 0 2 4
23. Gradient Descent Weakness of Gradient Descent 0 3 6 9 12 -8 -6 -4 -2 0 2 4 let say start from here
24. Gradient Descent Weakness of Gradient Descent 0 3 6 9 12 -8 -6 -4 -2 0 2 4 let say start from here dy dx = 0
25. Gradient Descent Weakness of Gradient Descent 0 3 6 9 12 -8 -6 -4 -2 0 2 4 let say start from here dy dx = 0 then computer stop finding
26. Gradient Descent Weakness of Gradient Descent 0 3 6 9 12 -8 -6 -4 -2 0 2 4 let say start from here dy dx = 0 then computer stop finding What about this?! smaller than previous point wor
27. Gradient Descent Weakness of Gradient Descent Gradient Descent may stuck in a local minima thus can’t find the global minima
28. Gradient Descent Q&A