Algorithm behind AlphaGo and AlphaGo Zero

Briefly review of AlphaGo and AlphaGo Zero algorithm. How it works, and how it learns from self-play. Presented in Young Webmaster Camp Programming Meeting #5.

Transcript

1. Behind AlphaGo AlphaGo Zero Programming Meetup #5 Kosate Limpongsa #ywc12
2. Neung (1) Kosate Limpongsa Studying in Chulalongkorn University #YWC12 #CP41 Developer of JWC7 Developer of YWC13 Writer of • เจาะลึก AlphaGo ทํางานอย่างไร ? (ฉบับเขียนโปรแกรมไม่เป็นก็อ่านได้) • AlphaGo Zero ตัวใหม่: เหนือกว่าทุกตัวก่อนหน้า โดยไม่ใช่ข้อมูลจากมนุษย์แม้แต่นิดเดียว (อย่างไร ?)
3. 1 Introduction 2 Background 3 Algorithm 4 Result
4. I am not proficient at Go
5. I am not proficient at Go Just a nerd guy who read some papers
6. 2 Background 3 Algorithm 4 Result Introduction 1
7. 1 Introduction What is Go?
8. 1 Introduction What is Go? Photo from https://www.scichallenge.eu/blogs/stories-experiences/game-go-a-strategy-game/
9. 1 Introduction What is Go? Photo from AlphaGo Movie
10. 1 Introduction Why Go is hard?
11. 1 Introduction Why Go is hard? 19 x 19 = 361 Photo from https://en.wikipedia.org/wiki/Rules_of_Go
12. 1 Introduction Why Go is hard? Photo from https://youtu.be/SUbqykXVx0A
13. 1 Introduction Why Go is hard? Reference http://tromp.github.io/go/legal.html 208168199381979984699478633 344862770286522453884530548 4256394568209274196127380153 7852564845169851964390725991 601562812854608988831442712 9715319317557736620397247064 840935 More than atoms in universe
14. 1 Introduction Why Go is hard? ~ 9 trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion years If we use all computers in the world
15. 1 Introduction
16. 1 Introduction AlphaGo • Developed by Deepmind • Use deep learning • Combine with tree search
17. 1 Introduction AlphaGo Zero • Zero human knowledge • Harder, Better, Faster, Stronger
18. 3 Algorithm 4 Result Background 2
19. Reinforcement Learning 2 Background
20. 2 Background Reinforcement Learning Photo from https://www.youtube.com/watch?v=OBzvN9FLx4Q
21. 2 Background Photo from https://www.youtube.com/watch?v=OBzvN9FLx4Q Reinforcement Learning +1 win, -1 loss, 0 other
22. 2 Background Goal Photo from https://www.youtube.com/watch?v=OBzvN9FLx4Q Make a smart agent
23. Neural Network 2 Background
24. Deep Learning 2 Background or
25. 2 Background Neural Network Photo from http://cs231n.github.io/neural-networks-1/
26. 2 Background Neural Network Photo from https://stevenmiller888.github.io/mind-how-to-build-a-neural-network/
27. 2 Background Neural Network Photo from https://medium.com/kosate/9d6e5c059e7d Feed-Forward
28. 2 Background Neural Network Photo from https://medium.com/kosate/9d6e5c059e7d Feed-Forward
29. 2 Background Neural Network Photo from https://medium.com/kosate/9d6e5c059e7d Feed-Forward
30. 2 Background Convolutional Neural Network Photo from https://ai.icymi.email/tag/alex-krizhevsky/
31. 4 Result Algorithm 3
32. AlphaGo
33. 3 Algorithm / AlphaGo AlphaGo Architecture
34. 3 Algorithm / AlphaGo 1 Learns from human data Input Output
35. 3 Algorithm / AlphaGo 2 Self-play Input Output Smarter Version
36. 3 Algorithm / AlphaGo 3 Value evaluation Input Output Smarter Version Win +0.79
37. 3 Algorithm / AlphaGo AlphaGo • Rollout (Human data) • SL Policy (Human data) • RL Policy (Self-play) • Value Network (RL Policy)
38. 4 Monte Carlo Tree Search (MCTS)
39. 3 Algorithm / AlphaGo 4 MCTS
40. 3 Algorithm / AlphaGo 4 MCTS Photo from http://slideplayer.com/slide/8088626/ Tree Search
41. 3 Algorithm / AlphaGo 4 MCTS Tree Search
42. 3 Algorithm / AlphaGo 4 MCTS Selection Choose move from RL Policy u(P) + some probability Q
43. 3 Algorithm / AlphaGo 4 MCTS Expand
44. 3 Algorithm / AlphaGo 4 MCTS Evaluation Reward How to estimate reward in middle of the game? Remember?
 You can’t calculate all possibilities Barrier
45. 3 Algorithm / AlphaGo 4 MCTS Evaluation 1) Use Rollout plays until the end of game Play Play
46. 3 Algorithm / AlphaGo 4 MCTS Evaluation 1) Use Rollout plays until the end of game 2) Use Value Network estimates win prop. What percent do I win?
47. 3 Algorithm / AlphaGo 4 MCTS Backup 1) Use Rollout +1 reward 2) Use Value Network +0.79 win Update these values to Q in MCTS
48. 3 Algorithm / AlphaGo 4 MCTS Loop Loop
49. 3 Algorithm / AlphaGo 5 How AlphaGo makes decision Select move with max(Q + u(P)) in MCTS 0.73 0.63
50. That is everything about AlphaGo
51. AlphaGo Zero
52. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture
53. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture
54. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture
55. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture Policy-Value Network Win +0.79 Predict possibilities in all moves Predict possibilities to win
56. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture MCTS
57. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture MCTS MCTS
58. 3 Algorithm / AlphaGo Zero AlphaGo Zero Architecture AlphaGo AlphaGo Zero It is no different
59. 3 Algorithm / AlphaGo Zero 1 Self-play
60. 3 Algorithm / AlphaGo Zero 1 Self-play Use MCTS from beginning
61. 1 Self-play (MCTS) Selection Choose move from network U + some probability Q 3 Algorithm / AlphaGo Zero
62. 1 Self-play (MCTS) Expand 3 Algorithm / AlphaGo Zero
63. 3 Algorithm / AlphaGo Zero 1 Self-play (MCTS) Evaluation Use only Value from network Win 79%
64. 3 Algorithm / AlphaGo Zero 1 Self-play (MCTS) Backup Use Value 79% win Update these values to Q in MCTS
65. 3 Algorithm / AlphaGo Zero 1 Self-play (MCTS) Loop
66. 3 Algorithm / AlphaGo Zero But how it learns?
67. 3 Algorithm / AlphaGo Zero 2 Training Pi
68. 3 Algorithm / AlphaGo Zero 2 Training Pi
69. 3 Algorithm / AlphaGo Zero 2 Training Pi
70. 3 Algorithm / AlphaGo Zero 2 Training Pi π
71. 3 Algorithm / AlphaGo Zero 2 Training Pi
72. 3 Algorithm / AlphaGo Zero 2 Training Z-reward Notice?
73. 3 Algorithm / AlphaGo Zero 2 Training Z-reward +1 win -1 loss
74. 3 Algorithm / AlphaGo Zero 2 Training Z-reward Pi +1.00
75. 3 Algorithm / AlphaGo Zero 2 Training Z-reward Pi +1.00 +0.79
76. 3 Algorithm / AlphaGo Zero 2 Training Z-reward Pi +1.00 +0.79 Predicted Actual
77. Flashback time
78. Photo from https://medium.com/kosate/9d6e5c059e7d Feed-Forward +1.00 +0.79 Predicted Actual 3 Algorithm / AlphaGo Zero 2 Training Errors: 23.512
79. 3 Algorithm / AlphaGo Zero AlphaGo Zero Summary +1.00
80. 3 Algorithm / AlphaGo Zero +1.00 0.12 0.58 0.79 AlphaGo Zero Summary
81. 3 Algorithm / AlphaGo Zero AlphaGo Zero Summary
82. That is everything about AlphaGo Zero
83. Result 4
84. 1 Introduction AlphaGo Timeline AlphaGo Fan AlphaGo Lee Fan Hui (2Dan Player)
 AlphaGo wins 5– 0 Lee Sedol (9Dan Player)
 AlphaGo wins 4 – 1 Born
85. 1 Introduction AlphaGo Timeline AlphaGo Master AlphaGo Zero (Nearly Complete) AlphaGo Zero Complete 60 Professional Player
 AlphaGo wins 60 – 0 Ke Jie (1st World Player)
 AlphaGo wins 3 – 0
86. 4 Result Better
87. 4 Result Faster
88. 4 Result Stronger 5,185 4,858 3,739 3,144
89. 4 Result Stronger AlphaGo Zero AlphaGo Lee 100 – 0 AlphaGo Zero AlphaGo Master 89 – 11 vs vs
90. Nothing can beat AlphaGo Zero
91. Neung (1) Kosate Limpongsa Chulalongkorn University #YWC12 #CP41 Articles • เจาะลึก AlphaGo ทํางานอย่างไร ? https://medium.com/kosate/3a1cf3631289 • AlphaGo Zero ตัวใหม่ https://medium.com/kosate/9d6e5c059e7d Kosate Limpongsa kosatelim (at) gmail.com References • https://www.nature.com/articles/nature16961 • https://www.nature.com/articles/nature24270 medium.com/kosate
92. Extra slide AlphaZero
93. AlphaZero uses same algorithm as AlphaGo Zero Except preprocessing
94. But it is used in general cases Like chess and shogi
95. AlphaZero Chess Shogi AlphaZero
96. AlphaZero Unbeatable
97. AlphaZero Unbeatable
98. AlphaZero Sorry for the long slide Here a potato