Beating Go Thanks to the Power of Randomness

Go is a board game that is more than 2,500 years old (yes, this is not about the programming language!) and it is fascinating from multiple viewpoints. For instance, go bots still can’t beat professional players, unlike in chess.

This talk will show you what is so special about Go that computers still can’t beat humans. We will take a look at the most popular underlying algorithm and show you how the Monte Carlo method, basically random simulation, plays a vital role in conquering Go's complexity and creating the strong Go bots of today.

Transcript

1. View Slide

2. Ing cup 1985 – 2000
   (up to 1,400,000$)
   (1985-2000)
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. 31st October
   2015
   

   View Slide

7. 3rd November 2015
   

   View Slide

8. 13th November 2015
   

   View Slide

9. Beating Go thanks to the power
   of randomness
   Tobias Pfeiffer
   @PragTob
   pragtob.info
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. View Slide

44. View Slide

45. View Slide

46. View Slide

47. Go vs. Chess
    

    View Slide

48. Complex vs. Complicated
    

    View Slide

49. „While the Baroque rules of chess could only
    have been created by humans, the rules of
    go are so elegant, organic, and rigorously
    logical that if intelligent life forms exist
    elsewhere in the universe, they almost
    certainly play go.“
    Edward Lasker (chess grandmaster)
    

    View Slide

50. Range Stage
    30k-20k Beginner
    19k-10k Casual Player
    9k-1k Intermediate Amateur
    1d-7d Advanced Amateur
    1p-9p Professional
    

    View Slide

51. View Slide

52. View Slide

53. View Slide

54. View Slide

55. 5d win 1998
    

    View Slide

56. Why is Go so hard?
    

    View Slide

57. Larger board
    19x19 vs. 8x8
    

    View Slide

58. Almost every move is legal
    

    View Slide

59. Average branching factor:
    250 vs 35
    

    View Slide

60. State Space Complexity:
    10171 vs 1047
    

    View Slide

61. View Slide

62. Global impact of moves
    

    View Slide

63. Artifical Intelligence
    

    View Slide

64. 6
    8
    9
    5
    7
    9
    6
    6
    3
    5
    4
    7
    6
    5
    6
    8
    5
    7
    6
    6
    3
    4
    5
    8
    5
    7
    6
    3
    5
    5
    6
    3
    6
    MAX
    MIN
    MAX
    MIN
    MAX
    

    View Slide

65. 6
    8
    9
    5
    7
    9
    6
    6
    3
    5
    4
    7
    6
    5
    6
    8
    5
    7
    6
    6
    3
    4
    5
    8
    5
    7
    6
    3
    5
    5
    6
    3
    6
    MAX
    MIN
    MAX
    MIN
    MAX
    

    View Slide

66. 6
    8
    9
    5
    7
    9
    6
    6
    3
    5
    4
    7
    6
    5
    6
    8
    5
    7
    6
    6
    3
    4
    5
    8
    5
    7
    6
    3
    5
    5
    6
    3
    6
    MAX
    MIN
    MAX
    MIN
    MAX
    

    View Slide

67. 6
    8
    9
    5
    7
    9
    6
    6
    3
    5
    4
    7
    6
    5
    6
    8
    5
    7
    6
    6
    3
    4
    5
    8
    5
    7
    6
    3
    5
    5
    6
    3
    6
    MAX
    MIN
    MAX
    MIN
    MAX
    

    View Slide

68. 6
    8
    9
    5
    7
    9
    6
    6
    3
    5
    4
    7
    6
    5
    6
    8
    5
    7
    6
    6
    3
    4
    5
    8
    5
    7
    6
    3
    5
    5
    6
    3
    6
    MAX
    MIN
    MAX
    MIN
    MAX
    

    View Slide

69. Evaluation function
    

    View Slide

70. View Slide

71. Monte Carlo Method
    

    View Slide

72. What is Pi?
    

    View Slide

73. How do you determine Pi?
    

    View Slide

74. View Slide

75. View Slide

76. Browne, Cb, and Edward Powley. 2012. A survey of monte carlo tree search methods. Intelligence and AI 4, no. 1: 1-49
    

    View Slide

77. 2/4
    1/1 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    

    View Slide

78. 2/4
    1/1 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    Selection
    

    View Slide

79. 2/4
    1/1 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    0/0
    B5
    Expansion
    

    View Slide

80. 2/4
    1/1 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    0/0
    B5
    Simulation
    

    View Slide

81. View Slide

82. 3/5
    2/2 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    1/1
    B5
    Backpropagation
    

    View Slide

83. 3/5
    2/2 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    1/1
    B5
    Perspective
    

    View Slide

84. 2/5
    1/2 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    1/1
    B5
    Perspective
    

    View Slide

85. 2/4
    1/1 0/1 1/1 0/1
    A1
    D5
    F13
    C7
    Selection
    

    View Slide

86. Multi Armed Bandit
    

    View Slide

87. Exploitation vs Exploration
    

    View Slide

88. wins
    visits
    +explorationFactor
    √ln(totalVisits)
    visits
    

    View Slide

89. 15042
    86/193
    0/1 1/2 0/2
    36/1116
    2/2
    58/151
    1/2 0/2
    3/3
    

    View Slide

90. 15042
    86/193
    0/1 1/2 0/2
    36/1116
    2/2
    58/151
    1/2 0/2
    3/3
    

    View Slide

91. 15042
    86/193
    0/1 1/2 0/2
    36/1116
    2/2
    58/151
    1/2 0/2
    3/3
    

    View Slide

92. Not Human like?
    

    View Slide

93. Characteristics
    

    View Slide

94. Aheuristic
    

    View Slide

95. Generate a valid random move
    

    View Slide

96. Who has won?
    

    View Slide

97. View Slide

98. General Game Playing
    

    View Slide

99. Anytime
    

    View Slide

100. Lazy
     

     View Slide

101. View Slide

102. View Slide

103. View Slide

104. View Slide

105. -2 -1 0 1 2
     8 86% 88% 90% 94% 98%
     16 86% 92% 94% 94% 96%
     32 94% 96% 98% 96% 95%
     64 98% 99,6% 99,9% 99,4% 96%
     100 99,8% 99,9% 100% 99,99% 98%
     

     View Slide

106. -2 -1 0 1 2
     8 86% 88% 90% 94% 98%
     16 86% 92% 94% 94% 96%
     32 94% 96% 98% 96% 95%
     64 98% 99,6% 99,9% 99,4% 96%
     100 99,8% 99,9% 100% 99,99% 98%
     

     View Slide

107. -2 -1 0 1 2
     8 86% 88% 90% 94% 98%
     16 86% 92% 94% 94% 96%
     32 94% 96% 98% 96% 95%
     64 98% 99,6% 99,9% 99,4% 96%
     100 99,8% 99,9% 100% 99,99% 98%
     

     View Slide

108. -2 -1 0 1 2
     8 86% 88% 90% 94% 98%
     16 86% 92% 94% 94% 96%
     32 94% 96% 98% 96% 95%
     64 98% 99,6% 99,9% 99,4% 96%
     100 99,8% 99,9% 100% 99,99% 98%
     

     View Slide

109. View Slide

110. View Slide

111. Enhancements
     

     View Slide

112. All Moves As First
     

     View Slide

113. RAVE
     

     View Slide

114. Expert Knowledge
     

     View Slide

115. View Slide

116. View Slide

117. View Slide

118. Selection
     

     View Slide

119. Oh yeah
     

     View Slide

120. PragTob/Rubykon
     

     View Slide

121. PragTob/web-go
     

     View Slide

122. pasky/michi
     

     View Slide

123. ujh/iomrascalai
     

     View Slide

124. What have I learned?
     

     View Slide

125. Making X faster
     vs
     Doing less of X
     

     View Slide

126. Modularizing small components
     

     View Slide

127. Benchmark everything
     

     View Slide

128. Solving problems the human way
     vs
     Solving problems the computer
     way
     

     View Slide

129. Joy of Creation
     

     View Slide

130. Beating Go thanks to the power
     of randomness
     Tobias Pfeiffer
     @PragTob
     pragtob.info
     

     View Slide

131. Photo Credit
     ●
     https://en.wikipedia.org/wiki/Emperor_Yao#/media/File:Ma_Lin_-_Emperor_Yao.jpg
     ●
     https://en.wikipedia.org/wiki/Zuo_Zhuan#/media/File:Li_Yuanyang_Zuo_zhuan_first_page.png
     ●
     https://en.wikipedia.org/wiki/Four_arts#/media/File:The_Eighteen_Scholars_by_an_anonymous_Ming_artist_2.jpg
     ●
     https://en.wikipedia.org/wiki/Kibi_no_Makibi#/media/File:Kibino_Makibi.jpg
     ●
     https://en.wikipedia.org/wiki/Honinbo_Sansa#/media/File:Honinbo_Sansa.jpg
     ●
     http://www.bbc.co.uk/arts/yourpaintings/paintings/thomas-hyde-16361703-228754
     ●
     https://en.wikipedia.org/wiki/Oskar_Korschelt#/media/File:Oscar_Korschelt.jpg
     ●
     https://en.wikipedia.org/wiki/Atari#/media/File:Atari_Official_2012_Logo.svg
     ●
     http://www.computer-go.info/events/ing/2000/images/bigcup.jpg
     ●
     http://www.wired.com/2014/05/the-world-of-computer-go/
     ●
     https://en.wikipedia.org/wiki/File:Radha-Krishna_chess.jpg
     ●
     https://en.wikipedia.org/wiki/File:EnxadrismoGravuras.003.jpg
     ●
     http://archive.is/QG6a
     ●
     http://giphy.com/gifs/monkey-bubbles-chimp-2Faz9OUQfOcltIJTG
     ●
     https://en.wikipedia.org/wiki/The_Turk#/media/File:Turk-engraving5.jpg
     ●
     https://en.wikipedia.org/wiki/File:Kasparov-29.jpg
     ●
     CC BY 2.0
     – https://en.wikipedia.org/wiki/File:Deep_Blue.jpg
     –
     ●
     CC BY-SA 3.0
     – https://en.wikipedia.org/wiki/Konrad_Zuse#/media/File:Konrad_Zuse_%281992%29.jpg
     – https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning#/media/File:AB_pruning.svg
     – https://en.wikipedia.org/wiki/Go_%28programming_language%29#/media/File:Golang.png
     

     View Slide

132. Photo Credit
     ●
     https://en.wikipedia.org/wiki/Emperor_Yao#/media/File:Ma_Lin_-_Emperor_Yao.jpg
     ●
     https://en.wikipedia.org/wiki/Atari#/media/File:Atari_Official_2012_Logo.svg
     ●
     http://www.computer-go.info/events/ing/2000/images/bigcup.jpg
     ●
     http://www.wired.com/2014/05/the-world-of-computer-go/
     ●
     http://archive.is/QG6a
     ●
     https://en.wikipedia.org/wiki/The_Turk#/media/File:Turk-engraving5.jpg
     ●
     https://en.wikipedia.org/wiki/File:Kasparov-29.jpg
     ●
     CC BY 2.0
     – https://en.wikipedia.org/wiki/File:Deep_Blue.jpg
     –
     ●
     CC BY-SA 3.0
     – https://en.wikipedia.org/wiki/Konrad_Zuse#/media/File:Konrad_Zuse_%281992%29.jpg
     – https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning#/media/File:AB_pruning.svg
     – https://en.wikipedia.org/wiki/Go_%28programming_language%29#/media/File:Golang.png
     ●
     CC BY-SA 2.0
     – https://www.flickr.com/photos/mike_miley/7762037662/in/photolist-cPUtny-2Jyv1K-6rkH7Y-pDdKnE-6W7Amw-
     pDYyb5-pVK2bG-5cavw1-jbNWJC-6rgxSr-cKt4c-5w7uns-pDbh7H-4swKk-9TAvoC-nMY3Do-51yJaD-eUrQ5d-mHs87x-nEkW
     87-hmMnyg-o3Enjw-rf7AY5-8hAiN6-eY3iqs-9fmGiN-sSzYQ-oq6rm2-oA9xdT-froGke-gJ8cJ8-igt2FS-mRz9Gc-gMexAK-
     eSKEzV-nPy1Zu-527E2U-pEgLhp-ivCWw8-bpCbU-qb22fr-odobP9-htytWv-k4NMKa-dCNpPk-foM8Lk-o73rga-dNvVbs-na2
     qUc-eXLwhK
     

     View Slide

133. Photo Credit
     ●
     https://en.wikipedia.org/wiki/Emperor_Yao#/media/File:Ma_Lin_-_Emperor_Yao.jpg
     ●
     https://en.wikipedia.org/wiki/Zuo_Zhuan#/media/File:Li_Yuanyang_Zuo_zhuan_first_page.png
     ●
     https://en.wikipedia.org/wiki/Four_arts#/media/File:The_Eighteen_Scholars_by_an_anonymous_Ming_artist_2.jpg
     ●
     https://en.wikipedia.org/wiki/Kibi_no_Makibi#/media/File:Kibino_Makibi.jpg
     ●
     https://en.wikipedia.org/wiki/Honinbo_Sansa#/media/File:Honinbo_Sansa.jpg
     ●
     http://www.bbc.co.uk/arts/yourpaintings/paintings/thomas-hyde-16361703-228754
     ●
     https://en.wikipedia.org/wiki/Oskar_Korschelt#/media/File:Oscar_Korschelt.jpg
     ●
     https://en.wikipedia.org/wiki/Atari#/media/File:Atari_Official_2012_Logo.svg
     ●
     http://www.computer-go.info/events/ing/2000/images/bigcup.jpg
     ●
     http://www.wired.com/2014/05/the-world-of-computer-go/
     ●
     https://en.wikipedia.org/wiki/File:Radha-Krishna_chess.jpg
     ●
     https://en.wikipedia.org/wiki/File:EnxadrismoGravuras.003.jpg
     ●
     http://archive.is/QG6a
     ●
     http://giphy.com/gifs/monkey-bubbles-chimp-2Faz9OUQfOcltIJTG
     ●
     https://en.wikipedia.org/wiki/The_Turk#/media/File:Turk-engraving5.jpg
     ●
     https://en.wikipedia.org/wiki/File:Kasparov-29.jpg
     ●
     CC BY 2.0
     – https://en.wikipedia.org/wiki/File:Deep_Blue.jpg
     –
     ●
     CC BY-SA 3.0
     – https://en.wikipedia.org/wiki/Konrad_Zuse#/media/File:Konrad_Zuse_%281992%29.jpg
     – https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning#/media/File:AB_pruning.svg
     – https://en.wikipedia.org/wiki/Go_%28programming_language%29#/media/File:Golang.png
     

     View Slide

134. Photo Credit
     ●
     https://en.wikipedia.org/wiki/Emperor_Yao#/media/File:Ma_Lin_-_Emperor_Yao.jpg
     ●
     https://en.wikipedia.org/wiki/Zuo_Zhuan#/media/File:Li_Yuanyang_Zuo_zhuan_first_page.png
     ●
     https://en.wikipedia.org/wiki/Four_arts#/media/File:The_Eighteen_Scholars_by_an_anonymous_Ming_artist_2.jpg
     ●
     https://en.wikipedia.org/wiki/Kibi_no_Makibi#/media/File:Kibino_Makibi.jpg
     ●
     https://en.wikipedia.org/wiki/Honinbo_Sansa#/media/File:Honinbo_Sansa.jpg
     ●
     http://www.bbc.co.uk/arts/yourpaintings/paintings/thomas-hyde-16361703-228754
     ●
     https://en.wikipedia.org/wiki/Oskar_Korschelt#/media/File:Oscar_Korschelt.jpg
     ●
     https://en.wikipedia.org/wiki/Atari#/media/File:Atari_Official_2012_Logo.svg
     ●
     http://www.computer-go.info/events/ing/2000/images/bigcup.jpg
     ●
     http://www.wired.com/2014/05/the-world-of-computer-go/
     ●
     https://en.wikipedia.org/wiki/File:Radha-Krishna_chess.jpg
     ●
     https://en.wikipedia.org/wiki/File:EnxadrismoGravuras.003.jpg
     ●
     http://archive.is/QG6a
     ●
     http://www.usgo.org/news/2011/07/hikaru-anime-on-hulu-and-netflix/
     ●
     https://en.wikipedia.org/wiki/The_Turk#/media/File:Turk-engraving5.jpg
     ●
     https://en.wikipedia.org/wiki/File:Kasparov-29.jpg
     ●
     CC BY 2.0
     – https://en.wikipedia.org/wiki/File:Deep_Blue.jpg
     – https://www.flickr.com/photos/aigle_dore/14110664878/in/photolist-nuUR4u-e1q7YM-5Mqchf-rMcnKt-6rF4Td-aiMGos-nVks3G-7eKpi2-4iRRUa-ecdN2m-t33akk-8
     CQwoX-firCja-8TAfbC-5Do92i-4U6yXA-dQUgdC-2hkKMK-cMgbim-iniaf-7xxKyM-eqqmuT-a7WHU1-5ZbrEE-g97Nph-35ASJL-gJtoKD-9TDrt-fz3bSd-4qAGhJ-ge5BS1-bxiUwu-
     6wYoR8-5UbciZ-84AZHc-59efoV-8gZ1yt-9Le6DZ-dy74yw-pWJVFe-2xCwen-omzMF4-nGgBMj-rq82wx-4GrWvo-yPvGeK-6NuTMt-9eGoR4-9ZifBq-db2fLW
     ●
     CC BY-SA 3.0
     – https://en.wikipedia.org/wiki/Konrad_Zuse#/media/File:Konrad_Zuse_%281992%29.jpg
     – https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning#/media/File:AB_pruning.svg
     – https://en.wikipedia.org/wiki/Go_%28programming_language%29#/media/File:Golang.png
     

     View Slide

135. Photo Credit
     ●
     CC BY-NC-ND 2.0
     – https://www.flickr.com/photos/aerialcamera/15753422176/in/photolist-q15pzb-5o8noQ-9kzjxL-2j8Cjg
     -e5yjMU-7xTuVB-n75WB6-dCg74N-71JXoJ-8NwBqb-j3typA-79oGNv-aEvcKT-r9j7s2-6pSzwn-aURgGr-j69RDV-4Tw
     VKe-6dGZqk-6FjmMs-8kWfPL-jJMnA2-aA4SnC-7rCdVT-92CTsh-9vbC6n-92CTME-7bhyei-92zK8B-qzprcx-7yhuE8-
     gmpP3A-gmq6uT-9m5Gyx-9m5F9B-2G7F7A-o9fpEY-q2uByi-92DKJr-7T8jPc-92qCsX-acbDAF-7QutRi-cZLZLU-azER
     ev-a3Lcnj-gmoQYw-93s1fE-noZEfj-6jrkdA
     – https://www.flickr.com/photos/stargardener/7037360553/in/photolist-bHSj7D-ipFVRk-dbQMMF-9pkdk1-
     akZXq-ocP2RE-6Rog8i-4NZtZG-aM7Tr4-83N9cr-avkRuq-wUzMF-xvV8G-6EoNDC-bqG35H-8tZTNm-bit9C-xiQv5-7p
     W2xg-5z58z8-wtDY8-bA2bvb-duBtzt-9hnK36-pTKW9S-6GEZSe-9KaFui-9ZAgm1-djUsDh-oPTkQ4-7wwnMo-4wSaYW-
     JyEqK-4tZTqD-9cdenf-na9Bzc-pwiEWL-9ipZiR-prY2Z1-pyTq4i-6Qq3bR-bjFP7x-bXCB3s-77WG8U-pbnQ5v-avy5c
     r-3YdbZj-4wuUvu-qs91kS-dg6cjy
     – https://www.flickr.com/photos/andreastsonis/11518720353/in/photolist-ixSsfM-iFE9j2-8R7Now-cXNz
     15-9iL3iz-iw1VTu-9GvRkV-egTDcw-9iK9Nx-9CpqPB-oDZVWG-egMSC6-egMT2r-egMTez-r54iiS-egMSjr-egMR4Z-
     egTAMQ-egTCfj-egTFt3-egTCrs-egMTWc-egM1Nx-eeDtNx-9iKvwP-9iLhCp-9iPgrL-egTE8J-egTERq-egTDU5-egM
     Rpv-egTC5u-egTBXf-egTBEC-egTFg3-egTF5h-egMUca-egMQRF-egaYGL-9BRUZh-efDsaf-9B7mJK-efDtPh-9BNXsk
     -9Ctoyf-egN3dk-9BaGPu-9iLfpZ-9GvRFM-9GyJR7
     –
     ●
     https://en.wikipedia.org/wiki/Alphabet_Inc.#/media/File:Alphabet_Inc_Logo_2015.svg
     ●
     CC BY-NC 2.0
     – https://www.flickr.com/photos/sutekidane/2199385255/in/photolist-4mmqBr-7NqSe4-abChUK-9NA6gV-
     755hWp-q4tjgW-8FCyzU-4zcMni-abESMd-4hMNYF-6c24D1-tBRRPr-qDt6Bt-4hRU9J-sEKsZd-a2x9tv-ampYHm-7m
     2UsB-abDDQy-tkiVfc-pJr2of-4t4uQA-enwU-3d2tQV-d8cmaf-4Ymd6d-enwJ-huAHMi-gE97EH-zFu7N-otJ1TR-4V
     AkNv-utVH9-tkak53-56vGR-6LbDW8-c4R6PN-3Hn7Le-abEBUA-7JMTVd-7XodLF-eb2Sme-77gM4Z-6WxHh8-oLcvhR
     -9NA4Yk-4YgWen-oLctgr-otH1Dd-oLcrXe
     –
     ●
     CC BY 3.0
     

     View Slide