Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Journal Club: AlphaGo/AlphaGo Zero

Keita Watanabe
October 30, 2021
120

Journal Club: AlphaGo/AlphaGo Zero

Keita Watanabe

October 30, 2021
Tweet

More Decks by Keita Watanabe

Transcript

  1. .PUJWBUJPO w 5SFFCBTFEEFDJTJPONBLJOHGSBNFXPSLJTDPNNPO BDSPTTSPCPUJDT "7 BOEFUDʜ w .POUF$BSMP5SFF4FBSDI .$54 JTPOFPGUIFNPTU

    TVDDFTTGVMNFUIPEBNPOHPUIFSUSFFTFBSDI BMHPSJUINT w 3FDFOU.$54CBTFEEFDFTJPONBLJOHGSBNFXPSL GPS"7 $BJ TJHOJpDBOUMZJOqVFODFECZ "MQIB(P
  2. 0WFSWJFXPGUIJT QSFTFOUBUJPO w *OUSPEVDUJPOUP(P w "MQIB(P w 4-1PMJDZ/FUXPSL w 3-1PMJDZ/FUXPSL

    w 7BMVF/FUXPSL w .$54 .POUF$BSMP5SFF4FBSDI  w "MQIB(P;FSP w *NQSPWFNFOUTGSPN"MQIB(P
  3. 3VMFPG(P* 3FUSJFWFEGSPN8JLJQFEJB
 IUUQTFOXJLJQFEJBPSHXJLJ(P@ HBNF (PJTBOBEWFSTBSJBMHBNFXJUIUIFPCKFDUJWFPG TVSSPVOEJOHBMBSHFSUPUBMBSFBPGUIFCPBSEXJUIPOFT TUPOFTUIBOUIFPQQPOFOU"TUIFHBNFQSPHSFTTFT UIF QMBZFSTQPTJUJPOTUPOFTPOUIFCPBSEUPNBQPVUGPSNBUJPOT BOEQPUFOUJBMUFSSJUPSJFT$POUFTUTCFUXFFOPQQPTJOH

    GPSNBUJPOTBSFPGUFOFYUSFNFMZDPNQMFYBOENBZSFTVMUJOUIF FYQBOTJPO SFEVDUJPO PSXIPMFTBMFDBQUVSFBOEMPTTPG GPSNBUJPOTUPOFT 5IFGPVSMJCFSUJFT BEKBDFOUFNQUZQPJOUT PGBTJOHMFCMBDL TUPOF " BT8IJUFSFEVDFTUIPTFMJCFSUJFTCZPOF # $ BOE%  8IFO#MBDLIBTPOMZPOFMJCFSUZMFGU % UIBUTUPOFJTJOBUBSJz 8IJUFNBZDBQUVSFUIBUTUPOF SFNPWFGSPNCPBSE XJUIBQMBZ POJUTMBTUMJCFSUZ BU%  "CBTJDQSJODJQMFPG(PJTUIBUBHSPVQPGTUPOFTNVTUIBWFBU MFBTUPOFMJCFSUZUPSFNBJOPOUIFCPBSE"MJCFSUZJTBOPQFO QPJOU JOUFSTFDUJPO CPSEFSJOHUIFHSPVQ"OFODMPTFEMJCFSUZ PSMJCFSUJFT JTDBMMFEBOlFZFz ؟ BOEBHSPVQPGTUPOFTXJUI UXPPSNPSFFZFTJTTBJEUPCFVODPOEJUJPOBMMZBMJWF4VDI HSPVQTDBOOPUCFDBQUVSFE FWFOJGTVSSPVOEFE
  4. 3VMFPG(P*7
 7JDUPSZKVEHNFOU *GZPVXBOUUPLOPXNPSF KVTUBTL*WPPS&SJL 'JHPG 0UTVLJ 4DPSFPGTUPOFT PG FZFT ,PNJ#MBDL

    .PWFTpSTU  UBLFTBIBOEJDBQ5ZQJDBMMZ QPJOUT #MBDLUFSSJUPSZ 8IJUF UFSSJUPSZ
  #MBDLXJOT
  5. 8IZ(PJTTPEJ⒏DVMU "QQSPY4J[FPGUIF4FBSDI4QBDF 0UIFMMP   $IFTT   4IPHJ 

     (P   5BCMFPG 0UVLJ 4J[FPGUIFTFBSDITQBDFJTFOPSNPVT
  6. "CTUSBDU 5IFHBNFPG(PIBTMPOHCFFOWJFXFEBTUIFNPTUDIBMMFOHJOHPGDMBTTJD HBNFTGPSBSUJpDJBMJOUFMMJHFODFPXJOHUPJUTFOPSNPVTTFBSDITQBDFBOEUIF EJ⒏DVMUZPGFWBMVBUJOHCPBSEQPTJUJPOTBOENPWFT)FSFXFJOUSPEVDFBOFX BQQSPBDIUPDPNQVUFS(PUIBUVTFTWBMVFOFUXPSLTUPFWBMVBUFCPBSE QPTJUJPOTBOEQPMJDZOFUXPSLTUPTFMFDUNPWFT5IFTFEFFQOFVSBMOFUXPSLTBSF USBJOFECZBOPWFMDPNCJOBUJPOPGTVQFSWJTFEMFBSOJOHGSPNIVNBOFYQFSUHBNFT  BOESFJOGPSDFNFOUMFBSOJOHGSPNHBNFTPGTFMGQMBZ8JUIPVUBOZMPPLBIFBE TFBSDI

    UIFOFVSBMOFUXPSLTQMBZ(PBUUIFMFWFMPGTUBUFPGUIFBSU.POUF $BSMPUSFFTFBSDIQSPHSBNTUIBUTJNVMBUFUIPVTBOETPGSBOEPNHBNFTPG TFMGQMBZ8FBMTPJOUSPEVDFBOFXTFBSDIBMHPSJUINUIBUDPNCJOFT.POUF $BSMPTJNVMBUJPOXJUIWBMVFBOEQPMJDZOFUXPSLT6TJOHUIJTTFBSDIBMHPSJUIN PVS QSPHSBN"MQIB(PBDIJFWFEBXJOOJOHSBUFBHBJOTUPUIFS(PQSPHSBNT  BOEEFGFBUFEUIFIVNBO&VSPQFBO(PDIBNQJPOCZHBNFTUP5IJTJTUIF pSTUUJNFUIBUBDPNQVUFSQSPHSBNIBTEFGFBUFEBIVNBOQSPGFTTJPOBMQMBZFSJOUIF GVMMTJ[FEHBNFPG(P BGFBUQSFWJPVTMZUIPVHIUUPCFBUMFBTUBEFDBEFBXBZ    
  7. 0WFSWJFXPG"MQIB(P 1PMJDZ /FUXPSL 7BMVF /FUXPSL 3PMMPVU 1PMJDZ 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG8JO3BUF

    6TFEGPSQMBZPVU -PHJTUJD3FHSFTTJPO 'BTU 6TFEGPS/PEF4FMFDUJPO&YQBOTJPO $// 'BTU $// 3FDPSEPGTUSPOH QMBZFST 4FMGQMBZ 3- .$54
  8. 3PMMPVU1PMJDZ w -PHJTUJD3FHSFTTJPOXJUIXFMMLOPXOGFBUVSFT TFFUIFUBCMFCFMPX VTFEJO UIJTpFME w 5SBJOFEXJUINJMMJPOQPTJUJPOTGSPNUIF,(4(P4FSWFS IUUQT XXXHPLHTDPN

     w 5IJTNPEFMVTFEGPS3PMMPVU EFUBJMTXJMMCFFYQMBJOFEMBUFS  *OUPUBM'FBUVSFT&YUFOEFE5BCMFPG 4JMWFS
  9. -PHJTUJD3FHSFTTJPO    x1 x2 x109747 Σ u =

    109747 ∑ k=1 wk xk ˜ p = 1 1 + e−u w -PHJTUJD3FHSFTTJPOXJUIXFMMLOPXOGFBUVSFTVTFE JOUIJTpFME w 5SBJOFEXJUINJMMJPOQPTJUJPOTGSPNUIF,(4(P 4FSWFS IUUQTXXXHPLHTDPN
  10. 0WFSWJFXPG"MQIB(P 1PMJDZ /FUXPSL 7BMVF /FUXPSL 3PMMPVU 1PMJDZ 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG8JO3BUF

    6TFEGPSQMBZPVU -PHJTUJD3FHSFTTJPO 'BTU 6TFEGPS/PEF4FMFDUJPO&YQBOTJPO $// 'BTU $// 3FDPSEPGTUSPOH QMBZFST 4FMGQMBZ 3- .$54
  11. 4-1PMJDZ/FUXPSL w $POWPMVUJPOBM/FVSBM/FUXPSL w 5SBJOFEXJUI,(4EBUBTFUNJMMJPOQPTJUJPOTGSPNHBNFTQMBZFECZ,(4 UPEBO w $IBOOFMT 'FBUVSFT JTQSFQBSFE

    /FYUTMJEFFYQMBJOTEFUBJMT  https://senseis.xmp.net/?Go 19 x 19 48 Channel 19 19 5 5 5 5 3 3 3 3 19 19 .... .... 3 3 3 3 19 19 19 19 0VUQVU1SPCPGUIFOFYUNPWF
  12. 0WFSWJFXPG"MQIB(P 1PMJDZ /FUXPSL 7BMVF /FUXPSL 3PMMPVU 1PMJDZ 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG8JO3BUF

    6TFEGPSQMBZPVU -PHJTUJD3FHSFTTJPO 'BTU 6TFEGPS/PEF4FMFDUJPO&YQBOTJPO $// 'BTU $// 3FDPSEPGTUSPOH QMBZFST 4FMGQMBZ 3- .$54
  13. 7BMVF/FUXPSL w "MQIB(PVTFTUIF3-QPMJDZOFUXPSL UPHFOFSBUFUSBJOJOHEBUBGPSUIF 7BMVF/FUXPSL XIJDIQSFEJDUXJO SBUF w 5SBJOJOHEBUB 1PTJUJPO

    8JO-PTF  NJMMJPO w *UUPPLXFFLXJUI(16 w 5SBJOJOHBMTPUPPLXFFLXJUI (16 w 5IFOFUXPSLQSPWJEFT&WBMVBUJPO GVODUJPOGPS(P UIBUDPOTJEFSFEUPCF IBSEQSFWJPVTMZ  'JHPG 4JMWFS
  14. 3PMMPVU1PMJDZ 1PMJDZ/FUXPSL 7BMVF/FUXPSL .PEFM -PHJTUJD 3FHSFTTJPO $// -BZFST $// -BZFST

    5JNFGPS FWBMVBUJPOPGB TUBUF ЖT NT NT 5JNFGPSQMBZPVU NPWFT NT T  PGQMBZPVUT QFSTFD "CPVU "CPVU  "DDVSBDZ   
  15. 0WFSWJFXPG"MQIB(P 1PMJDZ /FUXPSL 7BMVF /FUXPSL 3PMMPVU 1PMJDZ 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG.PWF 1SFEJDUJPOPG8JO3BUF

    6TFEGPSQMBZPVU -PHJTUJD3FHSFTTJPO 'BTU 6TFEGPS/PEF4FMFDUJPO&YQBOTJPO $// 'BTU $// 3FDPSEPGTUSPOH QMBZFST 4FMGQMBZ 3- .$54
  16. .POUF$BSMP5SFF4FBSDI .POUF$BSMPUSFFTFBSDI .$54 JTBIFVSJTUJD TFBSDIBMHPSJUINGPSEFDJTJPOQSPDFTTFT 5IFGPDVTPG.POUF$BSMPUSFFTFBSDIJTPO UIFBOBMZTJTPGUIFNPTUQSPNJTJOHNPWFT  FYQBOEJOHUIFTFBSDIUSFFCBTFEPOSBOEPN TBNQMJOHPGUIFTFBSDITQBDF5IFBQQMJDBUJPO

    PG.POUF$BSMPUSFFTFBSDIJOHBNFTJTCBTFE PONBOZQMBZPVUT*OFBDIQMBZPVU UIFHBNF JTQMBZFEPVUUPUIFWFSZFOECZTFMFDUJOH NPWFTBUSBOEPN5IFpOBMHBNFSFTVMUPG FBDIQMBZPVUJTUIFOVTFEUPXFJHIUUIF OPEFTJOUIFHBNFUSFFTPUIBUCFUUFSOPEFT BSFNPSFMJLFMZUPCFDIPTFOJOGVUVSF QMBZPVUT #SPXOF
  17. .$54&YBNQMF N: 0, Q: 0 *OJUJBM4UBUF /PGWJTJUTUPUIFTUBUF 2&YQFDUFESFXBSE 4FMFDUJPO 4FMFDUOPEFUIBUNBYJNJ[FT

    Q(s, a) + Cp 2 log ns ns,a N: 1, Q: 0 N: 1, Q: 0 N: 0, Q: 0 N: 0, Q: 0 N: 0, Q: 0 'JSTUUFSN&TUJNBUFESFXBSE 4FDPOEUFSN#JBTUFSN
 *UCBMBODFT&YQMPSBUJPOWT&YQMPJUBUJPO "VFS 1  
 *OUIJTDBTF JU`TSBOEPN ᶃ ᶄ
  18. N: 1, Q: 0 N: 1, Q: 0 N: 0,

    Q: 0 N: 0, Q: 0 N: 0, Q: 0 Win 3PMMPVU 3BOEPNMZQMBZHBNFBOE
 pOEPVUXJOMPTF N: 1, Q: 0 N: 1, Q: 1 N: 0, Q: 0 N: 0, Q: 0 N: 0, Q: 0 Win #BDLVQ 3FOFX2PGUIFTUBUF ᶅ ᶆ
  19. N: 1, Q: 0 N: 1, Q: 1 N: 1,

    Q: 1 N: 1, Q: -1 N: 1, Q: 1 N: 5, Q: 0.25 N: 2, Q: 1 N: 1, Q: 1 N: 1, Q: -1 N: 1, Q: 1 N: 0, Q: 0 N: 0, Q: 0 N: 0, Q: 0 N: 5, Q: 0.25 N: 2, Q: 1 N: 1, Q: 1 N: 1, Q: -1 N: 1, Q: 1 &YQBOTJPO &YQBOEUSFFXIFOBOPEFJTWJTJUFE
 DFSUBJOQSFEFpOFEUJNFT
 JOUIJTDBTF ᶇ ᶈ ᶉ
  20. w #JBTJTFWBMVBUFECZUIFPSJHJOBM#JBT 0VUQVUPGUIF 4-QPMJDZOFUXPSL
 w &WBMVBUJPOPGXJOSBUFQMBZPVU 0VUQVUPGUIF WBMVF/FUXPSL w .BTTJWFQBSBMMFMDPNQVUBUJPOVTJOHCPUI(16T

      BOE$16T  Q(s, a) = (1 − λ) Wv (s, a) Nv (s, a) + λ Wr (s, a) Nr (s, a) u(s, a) = cpuct P(s, a) ∑ b Nr (s, b) 1 + Nr (s, a) .$54JO"MQIB(P 7BMVF/FUXPSL .$54 P(s, a)
  21. "CTUSBDU "MPOHTUBOEJOHHPBMPGBSUJpDJBMJOUFMMJHFODFJTBOBMHPSJUINUIBUMFBSOT  UBCVMBSBTB TVQFSIVNBOQSPpDJFODZJODIBMMFOHJOHEPNBJOT3FDFOUMZ  "MQIB(PCFDBNFUIFpSTUQSPHSBNUPEFGFBUBXPSMEDIBNQJPOJOUIFHBNFPG (P5IFUSFFTFBSDIJO"MQIB(PFWBMVBUFEQPTJUJPOTBOETFMFDUFENPWFTVTJOH EFFQOFVSBMOFUXPSLT5IFTFOFVSBMOFUXPSLTXFSFUSBJOFECZTVQFSWJTFE MFBSOJOHGSPNIVNBOFYQFSUNPWFT

    BOECZSFJOGPSDFNFOUMFBSOJOHGSPNTFMG QMBZ)FSFXFJOUSPEVDFBOBMHPSJUINCBTFETPMFMZPOSFJOGPSDFNFOU MFBSOJOH XJUIPVUIVNBOEBUB HVJEBODFPSEPNBJOLOPXMFEHFCFZPOE HBNFSVMFT"MQIB(PCFDPNFTJUTPXOUFBDIFSBOFVSBMOFUXPSLJTUSBJOFEUP QSFEJDU"MQIB(P`TPXONPWFTFMFDUJPOTBOEBMTPUIFXJOOFSPG"MQIB(P`T HBNFT5IJTOFVSBMOFUXPSLJNQSPWFTUIFTUSFOHUIPGUIFUSFFTFBSDI SFTVMUJOH JOIJHIFSRVBMJUZNPWFTFMFDUJPOBOETUSPOHFSTFMGQMBZJOUIFOFYUJUFSBUJPO 4UBSUJOHUBCVMBSBTB PVSOFXQSPHSBN"MQIB(P;FSPBDIJFWFETVQFSIVNBO QFSGPSNBODF XJOOJOHrBHBJOTUUIFQSFWJPVTMZQVCMJTIFE DIBNQJPO EFGFBUJOH"MQIB(P 5BCVMBSBTBJTB-BUJOQISBTFPGUFOUSBOTMBUFEBTDMFBOTMBUF  
  22. 1PJOU%VBM/FUXPSL https://senseis.xmp.net/?Go 19 x 19 48 Channel 19 19 5

    5 5 5 3 3 3 3 19 19 .... .... 3 3 3 3 19 19 19 19 Output1: Prediction of the next move 19 19 Output Layer Output2: Win Rate w -BZFST $POWPMVUJPOBM/FVSBM/FUXPSL w &BDIMBZFSYDPOWPMVUJPOMBZFS #BUDIOPSNBMJ[BUJPO 3FMV w -BZFSdBSF3FT/FU w 5SBJOFECZTFMGQMBZ EFUBJMTBSFEFTDSJCFEMBUFS  w $IBOOFMT 'FBUVSFT JTQSFQBSFE UIFOFYUTMJEFTIPXTEFUBJMT  w -FBSOJOHNFUIPEPGUIJTOFUXPSLEJTDVTTFEMBUFS 'PSOPX MFU`TBTTVNFXFIBWFUSBJOFEJUOJDFMZ  p v
  23. "MQIB(P;FSPJTMFTTEFQFOET POIBOEDSBGUFEGFBUVSFT 'FBUVSFTPG"MQIB(P 4JMWFS 'FBUVSF PGQMBOFT 1PTJUJPOPGCMBDL TUPOFT  1PTJUJPOPGCMBDL

    TUPOFT  1PTJUJPOPGCMBDL TUPOFTL d TUFQT CFGPSF  1PTJUJPOPGXIJUF TUPOFTL d TUFQT CFGPSF  5VSO  'FBUVSFTPG"MQIB(P;FSP 4JMWFS
  24. 1PJOU*NQSPWFNFOUPG .$54 w .$54BMHPSJUINVTFTUIFGPMMPXJOHWBMVFGPSTUBUF TFMFDUJPO w /PQMBZPVU JUKVTUSFMJFTPOWBMVF Q(s, a)

    + u(s, a) Q(s, a) = W(s, a) N(s, a) u(s, a) = cpuct p(s, a) ∑ b N(s, b) 1 + N(s, a) 8JO3BUF #JBT 1SFEJDUJPO PGUIFNPWFB
  25. 1PJOU*NQSPWFNFOUT PO3- (p, v) = fθ (s)BOEl = (z −

    v)2 − πT log p + c∥θ∥2 w 5IFEVBMOFUXPSL QBSBNFUFS BDDVNVMBUFEBUB CZTFMGQMBZ TUFQSFQFBUFEUIPVTBOEUJNFT  w #BTFEPOUIFSFTVMU VQEBUFUIFQBSBNFUFSPGUIF OFUXPSL TUFQ BOEHFUOFXQBSBNFUFS w -FUUXPOFUXPSLJOTUBOUJBUJPOTDPNQFUF VQEBUFUIF OFUXPSLQBSBNFUFSJGUIFOFXQBSBNFUFSTFUXJOT w 3FQFBUTUFQBOETUFQ θ′ θ
  26. 4UFQ%BUB"DDVNVMBUJPO w %PBTFMGQMBZ4UPSFUIFPVUDPNF[ w 4UPSFBMM T К [ UVQMFTJOUIFHBNF w

    5IFQPMJDZКJTDBMDVMBUFEBT w 3FQFBUUIFBCPWFQSPDFTTFTUJNFT πa = N(s, a)1/γ ∑ b N(s, b)1/γ
  27. 4UFQ1BSBNFUFSVQEBUF w $BMDVMBUFUIFMPTTGVODUJPOVTJOH T К [ FWBMVBUFE JOUIFQSFWJPVTTUFQ (p, v)

    = fθ (s)BOEl = (z − v)2 − πT log p + c∥θ∥2 w 6QEBUFQBSBNFUFSVTJOHHSBEJFOUEFTDFOUNFUIPE θ′ ← θ − α ⋅ Δθ
  28. 3FGFSFODFT  4JMWFS % )VBOH " .BEEJTPO $+ (VF[ "

    4JGSF - WBOEFO%SJFTTDIF ( ʜ)BTTBCJT %  .BTUFSJOHUIF HBNFPG(PXJUIEFFQOFVSBMOFUXPSLTBOEUSFFTFBSDI/BUVSF   rIUUQTEPJPSH OBUVSF
 "MQIB(P  0UTVLJ 5 .JZBLF  4BJLZPJHPFBJBSVGBHPLBJUBJTIJOTIP4IJOTPHBLVTIVNPOUFLBSVSPLJUBOTBLVLZPLB HBLVTIVLBSBNJUBTPOPTIJLVNJ4IPFJTIB  4JMWFS % 4DISJUUXJFTFS + 4JNPOZBO , "OUPOPHMPV * )VBOH " (VF[ " ʜ)BTTBCJT %  .BTUFSJOHUIFHBNF PG(PXJUIPVUIVNBOLOPXMFEHF/BUVSF   rIUUQTEPJPSHOBUVSF
 "MQIB(P;FSP  #SPXOF $ 1PXMFZ & 8IJUFIPVTF % -VDBT 4 .FNCFS 4 $PXMJOH 1* ʜ$PMUPO 4  "4VSWFZPG.POUF $BSMP5SFF4FBSDI.FUIPET*&&&53"/4"$5*0/40/$0.165"5*0/"-*/5&--*(&/$&"/%"**/(".&4    IUUQTEPJPSH5$*"*(  "VFS 1  'JOJUFUJNFBOBMZTJTPGUIFNVMUJBSNFECBOEJUQSPCMFNXJUILOPXOUSFOE*&&&$POHSFTTPO &WPMVUJPOBSZ$PNQVUBUJPO $&$   rIUUQTEPJPSH$&$  $BJ 1 -VP : 4BYFOB " )TV % -FF 84  -F54%SJWF%SJWJOHJOB$SPXECZ-FBSOJOHGSPN5SFF4FBSDI 3FUSJFWFEGSPNIUUQTBSYJWPSHQEGQEG