Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Natural Policy Gradient and TRPO by Andrew Ga...

Avatar for Tokyo Machine Learning Society Tokyo Machine Learning Society
October 28, 2016
860
1

A Natural Policy Gradient and TRPO by Andrew Gambardella - RRS #1

Avatar for Tokyo Machine Learning Society

Tokyo Machine Learning Society

October 28, 2016

More Decks by Tokyo Machine Learning Society

See All by Tokyo Machine Learning Society
Natural Gradient Learning by Shun'ichi Amari, RIKEN - RRS #1
Avatar for Tokyo Machine Learning Society tmls
1
430
Generative Adversarial Networks by Hasanov Vagif - TMLS #4
Avatar for Tokyo Machine Learning Society tmls
3
540
Character-Level Machine Translation with Bi-scale RNNs by Joe Bullard - TMLS #4
Avatar for Tokyo Machine Learning Society tmls
2
420
NVIDIA Deep Learning Platform and latest information by Takeshi Izaki, Nvidia - TMLS #4　
Avatar for Tokyo Machine Learning Society tmls
1
350
Neural Translation by David and Stefano - TMLS#3
Avatar for Tokyo Machine Learning Society tmls
3
340
Extending Cloud Vision API by Ray Sakai -TMLS#3
Avatar for Tokyo Machine Learning Society tmls
2
200
Bayesian Optimization with TensorFlow/Keras by Keisuke Kamataki - TMLS #2
Avatar for Tokyo Machine Learning Society tmls
1
9.3k

Featured

See All Featured
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.9k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
3
1.1k
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
2.8k
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
220
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
1
260
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
350
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
11k
A Soul's Torment
Avatar for Nat Ma seathinner
6
3k
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
820
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
141
7.6k

Transcript

  1. "/BUVSBM1PMJDZ(SBEJFOU
 
 5SVTU3FHJPO1PMJDZ0QUJNJ[BUJPO "OESFX(BNCBSEFMMB
 3FBDUJWF3FTFBSDI4FNJOBS

  2. 3FJOGPSDFNFOU-FBSOJOH

  3. "QQSPBDIFT ↟ 7BMVFCBTFE ↟ 1PMJDZCBTFE ↟ -FBSOBQPMJDZ BDUPS BOETUBUFWBMVFGVODUJPO DSJUJD

     KPJOUMZ ↟ 0UIFST

  4. %FFQ3FJOGPSDFNFOU-FBSOJOH

  5. #VUXIZMFBSOB2GVODUJPO ↟ *ONBDIJOFMFBSOJOH XFPQUJNJ[FGPSUIFUIJOHXFDBSF BCPVU ↟ *OUIJTDBTF JUsTUIFFYQFDUFEUPUBMSFXBSEPGUIFQPMJDZ ↟ $BOFNQJSJDBMMZTIPXUIBU%2/JTOPUWFSZHPPE

    DPNQBSFEUPQPMJDZHSBEJFOUBOEBDUPSDSJUJDNFUIPET TFF"TZODISPOPVT.FUIPETGPS%FFQ3FJOGPSDFNFOU -FBSOJOH

  6. 1PMJDZ(SBEJFOU

  7. 3&*/'03$&FTUJNBUPS
 BMTPLOPXOBTMJLFMJIPPESBUJPFTUJNBUPSTDPSF GVODUJPOFTUJNBUPS -FUYCFBSBOEPNWBSJBCMFXJUIY_Q ↟Ƕ 5IFO

  8. .BOZEJGGFSFOUDIPJDFTGPSG

  9. 1SPTPGQPMJDZHSBEJFOUNFUIPET DPNQBSFEUPEFFQ2MFBSOJOH ↟ 0QUJNJ[JOHGPSXIBUXFBDUVBMMZXBOUnNBYJNVN FYQFDUFEUPUBMSFXBSEPGUIFQPMJDZ ↟ #FUUFSGPSIJHIEJNFOTJPOBMUBTLT ↟ #FUUFSGPSDPOUJOVPVTUBTLT NBQZPVSTUBUFUPUIFNFBO

    BOEWBSJBODFPGB(BVTTJBOEJTUSJCVUJPO  ↟ $BOCPPUTUSBQPGGPGQPMJDJFTDSFBUFECZJNJUBUJPOMFBSOJOH ↟ $BOMFBSOBTUPDIBTUJDQPMJDZ HPPEJGTPNFTUBUFTBSF BMJBTFE

  10. $POTPGQPMJDZHSBEJFOUNFUIPET DPNQBSFEUPEFFQ2MFBSOJOH ↟ %2/NPSFTBNQMFFGGJDJFOUPODFSUBJOUBTLT

  11. $POTPGQPMJDZHSBEJFOU NFUIPETJOHFOFSBM ↟ 'BJSMZEJGGJDVMUUPEFCVH ↟ &TUJNBUPSPGUIFHSBEJFOUIBTIJHIWBSJBODF ↟ $POWFSHFUPBMPDBMNJOJNVN DBOsUUSBJOCJHOFVSBMOFUT CFDBVTFPGQSFWJPVTQPJOUBOZHVBSBOUFFTBCPVUMPDBM

    NJOJNBPGFYUSFNFMZIJHIEJNFOTJPOBMGVODUJPOTEPOPU IPME

  12. )PXUPEFBMXJUIFTUJNBUPS WBSJBODF ↟ $FSUBJOUSJDLTDBOIFMQVT 4FF3FJOGPSDFNFOU-FBSOJOH /FVSBM5VSJOH.BDIJOFT PSTFF-FBSOJOH0OMJOF "MJHONFOUTXJUI$POUJOVPVT3FXBSET1PMJDZ(SBEJFOU  ↟

    8FDBOBMTPEPHSBEJFOUEFTDFOUVTJOHBEJSFDUJPOBOE PSTUFQTJ[FUIBUNBLFTNPSFTFOTFGPS3-UIBOWBOJMMB 4(%

  13. /BUVSBM1PMJDZ(SBEJFOU ↟ 6TF"NBSJsTUIFPSZUPEFGJOFBNFUSJDCBTFEOPUPOUIFDIPJDF PGDPPSEJOBUFT CVUPOUIFNBOJGPMEUIBUUIFTFDPPSEJOBUFT QBSBNFUFSJ[F ↟ "TTIPXOCZ"NBSJ UIF'JTIFSJOGPSNBUJPONBUSJYJTBO JOWBSJBOUNFUSJDPOUIFTQBDFPGQBSBNFUFSTPGQSPCBCJMJUZ

    EJTUSJCVUJPOT*UEFGJOFTUIFTBNFEJTUBODFCFUXFFOUXPQPJOUT SFHBSEMFTTPGUIFDIPJDFPGDPPSEJOBUFT JFUIF QBSBNFUFSJ[BUJPO  ↟ 5IJTHJWFTTUFFQFTUEFTDFOUEJSFDUJPO(− Ƕ ˜ǵ Ƕ XIFSF˜ǵ Ƕ  JTUIFEFTDFOUEJSFDUJPOHJWFOCZUIFtWBOJMMBuQPMJDZHSBEJFOU

  14. 8IZVTFUIFOBUVSBMHSBEJFOU GPS3- ↟ 8FXBOUUIFTUBUFTXFTFFXJUIPVSOFXQPMJDZUPCF TJNJMBSUPUIPTFXFTBXXJUIPVSPMEQPMJDZ TPXFDBO SFVTFLOPXMFEHFGSPNQBTUSPMMPVUT ↟ #ZDPOTUSBJOJOHPVSOFXQPMJDZUPCFtDMPTFuUPPVSPME QPMJDZJOUIJTXBZ

    XFDBODPOUSPMUIFBDUJPOEJTUSJCVUJPO DIBOHFBTBQSPYZ

  15. 8IZVTFUIFOBUVSBMHSBEJFOU GPS3- ↟ *GXFUBLFBTNBMMTUFQJO&VDMJEFBOQBSBNFUFSTQBDF  FWFOBTNBMMTUFQDPVMEDBVTFXJMEDIBOHFTJOUIF BDUJPOEJTUSJCVUJPO*UJTUIFSFGPSFFTTFOUJBMUIBUXFUBLF TUFQTJOUIFOBUVSBMHSBEJFOUEJSFDUJPO ↟ "MTP

    IPXDBOXFDIPPTFBOBQQSPQSJBUFTUFQTJ[F  XIFOUIFTUBUJTUJDTPGPVSEBUBBSFDPOTUBOUMZDIBOHJOH  BTPVSQPMJDZDIBOHFT TPEPFTPVSEBUB
  16. None

  17. "TMJHIUJTTVF ↟ 5IFOBUVSBMQPMJDZHSBEJFOUNVTUGPSNBOEJOWFSUUIF FNQJSJDBM'*. ↟ 5IJTJTUPPFYQFOTJWF

  18. 5SVODBUFE/BUVSBM1PMJDZ (SBEJFOU 5/1( ↟ *OTUFBEPGDPNQVUJOHUIF'*. DPNQVUFUIFTBNF EJSFDUJPOPGOBUVSBMQPMJDZHSBEJFOUVTJOHBDPOKVHBUF HSBEJFOUBMHPSJUINUIBUPOMZSFRVJSFTDPNQVUJOHBO BQQSPYJNBUJPOPG( Ƕ

    WGPSBSCJUSBSZWFDUPSW

  19. 5SVTU3FHJPO1PMJDZ 0QUJNJ[BUJPO 5310 ↟ $POTUSBJOUIF,-EJWFSHFODFCFUXFFOPVSPMEQPMJDZ BOEPVSOFXQPMJDZUPCFXJUIJOTPNFUSVTUSFHJPO SBEJVTDz ↟ #PVOEPO,-EJWFSHFODFJTBMTPBCPVOEPOIPXNVDI PVSQFSGPSNBODFDBOEFHSBEFXJUIJOUIJTPOFTUFQ

  20. 5/1(BOE5310 ↟ 5/1(DBOCFPCUBJOFEBTBTQFDJBMDBTFPG5310XJUIB MJOFBSBQQSPYJNBUJPOUP-BOEBRVBESBUJDBQQSPYJNBUJPO UPUIF%,- DPOTUSBJOU/PUJDFTUFQTJ[FǹJO5/1(
 
 
 
 


    
 
 
 5310 5/1(XBQQSPYJNBUJPO
  21. None

  22. "OEOPXGPSTPNFUIJOH DPNQMFUFMZEJGGFSFOU

  23. (FOFSBUJWF"EWFSTBSJBM /FUXPSLT ↟ "SFDFOUNFUIPEGPSVOTVQFSWJTFEMFBSOJOH ↟ (FOFSBUPS(TBNQMFTGSPNBEJTUSJCVUJPOBOEEJTDSJNJOBUPS% DMBTTJGJFTTBNQMFTBTSFBMPSGBMTF ↟ (JTUSZJOHUPtUSJDLu%CZJODSFBTJOHUIFQSPCBCJMJUZUIBU%XJMM DMBTTJGZ(sTTBNQMFTBTSFBM%JTUSZJOHUPNJOJNJ[FUIF

    QSPCBCJMJUZPGCFJOHUSJDLFE ↟ 5IJTHJWFTVTUIFBEWFSTBSJBMMPTTGVODUJPO
 
 


  24. 8IBUEPFTUIJTIBWFUP EPXJUI3-

  25. *NJUBUJPO-FBSOJOH ↟ *NJUBUJPOMFBSOJOHTFUUJOH(JWFOUSBKFDUPSJFTGSPNBO FYQFSU MFBSOBQPMJDZ ↟ 3FHVMBSTVQFSWJTFEMFBSOJOH CFIBWJPSBMDMPOJOH XJMM BMXBZTXPSLJOUIFJOGJOJUFEBUBSFHJNF

    ↟ $BOXFNBLFTPNFUIJOHNPSFTBNQMFFGGJDJFOU  LOPXJOHUIBUXFBSFMPPLJOHGPSBO.%1QPMJDZ

  26. (FOFSBUJWF"EWFSTBSJBM *NJUBUJPO-FBSOJOH ↟ *NJUBUJPOMFBSOJOHDBOCFGPSNVMBUFEBTBNJOJNBYQSPCMFN XIFSFUIFBJNJTUPMFBSOUIFDPTUGVODUJPOUIBUNJOJNJ[FT UIFBNPVOUCZXIJDIUIFPQUJNBMQPMJDZPVUQFSGPSNTUIF FYQFSUQPMJDZ ↟ #VUMFBSOJOHBDPTUGVODUJPOJTFRVJWBMFOUUPNJOJNJ[JOHUIF SFHVMBSJ[FEEJTUBODFCFUXFFOTUBUFBDUJPOPDDVQBODZ

    EJTUSJCVUJPOTGPSUIFFYQFSUQPMJDZBOEMFBSOFEQPMJDZ ↟ #ZVTJOHBEFFQOFVSBMOFUXPSLXJUIUIFDSPTTFOUSPQZMPTT BTUIFEJTUBODFCFUXFFOPDDVQBODZEJTUSJCVUJPOT XFDBO GPSHPMFBSOJOHBDPTUGVODUJPODPNQMFUFMZ

  27. (FOFSBUJWF"EWFSTBSJBM *NJUBUJPO-FBSOJOH

  28. None

  29. 3FTPVSDFT ↟ %FFQ3FJOGPSDFNFOU-FBSOJOH1POHGSPN1JYFMT "OESFK ,BSQBUIZsTCMPH ↟ -FDUVSF1PMJDZ(SBEJFOU %BWJE4JMWFST3-$PVSTF ↟ %FFQ3FJOGPSDFNFOU-FBSOJOH"5VUPSJBM

    0QFO"* ↟ 1PMJDZ(SBEJFOU.FUIPET 4DIPMBSQFEJB

  30. 3FGFSFODFT ↟ "NBSJ 4/BUVSBMHSBEJFOUXPSLTFGGJDJFOUMZJOMFBSOJOH/FVSBM$PNQVUBUJPO      ↟

    %VBO: $IFO9 )PVUIPPGU3 4DIVMNBO+ BOE"CCFFM1#FODINBSLJOHEFFQ SFJOGPSDFNFOUMFBSOJOHGPSDPOUJOVPVTDPOUSPMBS9JWQSFQSJOUBS9JW  ↟ )P + BOE&SNPO 4(FOFSBUJWFBEWFSTBSJBMJNJUBUJPOMFBSOJOHBS9JWQSFQSJOUBS9JW   ↟ ,BLBEF 4"OBUVSBMQPMJDZHSBEJFOU*O"EWBODFTJO/FVSBM*OGPSNBUJPO1SPDFTTJOH 4ZTUFNT QQm.*51SFTT  ↟ -VP: $IJV$ +BJUMZ. BOE4VUTLFWFS*-FBSOJOHPOMJOFBMJHONFOUTXJUIDPOUJOVPVT SFXBSETQPMJDZHSBEJFOUBS9JWQSFQSJOUBS9JW  ↟ .OJI7 #BEJB" .JS[B. (SBWFT" -JMMJDSBQ5 )BSMFZ5 4JMWFS% BOE,BWVLDVPHMV, "TZODISPOPVTNFUIPETGPSEFFQSFJOGPSDFNFOUMFBSOJOHBS9JWQSFQSJOUBS9JW  

  31. 3FGFSFODFT ↟ .OJI 7 ,BWVLDVPHMV , 4JMWFS % (SBWFT "

    "OUPOPHMPV * 8JFSTUSB % BOE 3JFENJMMFS .1MBZJOH"UBSJXJUIEFFQSFJOGPSDFNFOUMFBSOJOHBS9JWQSFQSJOUBS9JW   ↟ 1GBV % BOE7JOZBMT 0$POOFDUJOHHFOFSBUJWFBEWFSTBSJBMOFUXPSLTBOEBDUPS DSJUJDNFUIPETBS9JWQSFQSJOUBS9JW  ↟ 4DIVMNBO + -FWJOF 4 .PSJU[ 1 +PSEBO . BOE"CCFFM 15SVTUSFHJPOQPMJDZ PQUJNJ[BUJPOBS9JWQSFQSJOUBS9JW  ↟ 4DIVMNBO + .PSJU[ 1 -FWJOF 4 +PSEBO . BOE"CCFFM 1)JHIEJNFOTJPOBM DPOUJOVPVTDPOUSPMVTJOHHFOFSBMJ[FEBEWBOUBHFFTUJNBUJPOBS9JWQSFQSJOUBS9JW   ↟ ;BSFNCB 8 BOE4VUTLFWFS *3FJOGPSDFNFOUMFBSOJOHOFVSBMUVSJOHNBDIJOFT BS9JWQSFQSJOUBS9JW 

  32. 2VFTUJPOT