reinforcement_learning_.pdf

ъച೟ण੉ۿ߂पण 8POTFPL+VOH

%ZOBNJDQSPHSBNNJOH .POUF$BSMPNFUIPET 4BSTB 2MFBSOJOH ъച೟ण੉ۆ ъച೟ण੉ۿ߂पण 4VQFS.BSJPXJUI  %FFQ3FJOGPSDFNFOU-FBSOJOH ъച೟णജ҃߂୭न౟ے٘
'BTUFS-FBSOJOH ٩۞׬ ъച೟णपण

#FGPSF%FFQMFBSOJOH "GUFS%FFQMFBSOJOH 5BCVMBS *NBHF UFYU WPJDFj

$MBTTJD3- %FFQ-FBSOJOH Ҋ੹ъച೟ण੉઺ਃೠ੉ਬ

2MFBSOJOH $//%2/ %2/

पणਸਤ೧ https://github.com/wonseokjung/daejeon_rl_1219

$PMBCਵ۽गಌܻ݃য়೟ण

$PMBCਵ۽गಌܻ݃য়೟ण Ѩ࢝যDPMBCژחHPPHMFDPMBC

$PMBCਵ۽गಌܻ݃য়೟ण ࢶఖ

$PMBCਵ۽गಌܻ݃য়೟ण 0QFOOPUFCPPLࢶఖ

$PMBCਵ۽गಌܻ݃য়೟ण (JUIVC઱ࣗ ഑਷ઁ੉ܴ XPOTFPLKVOH

$PMBCਵ۽गಌܻ݃য়೟ण ࢶఖ

$PMBCਵ۽गಌܻ݃য়೟ण /PUFCPPLTFUUJOHT

$PMBCਵ۽गಌܻ݃য়೟ण (16ࢎਊ

$PMBCਵ۽गಌܻ݃য়೟ण ҳӖ۽Ӓੋ

$PMBCਵ۽गಌܻ݃য়೟ण प೯ߡౡ௿ܼೞৈಁః૑߂गಌܻ݃য়ജ҃׮਍۽٘

3FJOGPSDFNFOU-FBSOJOH

ALL ANIMALS HAVE THE ABILITY TO LEARN - ݽٚزޛ਷೟णמ۱੉੓׮ -
ৈѐ੄न҃ࣁನ݅ਸыҊ੓ח৘ࢂԘ݃ࢶ୽ژೠ೟णמ۱੉੓׮ - ݠ୍ܻࣻ߈ࢎIFBEXJUIESBXTSFGMFYਤ೷ೠޛ୓о੓ਸѪ੉ۄ౸ױীٮܲ߈ࢎ೯ز - ৘ࢂԘ݃ࢶ୽੄ݠܻܳѤܻ٘ݶੌ੿Ѣٍܻܳ۽р׮ HOW ANIMALS LEARN

HABITUATION HOW ANIMALS LEARN 'JSTUUSZ 4FDPOEUSZ 5IJSEUSZ

LAW OF EFFECT - &EXBSE5IPSOEJLF - -BXPGFGGFDUযڃ೯ز੄Ѿҗо݅઒झ۞਋ݶ׮਺ীبӒ೯زਸ߈ࠂೠ׮ ߈؀۽݅઒ೞ૑ঋਵݶӒ೯زਸೞ૑ঋח׮
- 3FJOGPSDFNFOU ъച ੉੹ীੌযդ೯زਸ߈ࠂೞѱ݅٘ח੗ӓ - 1VOJTINFOU ୊ߥ ੉੹ীੌযդ೯زਸೖೞѱ݅٘ח੗ӓ HOW ANIMALS LEARN

EXAMPLE OF THE LAW OF EFFECT HOW ANIMALS LEARN

INTERACTION WITH ENVIRONMENT &OWJSPONFOU &YQFSJFODF -FBSO *OUFSBDUJPO HOW HUMANS LEARN

HOW HUMANS LEARN? - 3FJOGPSDFNFOU੉੹ীੌযդ೯زਸ߈ࠂೞѱ݅٘ח੗ӓ - 1VOJTINFOU੉੹ীੌযդ೯زਸೖೞѱ݅٘ח੗ӓ HOW HUMANS LEARN

.BUIFNBUJDBMGSBNFXPSLGPSEFBMJOH XJUIEFDJTJPONBLJOH 1. Models an interaction between and Agent and
an World 2. Agent makes a decision 3. World responds to that decision with consequences - observation, reward "DUJ "HFOU &O 3F At Rt 4UB St Rt+1 St+1 &OWJSPONFOU "HFOU

'JSTUCJHTVDDFTTFTJO3FJOGPSDFNFOU-FBSOJOH 1. It came from the combination of Deep Learning
2. Playing Atari 3. AlphaGo 4. Robotic manipulation

3FJOGPSDFNFOU-FBSOJOH TFRVFOUJBMEFDJTJPONBLJOH

8IBUEPFTFOEUPFOEMFBSOJOHNFBOGPS TFRVFOUJBMEFDJTJPONBLJOH 1. You are walking to the jungle and
see the tiger 2. You need to take some action (You may wanna run away ) 3. Tiger -> perception (“oh yeah it is a tiger”) -> control system -> “Run”

4JNQMJGJFE 1. You don’t even know that is a tiger
2. You just know that if getting eaten is a bad thing, not getting eaten is a good thing 3. Tiger -> control system -> “Run”

Action, Observation and Rewards 1. Agent makes decisions : actions
2. The world responds with consequences : observations and rewards

)PXUPBOJNBMTMFBSO 1. Actions : muscle contractions 2. Observations : sight,
smell 3. Rewards : food

&OWJSPONFOU 3FXBSE At Rt St Rt+1 St+1 5BQUIFCBMM 1PTJUJWF3FXBSE 

3PCPUJDT 1. Actions : motor current or torque 2. Observations
: Camera images 3. Rewards : task success measure

*OWFOUPSZ.BOBHFNFOU 1. Actions : what to purchase 2. Observations :
Inventory levels 3. Rewards : profit

*NBHFDMBTTJGJDBUJPO 1. Actions : label the output 2. Observations :
Image pixels 3. Rewards : correct or not correct

3FJOGPSDFNFOU-FBSOJOH $PNQBSFUP4VQFSWJTFEMFBSOJOH

5FSNJOPMPHZBOEOPUBUJPO CAT  DOG  TIGER 1. Supervised learning   input :
pixels   output : categorical random variable (label of the object)   Model : what you want to learn

1. pet a cat 2. ignoring 3. give foods *O3FJOGPSDFNFOU-FBSOJOH
1. output could be not labeling, but actions

4FRVFOUJBM%FDJTJPO 1. pet a cat 2. ignoring 3. give foods

4FRVFOUJBM%FDJTJPO 1. pet a cat 2. ignoring 3. give foods
at πθ (at ∣ ot ) ot st ot at πθ (at ∣ ot ) πθ (at ∣ st ) - state - observation - action - policy - policy ( fully observed )

3FJOGPSDFNFOU-FBSOJOH TUBUF PCTFSWBUJPO SFXBSEGVODUJPO

TUBUFBOEPCTFSWBUJPO 1. State : Underlying state of the world ,
(ex : position, momentum, cat, mouse )  2. Observation : Image pixel (Underlying the state of the world ) but those are actually hidden inside the image , you actually the image to get those out

TUBUFBOEPCTFSWBUJPO 1. State : Summary of the world , using
it to predict the world 2. Observation : Consequence of the state but lossy consequence Observation State

3FXBSEGVODUJPOT )JHI3FXBSE ֫਷ࠁ࢚ উ੹ೞѱݾ੸૑ীب଱ -PX3FXBSE ծ਷ࠁ࢚ ҮాࢎҊ
1PMJDZউ੹ೞѱ਍੹ೞѱೞח੿଼೟ण )JHI3FXBSE -PX3FXBSE 3FXBSEGVODUJPOਸా೧೟ण

(SBQIJDBMMZNPEFM πθ (at ∣ ot ) πθ (at ∣ st
) - policy - policy ( fully observed ) st ot at - state - observation - action o1 s1 a1 o2 s2 a2 o3 s3 a3 1. Drawing a graphically model to relate state, observation, and action  2. Observing previous observations might give you more information p(st+1 ∣ st , at ) p(st+1 ∣ st , at )

.BSLPW%FDJTJPO1SPCMFN πθ (at ∣ ot ) πθ (at ∣ st
) - policy - policy ( fully observed ) st ot at - state - observation - action o1 s1 a1 o2 s2 a2 o3 s3 a3 p(st+1 ∣ st , at ) p(st+1 ∣ st , at ) .BSLPW%FDJTJPO1SPCMFN    TUBUF BDUJPO SFXBSE USBOTJUJPO١ਵ۽಴അغחъച೟णীࢲ੄ࣁ࢚ਸ੿੄

MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF
St Rt+1 St+1 REINFORCEMENT LEARNING

AGENT "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St Rt+1
St+1 REINFORCEMENT LEARNING

ACTION "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St Rt+1
St+1 REINFORCEMENT LEARNING

OBSERVATION, REWARD "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St
Rt+1 St+1 REINFORCEMENT LEARNING

TRAJECTORY REINFORCEMENT LEARNING (St , At , Rt+1 , St+1
) (St+1 , At+1 , Rt+2 , St+2 ) (St+2 , At+2 , Rt+3 , St+3 )

.BSLPWDIBJO s ∈ S s T 4UBUFTQBDF 5SBOTJUJPOPQFSBUPS 4UBUFTQBDF M
= S, T

.BSLPWDIBJO HSBQIJDBMMZ μt,i = p(st = i) s1 s2 s3
p(st+1 ∣ st , at ) p(st+1 ∣ st , at ) Ti,j = p(st+1 = i ∣ st = j) TUBUFKо઱য઎ਸٸTUBUFоJ੉ؼഛܫ UJNFTUFQUীTUBUFоJੌഛܫ

.BSLPWEFDJTJPOQSPDFTT s1 a1 s2 a2 s3 p(st+1 ∣ st ,
at ) p(st+1 ∣ st , at ) s ∈ S s A T a ∈ A 4UBUFTQBDF "DUJPOTQBDF 5SBOTJUJPOPQFSBUPS 4UBUFTQBDF "DUJPOTQBDF M = {S, A, T, r} r 3FXBSEGVODUJPO

.BSLPWEFDJTJPOQSPDFTT μt,i = p(st = i) Ti,j,k = p(st+1 =
i ∣ st = j, at = k) UJNFTUFQUীTUBUFоJੌഛܫ UJNFTUFQUীࢲTUBUFоK੉ҊBDUJPO੉Lੌ⮶ UJNFTUFQU ীࢲTUBUFоJੌഛܫ ξt,k = p(at = k) UJNFTUFQUীBDUJPO੉Lੌഛܫ r : SxA → R SFXBSEGVODUJPO

1BSUJBM0CTFSWFE.BSLPWEFDJTJPOQSPDFTT s ∈ S s A T a ∈ A
4UBUFTQBDF "DUJPOTQBDF 5SBOTJUJPOPQFSBUPS 4UBUFTQBDF "DUJPOTQBDF M = {S, A, O, T, E, r} O 0CTFSWBUJPOTQBDF E &NJTTJPOQSPCBCJMJUZ r 3FXBSEGVODUJPO o ∈ O PCTFSWBUJPOTQBDF o1 s1 a1 o2 s2 a2 o3 s3 a3 p(st+1 ∣ st , at ) p(st+1 ∣ st , at )

5IFHPBMPGSFJOGPSDFNFOUMFBSOJOH πθ (a ∣ s) θ ੋҕन҃ݎ੄XFJHIUT 1PMJDZח౵ۄݫఠܳ؀߸ೠ׮ θ ੋҕन҃ݎ਷ੑ۱ਵ۽TUBUFܳ߉ҊBDUJPOਸ୹۱ೠ׮
ജ҃਷BDUJPOਸੑ۱ਵ۽߉Ҋ׮਺TUBUFܳ୹۱ೠ׮

5IFHPBMPGSFJOGPSDFNFOUMFBSOJOH pθ (s1 , a1 , . . . .
. , ST , aT ) = p(s1 ) T ∏ t=1 πθ (at ∣ st )p(st+1 ∣ st , at ) pθ (τ) θ* = argmaxθ Eτ∼pθ (τ) [∑ t r(st , at )]

RETURN 4UBUFWBMVF REINFORCEMENT LEARNING 4UBUF"DUJPOWBMVF

3FUVSOPG&QJTPEF &QJTPEFزউ3FUVSOػ3FXBSE੄೤ 5PUBM3FXBSE

%JTDPVOUFE3FUVSO %JTDPVOUFEGBDUPSо੸ਊػ3FXBSE੄೤ 5PUBM3FXBSEXJUI%JTDPVOUFE

.%1ীࢲ੄Y(SJEXPSME Grid World Environment

.%1ীࢲ੄Y(SJEXPSME 4UBUFӒܻ٘੄ઝ಴ "DUJPO࢚ ೞ ઝ ਋ 3FXBSEೣ੿ ݾ಴  5SBOTJUJPO1SPCBCJMJUZ %JTDPVOUGBDUPS
3FXBSE 3FXBSE 4UBUF "DUJPO Grid World Environment

4UBUFWBMVFGVODUJPO 1PMJDZܳٮܲTUBUFWBMVFGVODUJPO 4UBUFWBMVF

"DUJPO7BMVFGVODUJPO 1PMJDZܳٮܲBDUJPOWBMVFGVODUJPO 4UBUFBDUJPOWBMVF

#FMMNBOFRVBUJPO "'VOEBNFOUBMQSPQFSUZPGWBMVFGVODUJPO

0QUJNBM1PMJDZܳ଺੗TUBUF 7BMVFܳ୭؀۽ 0QUJNBMTUBUFWBMVFGVODUJPO

0QUJNBM1PMJDZܳ଺੗TUBUFBDUJPO 7BMVFܳ୭؀۽ 0QUJNBMTUBUFBDUJPOWBMVFGVODUJPO

#FMMNBOFRVBUJPO 0QUJNBMJUZ #FMMNBOPQUJNBMJUZFRVBUJPOW

#FMMNBOFRVBUJPO 0QUJNBMJUZ #FMMNBOPQUJNBMJUZFRVBUJPOR

%ZOBNJDQSPHSBNNJOH

ઑѤ Transition Probability ژೠ઱য઎׮ State৬ action੉ discrete ೞݴ ੘׮.

Y(SJEXPSMEীࢲ%ZOBNJD1SPHSBNNJOH Grid World Environment

Y(SJEXPSME 4UBUFӒܻ٘੄ઝ಴ "DUJPO࢚ ೞ ઝ ਋ 3FXBSEೣ੿ ݾ಴  5SBOTJUJPO1SPCBCJMJUZ %JTDPVOUGBDUPS
3FXBSE 3FXBSE അ੤TUBUF "DUJPO Grid World Environment ׮਺TUBUF ׮਺TUBUF

6QEBUF3VMF  #FMMNBOFRVBUJPOਸࢎਊೞৈসؘ੉౟ೠ׮ 4UBUF

فઙܨ੄0QUJNBM7BMVFGVODUJPOT 4UBUF7BMVF #FMMNBOPQUJNBMJUZFRVBUJPOT୽઒

فઙܨ੄0QUJNBM7BMVFGVODUJPOT "DUJPO7BMVF #FMMNBOPQUJNBMJUZFRVBUJPOT୽઒

Dynamic Programming  فઙܨ੄୭੸7BMVFGVODUJPO State-action Value function

1PMJDZ*UFSBUJPO 7BMVF*UFSBUJPO Dynamic Programming

1PMJDZJUFSBUJPO 1.Policyܳٮۄ state-valueܳ҅࢑ೞ੗ Policy Evaluation ؊જ਷Policyܳ଺੗ Policy Improvement ୭੸੄1PMJDZܳ ଺ӝਤೠفо૑
җ੿

Policy iteration- Policy Evaluation 6QEBUF3VMFਸࢎਊೞৈ&WBMVBUJPOਸೠ׮ 7BMVFVQEBUF 1PMJDZ 5SBOTJUJPO 1SPCBCJMJUZ 3FXBSE
/FYU4UBUF  FTUJNBUFEWBMVF

ݽٚTUBUFܳ7 T ਵ۽ୡӝചदఅ׮ пTUBUFܳ6QEBUF3VMFਸࢎਊೞৈ7 T ܳসؘ੉౟ೠ׮ Policy iteration- Policy
Evaluation সؘ੉౟ೞݴ7 T ੄߸ച۝੉ݒ਋੘ਸٸসؘ੉౟ܳݥ୸׮ Policyܳٮۄ state-valueܳ҅࢑ೞ੗

Policy iteration- Improvement 1PMJDZܳٮۄ7BMVFGVODUJPOਸ҅࢑ೠ੉ਬח؊ա਷ 1PMJDZܳ଺ӝਤ೧ࢲ੉׮ (SFFEZ1PMJDZ

Policy iteration- Improvement (SFFEZ1PMJDZ੸ਊ

1PMJDZJUFSBUJPO 1PMJDZJUFSBUJPO਷0QUJNBMQPMJDZܳ଺ਸٸө૑ 1PMJDZ&WBMVBUJPOҗ1PMJDZ*NQSPWFNFOUܳ߈ࠂೠ׮

(SJE8PSME&OWJSPONFOU Y(SJEXPSME

Y(SJEXPSME 4UBUFӒܻ٘੄ઝ಴ "DUJPO࢚ ೞ ઝ ਋ 3FXBSEೠ஢਑૒ੌٸ݃׮ 5SBOTJUJPO1SPCBCJMJUZ %JTDPVOUGBDUPS (SJE8PSME&OWJSPONFOU1PMJDZJUFSBUJPO
3FXBSE (PBM "DUJPO (PBM 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE 3FXBSE

(SJE8PSME&OWJSPONFOU1PMJDZJUFSBUJPO Lੌٸ ୡӝച 7L (SFFE1PMJDZ

7L (SFFE1PMJDZ L
(SJE8PSME&OWJSPONFOU1PMJDZJUFSBUJPO

7L (SFFE1PMJDZ LJOG
(SJE8PSME&OWJSPONFOU1PMJDZJUFSBUJPO

1PMJDZJUFSBUJPO दো

1PMJDZ*UFSBUJPO 7BMVF*UFSBUJPO Dynamic Programming

7L (SFFE1PMJDZ Lࣻ۴ೞݶ
(SJE8PSME&OWJSPONFOU7BMVFJUFSBUJPO

7BMVF*UFSBUJPO 4LJQUIFQPMJDZBOEDPNQVUFWBMVFTEJSFDUMZ 4UBUF "DUJPO

7BMVFJUFSBUJPO दো

.PEFM੉হ׮ݶ पઁ۽҃೷ਸ೧ࠁݴജ҃җ࢚ഐ੘ਊਸ೧ঠೠ׮

.POUF$BSMPNFUIPE

.POUF$BSMPNFUIPEח%ZOBNJDQSPHSBNJOH୊ۢ ݽٚ੿ࠁܳঌҊद੘ೞחѪ੉ইצ पઁ۽҃೷ਸೞݴജ҃җ࢚ഐ੘ਊਸೠ׮ .POUF$BSMP

पઁ۽҃೷ਸೞݴߓ਋חߑߨ੉જ਷੼਷FOWJSPONFOU੄ ੿ࠁоহযبपઁ۽҃೷ਸೞݴPQUJNBMCFIBWJPSਸ  ੉ܖӝٸޙ੉׮ .POUF$BSMPNFUIPE

.POUF$BSMPNFUIPE .POUF$BSMPחFQJTPEFCZFQJTPEF۽সؘ੉౟ೠ׮ ীೖࣗ٘੄݃૑݄झప੉౟UFSNJOBMTUBUFө૑ оࢲসؘ੉౟ೠ׮ .POUF$BSMPח҃೷ਸೞݴSFUVSOػTBNQMFਸ੉ਊೞৈ TUBUFBDUJPOWBMVFܳಣӐೞৈসؘ੉౟ೠ׮

(PBM .POUF$BSMP(SJE8PSME ՘ө૑оࠄٍ6QEBUF 4UBSU

5FNQPSBMEJGGFSFODFMFBSOJOH

5FNQPSBM%JGGFSFODF-FBSOJOH .POUF$BSMPNFUIPE৬э੉  ݽ؛হ੉҃೷ਸాೞৈWBMVFܳஏ੿ೞݴ ઺р઺рWBMVFܳFTUJNBUFೞחѪ੉оמ

5FNQPSBM%JGGFSFODF-FBSOJOH അ੤TUBUFীࢲBDUJPOਸࢶఖೞݴ߉ਸ3FXBSEҗ׮਺4UBUFীEJTDPVOUGBDUPSо੸ਊػ TUBUFWBMVFܳFTUJNBUFೞݴVQEBUFೠ׮

5%੄੢੼ ജ҃੄ݽ؛ਸঌ૑ޅ೧بࢎਊоמ %ZOBNJDQSPHSBNNJOHীࢲ୊ۢ0OMJOF೟ण੉׮ ՘ө૑ӝ׮ܻ૑ঋইب ઺р઺рVQEBUFооמೞӝীFQJTPEFо ݆੉ӡѢա഑਷DPOUJOVFೠNPEFMীࢲࢎਊೞӝજ׮

5%੄ই੉٣য 5FNQPSBM%JGGSFOFDF-FBSOJOH 4BSTB 2MFBSOJOH 5FNQPSBM%JGGSFOFDF-FBSOJOH੉4BSTB৬2MFBSOJOH੄߄ఔই੉٣যоغ঻׮ 0OQPMJDZ 0GGQPMJDZ

4BSTB 2MFBSOJOH Temporal-Diffrenece Learning

4BSTB POQPMJDZߑߨਸࢎਊೞח4BSTB TUBUFWBMVFGVODUJPO؀नBDUJPOWBMVFGVODUJPOਸ೟ण

4BSTB ׮਺UJNFTUFQীࢲTUBUF৬BDUJPOܳل׮ࢎਊೞৈBDUJPOWBMVFܳFTUJNBUFೠ׮

4BSTBQTFVEPDPEF

4BSTBQTFVEPDPEF 0OQPMJDZ

4BSTBHSJEXPSME (PBM 4UBSU "U 4U

4BSTBHSJEXPSME
҃೷ೞ૑ঋ਷झప੉౟੄੿ࠁоহ׮

4BSTBHSJEXPSME
҃೷ਸৈ۞ߣ೧ࠁݴBDUJPOWBMVFܳসؘ੉౟ೠ׮ 1PMJDZח0OQPMJDZ

4BSTB दো https://github.com/wonseokjung/daejeon_rl_1219 Sarsa ಫ؊ -> sarsa_agent प೯

4BSTB 2MFBSOJOH Temporal-Diffrenece Learning

2MFBSOJOH 2MFBSOJOH੉ۄҊܻࠛחPGGQPMJDZ5%DPOUSPMੋ೧ъച೟ण੉ߊ੹ೞח҅ӝоغ঻׮ 8BULJOT FYQMPSBUJPOҗFYQMPJUBUJPOਸэ੉ೠ׮

2MFBSOJOHQTFVEPDPEF 0GGQPMJDZ

RMFBSOJOHHSJEXPSME (PBM 4UBSU "SHNBY 4U

2MFBSOJOHHSJEXPSME
҃೷ೞ૑ঋ਷झప੉౟੄੿ࠁоহ׮

2MFBSOJOHHSJEXPSME
҃೷ਸৈ۞ߣ೧ࠁݴBDUJPOWBMVFܳসؘ੉౟ೠ׮ 1PMJDZח0GGQPMJDZ

8IZJTUIJTBMHPSJUINPGGQPMJDZ BDUJPOਸࢶఖೞחѪ੉QPMJDZ৬࢚ҙহ੉BDUJPO઺BDUJPOWBMVFо  о੢௾Ѫਸࢶఖೞӝٸޙ੉׮

2MFBSOJOH दো 2MFBSOJOH R@MFBSOJOH@BHFOU

4VQFSNBSJPXJUI%FFQ3-

MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF
St Rt+1 St+1 SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ

SUPERMARIO WITH R.L https://github.com/wonseokjung/gym-super-mario-bros pip install gym-super-mario-bros  import gym_super_mario_bros  env
= gym_super_mario_bros.make(‘SuperMarioBros-v0') env.reset() env.render() INSTALL AND IMPORT ENVIRONMENT

WORLDS & LEVELS ( WORLD 1~4) SUPERMARIO WITH R.L 8PSME
8PSME 8PSME 8PSME env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')

WORLDS & LEVELS ( WORLD 5~8) SUPERMARIO WITH R.L 8PSME
8PSME 8PSME 8PSME env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')

ALL WORLDS AND LEVELS SUPERMARIO WITH R.L env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')

WORLDS & LEVELS SUPERMARIO WITH R.L 7FSTJPO env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
7FSTJPO 7FSTJPO 7FSTJPO

GOAL SUPERMARIO WITH R.L

REWARD AND PENALTY SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ ӥߊীоөਕ૑ݶ
ݾ಴ীب଱ೞݶ ݾ಴׳ࢿೞ૑ޅೞݶ दр੉૑զٸ݃׮ ӥߊীࢲݣয૑ݶ

STATE, ACTION SUPERMARIO WITH R.L env.observation_space.shape (240, 256, 3) #
[ height, weight, channel ] env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ]     from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros  env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)

OBSERVATION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [
[‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ]     from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros  env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ]

ACTION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [
[‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ]     from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros  env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ]

ACTION AFTER WRAPPER SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT =
[ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ]     import gym_super_mario_bros  env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv

EXPLOITATION AND EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info
= env.step(action) else :   action = np.argmax(output) Exploitation Exploration def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) ?

EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action)
else :   action = np.argmax(output) Exploitation Exploration if np.random.rand() < epsilon : action=np.random.randint(output) def epsilon_greedy(q_value,step): ?

EXPLOITATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action)
else :   action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) Exploitation Exploration ?

ENV.STEP( ) SUPERMARIO WITH R.L next_state, reward, done, info =
env.step(action) else :   action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output)

EXPLORATION RATE AND REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory
= deque([],maxlen=1000000) memory.append(state,action,reward,next_state) (St , At , Rt+1 , St+1 ) next_state, reward, done, info = env.step(action) eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000

REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1

REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action)

REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min
= 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)

REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min
= 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory.append(state,action,reward,next_state) memory = deque([],maxlen=1000000)

MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf loss
= tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 )

MINIMIZE LOSS SUPERMARIO WITH R.L (Rt+1 + γt+1 maxa′ qθ
(St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) import tensorflow as tf

MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf Optimizer
=tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) )

MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf training_op
= optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate)

MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf (Rt+1
+ γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss)

APPROXIMATE ACTION-VALUE SUPERMARIO WITH R.L

DOUBLE DQN SUPERMARIO WITH R.L JOQVU "DUJPO WBMVF &OW 2/FUXPSL
s’ s 3FQMBZNFNPSZ 2 T B a r (St , At , Rt+1 , St+1 )

1000EPISODE, 3000EPISODE, TRAINING SUPERMARIO WITH R.L FQJTPEF FQJTPEF

5000 EPISODE SUPERMARIO WITH R.L FQJTPEF %BZT

'BTUFSMFBSOJOH

TERMINOLOGY 3FJOGPSDFNFOU-FBOJOH    5SJBMBOEFSSPSਸೞݴ3FXBSEܳ୭؀ചೠ׮  %FFQ3-    /FVSBMOFUXPSLܳࢎਊೞৈ3-BMHPSJUINਸSFQSFTFOUೠѪ  .FUB-FBSOJOH    t-FBSOJOHIPXUP-FBSOuযځೠ-FBSOJOHীҙৈೞח5BTLܳ.BTUFS
REINFORCEMENT LEARNING

TERMINOLOGY REINFORCEMENT LEARNING 3FJOGPSDFNFOU-FBOJOH    5SJBMBOEFSSPSਸೞݴ3FXBSEܳ୭؀ചೠ׮  %FFQ3-    /FVSBMOFUXPSLܳࢎਊೞৈ3-BMHPSJUINਸSFQSFTFOUೠѪ  .FUB-FBSOJOH 
  t-FBSOJOHIPXUP-FBSOuযځೠ-FBSOJOHীҙৈೞח5BTLܳ.BTUFS

POLICY GRADIENTS - PPO REINFORCEMENT LEARNING

ALPHAGO, DOTA REINFORCEMENT LEARNING

ROBOTIC MANIPULATION REINFORCEMENT LEARNING

RL REQUIRES A LOT OF TRAINING TIME REINFORCEMENT LEARNING

8IBUIBTQSPWFODIBMMFOHJOHTPGBS 1. Humans can learn incredibly quickly   -
Deep RL methods are usually slow 2. Humans can reuse past knowledge   - Transfer learning in deep RL is an open problem 3. Not clear what the reward function should be 4. Not clear what the role of prediction should be

PRIOR KNOWLEDGE REINFORCEMENT LEARNING

HOW WE CAN ALLOW OUT A.I SYSTEM MAKE TO USE
PRIOR KNOWLEDGE? REINFORCEMENT LEARNING https://ubisafe.org/explore/demeanure-clipart-prior-knowledge/

META REINFORCEMENT LEARNING REINFORCEMENT LEARNING https://ubisafe.org/explore/demeanure-clipart-prior-knowledge/ Meta Reinforcement Learning

META REINFORCEMENT LEARNING REINFORCEMENT LEARNING https://ubisafe.org/explore/demeanure-clipart-prior-knowledge/ 4JOH5BTL

REINFORCEMENT LEARNING LEARNING DEXTEROUS IN HAND MANIPULATION

REINFORCEMENT LEARNING META-RL IN ROBOT

REINFORCEMENT LEARNING META RL : CHANGES TO PROBLEM FORMULATION 'JOJUFTFUPGUSBJOJOHUBTLTBOEBUFTUTFUPGUBTLT

REINFORCEMENT LEARNING

REINFORCEMENT LEARNING ALGORITHMS

REINFORCEMENT LEARNING PPO(JOINT) + FINE TUNING

REINFORCEMENT LEARNING RETRO CONTEST

REINFORCEMENT LEARNING MODULABS CTRL

1SPKFDUT8IBU*sWFEPOF

PROJECTS - ৈ۞о૑ജ҃ীࢲӒജ҃ীݏחъച೟णঌҊ્ܻਸ੸ਊ೧ࠁও׮ - ੉җ੿ীࢲৈ۞о૑੉गоߊࢤ೮׮ 0QFO"*HZN  %2/ 4VQFSNBSJP  %%2/ UVOFE
4POJD 3BJOCPX%2/ UVOFE 0QFO4JN %%1(

QUESTIONS ૕ޙ ઱য૓ജ҃੉ইצղоҾӘೠޙઁܳಽӝਤ೧ ജ҃ਸ݅٘חѪ੉оמೡө ೟णदрਸ઴ੌࣻ੓חߑߨ੉੓ਸө *TTVFT  ೟णदр੉ցޖցޖয়ېѦܽ׮ * ӝળ
0QFO"*(:.୭ࣗ࠙_ੌ઱ੌ੉࢚ 4VQFS.BSJP-FWFMੌࣗਃ 4POJD0QFO"*ઁҕࢲߡࢎਊदр 1SPTUIFUJDT׳੉࢚৘࢚ ъച೟ण਷ജ҃੉೙ࣻ׮ ઁҕೞחജ҃݅ࢎਊೞחѪ੉оמೞ׮

ENVIRONMENTS 0QFO"* %FFQNJOE-BC 4UBSDSBGU 4VQFSNBSJP 4POJD .JOFDSBGU

ജ҃ਸٜ݅ࣻ੓ਸө

CREATING ENVIRONMENTS

UNITY ML-AGENTS 6OJUZܳࢎਊೞৈѐੋ੉ъച೟णജ҃ਸٜ݅ࣻ੓׮  ژೠ.BDIJOF-FBSOJOH"HFOUTӝמਵ۽೟णਸࠁ׮ബҗ੸੉ѱೞחѪ੉оמೞ׮

CREATING ENVIRONMENTS   ѱ੐ীೠ੿ػѪ੉ইצ࢑সীࢲഝਊೠࣻ੓ח3FBM4JNVMBUPSبઁ੘ೞחѪ੉оמೞ ׮

UNITY ML AGENT - ജ҃ਸٜ݅ٸBDUJPOਸೞח"HFOUژೠٜ݅ࣻ੓׮ - झझ۽BDUJPOਸೞח"HFOUחп#SBJOਸо૑Ҋ੓ਵݴ ੉#SBJO਷ъച೟णਸాೞৈ ೟णदఃחѪ੉оמೞ׮ #SBJO
#SBJO "HFOU "HFOU

6OJUZܳࢎਊೞৈઁ੘ೠജ҃ীࢲ૒੽ъച೟णঌҊ્ܻਸ੸ਊ &OWJSPONFOU 3FJOGPSDFNFOU-FBSOJOH

"HFOUоৈۤ੉ۄݶ

MULTI-AGENTS? *OUFMMJHFOUIVNBOBHFOUT ࢎۈ ਷ࢎഥীࢲ׮ܲBHFOUT৬੿ࠁܳҕਬೠ׮ ੿ࠁܳҕਬೞݴDPPQFSBUJPO ഈ۱ ਸೞѢա*OEFQFOEFOU ة݀੸ਵ۽ ೞѱݾ಴ܳ׳ ࢿೠ׮

MULTI-AGENT ENVIRONMENT

MULTI-AGENTS - ୨ৈࢽ"HFOUTо੓׮ - ౵ۆ࢝߄աաܳദٙ೮ਸٸQFOBMUZܳ߉ਵݴ ֢ۆ࢝߄աաܳദٙद3FXBSEܳ߉ח ׮ - п"HFOUחة݀ػ#SBJOਸо૑Ҋ੓ਵݴة݀੸ਵ۽BDUJPOਸࢶఖೠ׮ &OWJSPONFOU
3FJOGPSDFNFOU-FBSOJOH

TRAINING USING IMITATION LEARNING ৈࢽ"HFOUTח*OEFQFOEFOUೞѱ֢ۆ࢝#BOBOBܳ଺ӝਤ೧ৈ۞о૑BDUJPOਸࢶఖೞ ݴߓ਍׮

ৈۤ੄"HFOUо҃੬ਸ೧ঠೡٸח

ADVERSARIAL LEARNING? - ҕాػݾ಴ܳ׳ࢿೞӝਤ೧੿ࠁܳҕਬೞݴ$PPQFSBUJPOೡٸب੓૑݅ ࠂय ୷ҳ ఌ ҳ పפझ١җэ੉थಁоഛपೠ҃਋ب੓׮

ADVERSARIAL SELF-PLAY

ADVERSARIAL LEARNING - ҕాػݾ಴ܳ׳ࢿೞӝਤ೧੿ࠁܳҕਬೞݴ$PPQFSBUJPOೡٸب੓૑݅ ࠂय ୷ҳ ఌ ҳ పפझ١җэ੉थಁоഛपೠ҃਋ب੓׮ -
౱׼Ҏਸ֍חTUSJLFS৬Ҏਸ݄ח(PBMLFFQFS۽ҳࢿغয੓׮ 4USJLFS (PBMLFFQFS 4USJLFS (PBMLFFQFS 0CKFDU &OWJSPONFOU

ADVERSARIAL LEARNING - 4USJLFS৬(PBMLFFQFSח࢚؀ಞীҕਸ֍Ҋ݄ӝਤ೧$PPQFSBUJPOೞݴ೯زೠ׮ 4USJLFS (PBMLFFQFS (PBM,FFQFS 74 4USJLFS 74
$PPQ $PPQ

೟णदрਸ઴ੌࣻ੓ਸө

੹ޙоܳࠁҊٮۄೞݶ؊ࡅܰѱߓ਎ࣻ੓૑ঋਸө

IMITATION LEARNING? - ࢎۈҗزޛ਷যځೠݾ಴ޛਸࠁҊӒٜ੉ೞחCFIBWJPSਸࠁҊߓ਍׮ - ఋѶ੉ೞח೯زਸݽߑೞݴߓ਋חߑߨਸ*NJUBUJPO-FBSOJOH੉ۄҊೠ׮

IMITATION LEARNING

TRAINING USING IMITATION LEARNING (SBWJUZ "HFOU "HFOU (SBWJUZ #BMM #BMM
*OJUJBMJ[BUJPO - ҕ਷઺۱ী੄೧ڄয૑ݴ п"HFOUחҕਸ߉ই߈؀"HFOU৔৉ਵ۽ֈѹঠೠ׮ - ߈؀ীਤ஖ೠ"HFOUחҕਸࠁҊ׮दֈӟ׮ &OWJSPONFOU 3FJOGPSDFNFOU-FBSOJOH

TRAINING USING IMITATION LEARNING "HFOU "HFOU 4USBU5SBJOJOH "DUJPO "DUJPO "DUJPO
"DUJPO "DUJPO "DUJPO "HFOUחৈ۞о૑BDUJPOਸࢶఖೞݴ؊݆਷3FXBSEܳ߉ਸࣻ੓חBDUJPOਸఐ࢝ ੉۠ߑध਷೟णغחؘदр੉݆੉ࣗਃػ׮

TRAINING WITHOUT IMITATION LEARNING - ۽ࠈ੉ѥҊ ׳۰ঠೞחয۰਍ജ҃ীࢲח೟णदр੉ݒ਋ӡݴ੗োझۣ૑ঋ਷ޙઁ оߊࢤೠ׮

TRAINING USING IMITATION LEARNING *NJUBUJPO-FBSOJOH 5FBDIFS 4UVEFOU - ؊ࡅܰҊബҗ੸ਵ۽ߓ਎ࣻ੓ѱ੹ޙо੄೯زਸࠁҊߓ਋ח*NJUBUJPO-FBSOJOHਸ ࢎਊೠ׮
- 4UVEFOUח5FBDIFSਸࠁҊߓ਑ - 5FBDIFS 1MBZFS ח੸੺ೠBDUJPOਸࢶఖೞݴTUVEFOUоࡈܻߓ਎ࣻ੓ѱب৬ળ׮

ೠߣীߓ਋ӝցޖয۵׮ݶ

CURRICULUM LEARNING - ࢎۈ਷కযաࢲ߄۽׳ܻחѪ੉оמೞ૑ঋ׮ - ٍ૘Ҋ ӝҊ ࢲҊ ѥҊ ׳ܻחѪ୊ۢױ҅߹۽೟णೠ׮
- ੉۠೟णߑߨਸъച೟णীࢲ$VSSJDVMVN-FBSOJOH੉ۄҊೠ׮

CURRICULUM LEARNING

CURRICULUM LEARNING "HFOUחрױೠUBTLࠗఠߓ਋ӝद੘ೞৈױ҅߹۽೟णਸೠ׮  ೠߣীয۰਍UBTLܳ೟णೞӝ൨ٜӝٸޙী੼੼ױ҅ܳ֫ৈ೟णਸदఅ׮

ENVIRONMENTS "HFOU (PBM 8BMM - "HFOUח(PBMীب଱ೞӝਤ೧BDUJPOਸࢶఖೠ׮ - 8BMM਷׮ࢽо૑֫੉۽߄Ո׮  "DUJPO "DUJPO
"DUJPO &OWJSPONFOU 3FJOGPSDFNFOU-FBSOJOH

TRAINING USING CURRICULUM LEARNING - ౵ۆ࢝BHFOUחTNBMMXBMMҗMBSHFXBMMਸೖೞѢաڪযֈযݾ಴۽оӝ ਤ೧೟णೠ׮ 

최신 알고리즘

5PUIF3BJOCPX ֙ਘ%FFQNJOEীࢲ3BJOCPX%2/ਸߊ಴

IUUQTHJUIVCDPNXPOTFPLKVOHXPOTFPLKVOHHJUIVCJPCMPCNBTUFS@QPTUT3-5PUIFSCNE ଵҊ IUUQTXPOTFPLKVOHHJUIVCJPSFJOGPSDFNFOUMFBSOJOHVQEBUF3-5PUIFSC "$

%FFQ2OFUXPSL input Env %PVCMF%2/  %*45 s’ s Replay memory Q(s,a)
a r /PJTZ (Rt+1 + γt+1 qθ (St+1 , argmaxa′ q(St) − qθ (St , At ))2 1SJPSJUJ[FE SFQMBZ ೟ण੉؊೙ਃೠ USBOTJUJPOTਸ TBNQMF Multi-step learning

5PUIF3BJOCPX %2/҅ৌ੄ঌҊ્ܻѐ৬"$ܳઑ೤ೞৈ݅ٚъച೟णঌҊ્ܻ੉׮ 4VQFS.BSJPী੸ਊೠ%2/ঌҊ્ܻ  ؀࠺ষ୒խ࢚थ

"UBSJജ҃ীࢲ੄ࢿמ࠺Ү

ѱ੐੉ইצ؊݆਷BDUJPOਸо૓BHFOUب  ъച೟णਵ۽೟ण੉оמೡө

A.I Prosthetics  High dimensional state  Continuous actions

Forth step ( Continuos action ) /*14"*GPS1SPTUIFUJDT$IBMMFOHF

%JTDSFUF"DUJPO $POUJOVPT"DUJPO "DUJPOJO3FBMXPSME %2/TPMWFE)JHIEJNFOTJPOBMTUBUF CVUOPUDPOUJOVFTBDUJPO https://twitter.com/iamruj

5XPNFUIPETPGDIPPTJOHBDUJPO BDUJPOWBMVF -FBSOJOHUIFBDUJPOWBMVF &TUJNBUFBDUJPOWBMVFਸ߄ఔਵ۽BDUJPOਸࢶఖೠ׮ 1PMJDJFTXPVMEOPUFWFOFYJTUXJUIPVUUIFBDUJPOWBMVFFTUJNBUFT 1BSBNFUFSJ[FEQPMJDZ
TFMFDUBDUJPOTXJUIPVUDPOTVMUJOHWBMVFGVODUJPO 7BMVFGVODUJPOTUJMMCFVTFEUPMFBSOQPMJDZQBSBNFUFS 7BMVFGVODUJPO੉BDUJPOਸࢶఖೞחӝળਵ۽ࢎਊغ૑ঋח׮ J(θ) 1FSGPSNBODFNFBTVSF qπ(s, a) = Eπ [Gt ∣ St = s, At = a]

%JTDSFUF"DUJPO $POUJOVPT"DUJPO 4FMFDUBDUJPOVTJOH1(.FUIPE https://www.cs.ubc.ca/~gberseth/blog/demystifying-the-many-deep-reinforcement-learning-algorithms.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf

&NFSHFODFPG-PDPNPUJPO#FIBWJPVSTJO3JDI&OWJSPONFOUT https://www.youtube.com/watch?v=hx_bgoTF7bs&t=98s

$PNNVOJUZীৢۄৡӖ઺

Latest trends

%FFQ.JNJD 3FGFSFODF.PUJPOਸࠁ Ҋٮۄೞחী੉੹౟ ؊੉࢚җ਺਷ೞ૑ঋ ب۾

https://www.youtube.com/watch?v=XCLSkFKTWyg 5IJTWJSUVBMTUVOUNBODPVMEJNQSPWFWJEFPHBNFQIZTJDT

"*4BGFUZ

ղо ೧ࠁҊ र਷ ೐۽ં౟

ъച೟णਵ۽&OFSHZTBWJOH https://www.youtube.com/watch?v=RFV8IkY52iY&t=79s

ҕࠗೞӝજ਷੗ܐ https://blog.openai.com/spinning-up-in-deep-rl/

֙ীೡ೐۽ં౟    पղ੗ਯ઱೯೐۽ં౟    ъച೟णਵ۽۽ࠈ౹೟ण೐۽ં౟

  .PEVMBCT$53-3FTFBSDIFS $JUZ6OJWFSTJUZPG/FX:PSL#BSVDI $PMMFHF %BUB4DJFODF.BKPS $POOFYJPO"*'PVOEFS %FFQ-FBSOJOH$PMMFHF3FJOGPSDFNFOU -FBSOJOH3FTFBSDIFS CTRL
MEMBERS 8POTFPL+VOH ,ZVOHIXBO,JN   .PEVMBCT$53-3FTFBSDIFS )BOTVOH6OJWFSTJUZ  &MFDUSPOJDJOGPSNBUJPO&OHJOFFSJOH.BKPS *OUFSFTUFE"* 3FJOGPSFNFOUMFBSOJOH (BNF )ZP+FPOH+FPO   .PEVMBCT$53-3FTFBSDIFS #JOHIBNUPO6OJWFSTJUZ.4PG.FDIBOJDBM &OHJOFFSJOH.BKPS .BUIJOTUSVDUPSBU$PMMFHF1SFQ*OTUJUVUF *OUFSFTU"QQMJDBUJPOPG6OJUZ.-"HFOUT JO3- ("/

CTRL MEMBERS 4FVOH+BF-FF   .PEVMBCT$53-3FTFBSDIFS 1SJODFUPO6OJWFSTJUZ$MBTTPG .BUIFNBUJDT.BKPS 4DSBUDIXPSL--$$PGPVOEFS 3FTFBSDI
&YQFSJNFOUTXJUIUIF.BSLPGG4VSGBDF@ XJUI.BUUIFXEF$PVSDZ*SFMBOEVOEFS )BOZBOH6OJWFSTJUZ  .4#JPJOGPSNBUJDT #*(-BC   6OJUZ%FWFMPQFS .BSWSVT   *OUFSFTUFE73"3   3FJOGPSDFNFOUMFBSOJOH "*JOUFSBDUJPO 1I% 2VBMJUZ4ZTUFNT-BC1045&$)   4FBSDI3FDPNNFOEFS4ZTUFN&OHJOFFS  *OUFSFTUFE3-"QQMJDBUJPOJO&WFSZXIFSF .PEVMBCT$53-3FTFBSDIFS .PEVMBCT$53-3FTFBSDIFS +XBXPO4FP 7JDT,XPO

CTRL MEMBERS :VOLZV$IPJ   .PEVMBCT$53-3FTFBSDIFS 73/&3% 4VIZVL1BSL   .PEVMBCT$53-3FTFBSDIFS .BTUFS
/BUVSBM-BOHVBHF1SPDFTTJOH ,PSFB 6OJWFSTJUZ %BUB*OHFTUJPO5FBN-FBEFSBU/$40'5 *OUFSFTUFE"QQMZ3FJOGPSDFNFOU-FBSOJOH 5FDIOJRVFTUP%BUB1SPDFTTJOH"SFB ,VSU   .PEVMBCT$53-3FTFBSDIFS 3-/&3%

  .PEVMBCT,"*33FTFBSDIFS $JUZ6OJWFSTJUZPG/FX:PSL#BSVDI $PMMFHF %BUB4DJFODF.BKPS $POOFYJPO"*'PVOEFS %FFQ-FBSOJOH$PMMFHF3FJOGPSDFNFOU -FBSOJOH3FTFBSDIFS 3FJOGPSDFNFOU-FBSOJOH
0CKFDU %FUFDUJPO $IBUCPU KAIR MEMBERS 8POTFPL+VOH $IFPMIVJ.JO   .PEVMBCT,"*33FTFBSDIFS #4JO.FDIBOJDBM&OH,PSFB6OJW .4DDBOEJEBUFJO.FDIBOJDBM&OH,PSFB 6OJW 3FTFBSDIGJFME%FFQSFJOGPSDFNFOUMFBSOJOH GPSSPCPUJDT 3PCPUDPOUSPMJOUFSGBDF *OUFSFTUT%FFQ3-BOEPQUJNBMDPOUSPM 3PCPUJDT $IJOFTF 8IJ,XPO   .PEVMBCT,"*33FTFBSDIFS JOUFSFTUFE3- NBOJQVMBUPS 4PHBOH6OJW$IFNJDBMBOE#JPFOHJOFFSJOH .FEJQJYFM"*SFTFBSDIFSMFBSOJOHGPS SPCPUJDT 3PCPUDPOUSPMJOUFSGBDF

  .PEVMBCT,"*33FTFBSDIFS ,ZVOHIFF6OJWFSTJUZ.FDIBOJDBMBOE 4PGUXBSFFOHJOFFSJOH EVBMNBKPS *OUFSFTUSPCPUJDTBOETJNVMBUJPO 3- NM BHFOU
  .PEVMBCT,"*33FTFBSDIFS 4FPVM/BUJPOBM6OJWFSTJUZ .FDIBOJDBM "FSPTQBDF&OHJOFFSJOH.BKPS 4QBDF/FSE *OUFSFTU3PCPUJDT /BWJHBUJPO$POUSPM #BUUMFHSPVOE .BDBSPO   .PEVMBCT,"*33FTFBSDIFS %BFKFPO6OJWFSTJUZ &MFDUSPOJD*OGPSNBUJPO$PNNVOJDBUJPO &OHJOFFSJOH *OUFSFTUFE3FJOGPDFNFOU-FBSOJOH 4JN3FBM KAIR MEMBERS 4VCJO:BOH 4FPZFPO:BOH +VOUBF,JN

  .PEVMBCT,"*33FTFBSDIFS 4VOHLZVOLXBO6OJW.FDIBOJDBM FOHJOFFSJOH#4 4FPVM/BUJPOBM6OJW.4TUVEFOU *OUFSFTUFE)VNBOPJE 5PSRVFDPOUSPM 3-  
.PEVMBCT,"*33FTFBSDIFS ,PSFB6OJWFSTJUZ $POUSPM3PCPUJDT 4ZTUFN *OUFSFTU2VBOUJ[FE/FVSBM/FUXPSLTBOE "4*$*NQMFNFOUBUJPO 4UBUFFTUJNBUJPO 3-   .PEVMBCT,"*33FTFBSDIFS #4JO$PNQVUFS4DJFODFBOE&OHJOFFSJOH 3FTFBSDI&OHJOFFSBU.FEJQJYFM *OUFSFTUFEJO$POWFY0QUJNJ[BUJPO $PNQVUFS7JTJPO BOE3FJOGPSDFNFOU -FBSOJOH KAIR MEMBERS %POHIZFPO,JN +FPOHIPPO,JN +JOXPP1BSL

  .PEVMBCT,"*33FTFBSDIFS #4JO.BUIFNBUJDTBOE$PNQVUFS 4DJFODFJO,PSFB6OJW .4$PVSTFJO&MFDUSJDBM&OHJO4FPVM /BUJPOBM6OJW 3FTFBSDIGJFME4UPDIBTUJD$POUSPM1SPDFTT KAIR MEMBERS 4JIZVO$IPJ
4FVOH+BF3ZBO-FF   .PEVMBCT,"*33FTFBSDIFS 1SJODFUPO6OJW$MBTTPG .BUIFNBUJDT FOEUPFOE"*3FTFBSDIFS %FFQ-FBSOJOH;FSP5P"MM4FBTPO1Z5PSDI $POUFOU$POUSJCVUPS *OUFSFTUT%BUBFGGJDJFOU3- $VSJPTJUZESJWFO -FBSOJOH .FUB-FBSOJOH

पणਸ೧ࠇद׮

(JUIVC IUUQTHJUIVCDPNXPOTFPLKVOH 'BDFCPPL IUUQTXXXGBDFCPPLDPNXTKVOH #MPH IUUQTXPOTFPLKVOHHJUIVCJP хࢎ೤פ׮

reinforcement_learning_.pdf

reinforcement_learning_.pdf

More Decks by Wonseok Jung

Featured

Transcript