Supermario-Tutorial-1

강화학습으로 슈퍼마리오 에이전트 학습시키기 시리즈 1편

Transcript

1. "*
   4VQFS.BSJP
   5VUPSJBM 8POTFPL
   +VOH 3FJOGPSDFNFOU
   -FBSOJOH
2. ੿ਗࢳ
 8POTFPL
   +VOH
   $JUZ
   6OJWFSTJUZ
   PG
   /FX
   :PSL
   
   #BSVDI
   $PMMFHF
   %BUB
   4DJFODF
   .BKPS
   $POOFYJPO"*
   'PVOEFS
   %FFQ-FBSOJOH$PMMFHF
   3FJOGPSDFNFOU
   -FBSOJOH
   3FTFBSDIFS
   .PEVMBCT
   $53-
   -FBEFS
   3FJOGPSDFNFOU
   -FBSOJOH
   0CKFDU
   %FUFDUJPO
   $IBUCPU
   (JUIVC
   IUUQTHJUIVCDPNXPOTFPLKVOH
   'BDFCPPL
   IUUQTXXXGBDFCPPLDPNXTKVOH
   #MPH
   IUUQTXPOTFPLKVOHHJUIVCJP
   
3. SUPERMARIO TUTORIAL SERIES 1.Environment and DQN 2.Deep Reinforcement Learning with Double Q-learning 3.Dueling Network Architectures for Deep Reinforcement Learning 4.Prioritized Experience Replay 5.Noisy Networks for ExplorationPrioritized DQN 6. A Distributional Perspective on Reinforcement Learning 7. Rainbow: Combining Improvements in Deep Reinforcement Learning REINFORCEMENT LEARNING
4. SUPERMARIO TUTORIAL SERIES 1. Environment and DQN 2.Deep Reinforcement Learning with Double Q-learning 3.Dueling Network Architectures for Deep Reinforcement Learning 4.Prioritized Experience Replay 5.Noisy Networks for ExplorationPrioritized DQN 6. A Distributional Perspective on Reinforcement Learning 7. Rainbow: Combining Improvements in Deep Reinforcement Learning REINFORCEMENT LEARNING
5. ݾର 1. Markov Decision Process 2. How to install Supermario envrionment 3.Supermario Envrionment 4. Training 5. DQN 6. Result REINFORCEMENT LEARNING
6. 1. MARKOV DECISION PROCESS
7. MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St Rt+1 St+1 REINFORCEMENT LEARNING
8. MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St Rt+1 St+1 SUPERMARIO WITH R.L 3FXBSE
   
   
   1FOBMUZ
   
   
9. MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF St Rt+1 St+1 SUPERMARIO WITH R.L 3FXBSE
   
   
   1FOBMUZ
   
   
10. SUPERMARIO WITH R.L https://github.com/wonseokjung/gym-super-mario-bros pip install gym-super-mario-bros
 import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0') env.reset() env.render() INSTALL AND IMPORT ENVIRONMENT
11. WORLDS & LEVELS ( WORLD 1~4) SUPERMARIO WITH R.L 8PSME
     8PSME
     8PSME
     8PSME
     env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
12. WORLDS & LEVELS ( WORLD 5~8) SUPERMARIO WITH R.L 8PSME
     8PSME
     8PSME
     8PSME
     env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')
13. ALL WORLDS AND LEVELS SUPERMARIO WITH R.L env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')        
14. ALL WORLDS AND LEVELS SUPERMARIO WITH R.L env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>')                
15. WORLDS & LEVELS SUPERMARIO WITH R.L 7FSTJPO
     env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>') 7FSTJPO
     7FSTJPO
     7FSTJPO
    
16. GOAL SUPERMARIO WITH R.L
17. REWARD AND PENALTY SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ ӥߊী
    оөਕ૑ݶ
    
    ݾ಴ী
    ب଱ೞݶ
    ݾ಴׳ࢿೞ૑
    ޅೞݶ
    
    दр੉
    ૑զٸ݃׮
    
    ӥߊীࢲ
    ݣয૑ݶ
    
18. STATE, ACTION SUPERMARIO WITH R.L env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] 
 
 from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT)
19. OBSERVATION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] 
 
 from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ]
20. OBSERVATION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] 
 
 from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ]
21. ACTION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] 
 
 from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ]
22. ACTION AFTER WRAPPER SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] 
 
 import gym_super_mario_bros
 env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv
23. EXPLOITATION AND EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action) else : 
 action = np.argmax(output) Exploitation Exploration def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) ?
24. EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action) else : 
 action = np.argmax(output) Exploitation Exploration if np.random.rand() < epsilon : action=np.random.randint(output) def epsilon_greedy(q_value,step): ?
25. EXPLOITATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action) else : 
 action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) Exploitation Exploration ?
26. ENV.STEP( ) SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action) else : 
 action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output)
27. EXPLORATION RATE AND REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) (St , At , Rt+1 , St+1 ) next_state, reward, done, info = env.step(action) eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000
28. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1
29. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1
30. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1
31. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action)
32. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
33. REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory.append(state,action,reward,next_state) memory = deque([],maxlen=1000000)
34. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 )
35. MINIMIZE LOSS SUPERMARIO WITH R.L (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) import tensorflow as tf
36. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) )
37. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate)
38. MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss)
39. APPROXIMATE ACTION-VALUE SUPERMARIO WITH R.L
40. DOUBLE DQN SUPERMARIO WITH R.L JOQVU "DUJPO
    WBMVF &OW 2/FUXPSL s’ s 3FQMBZ
    NFNPSZ 2 T B a r (St , At , Rt+1 , St+1 )
41. 1000EPISODE, 3000EPISODE, TRAINING SUPERMARIO WITH R.L 
    FQJTPEF 
    FQJTPEF
42. 5000 EPISODE SUPERMARIO WITH R.L 
    FQJTPEF 
    %BZT
43. REINFORCEMENT LEARNING
44. How about making your own A.I SuperMario?
45. (JUIVC
    IUUQTHJUIVCDPNXPOTFPLKVOH
    'BDFCPPL
    IUUQTXXXGBDFCPPLDPNXTKVOH
    #MPH
    IUUQTXPOTFPLKVOHHJUIVCJP хࢎ೤פ׮
    
 5IBOL
    ZPV
    
46. 2VFTUJPO
    
47. *ଵҊ ਊয ӝഐ Time step Action Transition Function Reward Set of states Set of actions Start state Discount factor t a P(s′, r ∣ s, a) r A S S0 γ Set of reward
 
 Policy Reward State R π r REINFORCEMENT LEARNING s
48. REFERENCES 1. SuperMario environment 
 https://github.com/Kautenja/gym-super-mario-bros