Share
Embed
Transcript
48 slides
-
ਗࢳ 8POTFPL+VOH $JUZ6OJWFSTJUZPG/FX:PSL#BSVDI$PMMFHF %BUB4DJFODF.BKPS $POOFYJPO"*'PVOEFS %FFQ-FBSOJOH$PMMFHF3FJOGPSDFNFOU-FBSOJOH3FTFBSDIFS .PEVMBCT$53--FBEFS 3FJOGPSDFNFOU-FBSOJOH 0CKFDU%FUFDUJPO
$IBUCPU (JUIVC IUUQTHJUIVCDPNXPOTFPLKVOH 'BDFCPPL IUUQTXXXGBDFCPPLDPNXTKVOH #MPH IUUQTXPOTFPLKVOHHJUIVCJP -
SUPERMARIO TUTORIAL SERIES 1.Environment and DQN 2.Deep Reinforcement Learning with
Double Q-learning 3.Dueling Network Architectures for Deep Reinforcement Learning 4.Prioritized Experience Replay 5.Noisy Networks for ExplorationPrioritized DQN 6. A Distributional Perspective on Reinforcement Learning 7. Rainbow: Combining Improvements in Deep Reinforcement Learning REINFORCEMENT LEARNING -
SUPERMARIO TUTORIAL SERIES 1. Environment and DQN 2.Deep Reinforcement Learning
with Double Q-learning 3.Dueling Network Architectures for Deep Reinforcement Learning 4.Prioritized Experience Replay 5.Noisy Networks for ExplorationPrioritized DQN 6. A Distributional Perspective on Reinforcement Learning 7. Rainbow: Combining Improvements in Deep Reinforcement Learning REINFORCEMENT LEARNING -
ݾର 1. Markov Decision Process 2. How to install Supermario
envrionment 3.Supermario Envrionment 4. Training 5. DQN 6. Result REINFORCEMENT LEARNING -
MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF
St Rt+1 St+1 REINFORCEMENT LEARNING -
MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF
St Rt+1 St+1 SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ -
MARKOV DECISION PROCESS "DUJPO "HFOU &OWJSPONFOU 3FXBSE At Rt 4UBUF
St Rt+1 St+1 SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ -
SUPERMARIO WITH R.L https://github.com/wonseokjung/gym-super-mario-bros pip install gym-super-mario-bros import gym_super_mario_bros env
= gym_super_mario_bros.make(‘SuperMarioBros-v0') env.reset() env.render() INSTALL AND IMPORT ENVIRONMENT -
WORLDS & LEVELS ( WORLD 1~4) SUPERMARIO WITH R.L 8PSME
8PSME 8PSME 8PSME env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>') -
WORLDS & LEVELS ( WORLD 5~8) SUPERMARIO WITH R.L 8PSME
8PSME 8PSME 8PSME env = gym_super_mario_bros.make('SuperMarioBros-<world>-<level>-v<version>') -
REWARD AND PENALTY SUPERMARIO WITH R.L 3FXBSE 1FOBMUZ ӥߊীоөਕݶ
ݾীبೞݶ ݾ׳ࢿೞޅೞݶ दрզٸ݃ ӥߊীࢲݣযݶ -
STATE, ACTION SUPERMARIO WITH R.L env.observation_space.shape (240, 256, 3) #
[ height, weight, channel ] env.action_space.n 256 SIMPLE_MOVEMENT = [ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) -
OBSERVATION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [
[‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] -
OBSERVATION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [
[‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] -
ACTION SPACE SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT = [
[‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] -
ACTION AFTER WRAPPER SUPERMARIO WITH R.L env.action_space.n 256 SIMPLE_MOVEMENT =
[ [‘nop’], [‘right’], [‘right’,’A’], [‘right’,’B’], [‘right’,’A’,’B’], [‘A’], [‘left’], ] import gym_super_mario_bros env = gym_super_mario_bros.make(‘SuperMarioBros-v0’) env.observation_space.shape (240, 256, 3) # [ height, weight, channel ] env =BinarySpaceToDiscreteSpaceEnv(env, SIMPLE_MOVEMENT) from nes_py.wrappers import BinarySpaceToDiscreteSpaceEnv -
EXPLOITATION AND EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info
= env.step(action) else : action = np.argmax(output) Exploitation Exploration def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) ? -
EXPLORATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action)
else : action = np.argmax(output) Exploitation Exploration if np.random.rand() < epsilon : action=np.random.randint(output) def epsilon_greedy(q_value,step): ? -
EXPLOITATION SUPERMARIO WITH R.L next_state, reward, done, info = env.step(action)
else : action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) Exploitation Exploration ? -
ENV.STEP( ) SUPERMARIO WITH R.L next_state, reward, done, info =
env.step(action) else : action = np.argmax(output) def epsilon_greedy(q_value,step): if np.random.rand() < epsilon : action=np.random.randint(output) -
EXPLORATION RATE AND REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory
= deque([],maxlen=1000000) memory.append(state,action,reward,next_state) (St , At , Rt+1 , St+1 ) next_state, reward, done, info = env.step(action) eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1 -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1 -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
next_state, reward, done, info = env.step(action) eps_max = 1 eps_decay_steps = 200000 eps_min = 0.1 -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state)
eps_max = 1 eps_min = 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min
= 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory = deque([],maxlen=1000000) memory.append(state,action,reward,next_state) -
REPLAY MEMORY BUFFER SUPERMARIO WITH R.L eps_max = 1 eps_min
= 0.1 eps_decay_steps = 200000 next_state, reward, done, info = env.step(action) memory.append(state,action,reward,next_state) memory = deque([],maxlen=1000000) -
MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf loss
= tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) -
MINIMIZE LOSS SUPERMARIO WITH R.L (Rt+1 + γt+1 maxa′ qθ
(St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) import tensorflow as tf -
MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf Optimizer
=tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) -
MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf training_op
= optimizer.minize(loss) (Rt+1 + γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) -
MINIMIZE LOSS SUPERMARIO WITH R.L import tensorflow as tf (Rt+1
+ γt+1 maxa′ qθ (St+1 , a′) − qθ (St , At ))2 (St , At , Rt+1 , St+1 ) loss = tf.reduce_mean(tf.squre( y - Q_action ) ) Optimizer =tf.train.AdamsOptimizer(learning_rate) training_op = optimizer.minize(loss) -
DOUBLE DQN SUPERMARIO WITH R.L JOQVU "DUJPO WBMVF &OW 2/FUXPSL
s’ s 3FQMBZNFNPSZ 2 T B a r (St , At , Rt+1 , St+1 ) -
*ଵҊ ਊয ӝഐ Time step Action Transition Function Reward Set
of states Set of actions Start state Discount factor t a P(s′, r ∣ s, a) r A S S0 γ Set of reward Policy Reward State R π r REINFORCEMENT LEARNING s