Deep and Reinforced Learning

Deep Learning and Reinforcement Learning

Transcript

1. %FFQ
   BOE
   3FJOGPSDFE
   -FBSOJOH 5FXFJ
   !
   */'03
   )&:%":
   
2. $POUFOUT 1. Intro …1 2. Deep Learning …2 3. Reinforcement Learning 4. Experiments …7 5. Discussion …1 About Me ML summer camp @ NCTU Learned RL and DL for… 3 months? http://ml.infor.org/
3. *OUSP Machine Learning? Human Learning? How we learned from our experience? Machine Learning Fields, Algorithms … blah blah blah
4. %FFQ
   -FBSOJOH Pertaining to Neural Nets Fundamental Ideas Learning Nonlinearity Back propagation Feature Selection and Dimensionality Reduction
5. %FFQ
   -FBSOJOH MLP CNN RNN Autoencoder, Deep Belief Nets…
6. 3FJOGPSDFNFOU
   -FBSOJOH Learning from observation of rewards Learning an optimal policy with best reward
7. .BSLPW
   %FDJTJPO
   1SPDFTT Environment + Agent State(s) Actions(a) for a state Reward R(s,a,s’) Probability for Transition P(s,a,s’) Policy — set of actions to take in each state Environment Action Reward New State Agent
8. .BSLPW
   %FDJTJPO
   1SPDFTT s a s s a a s s s
9. 2-FBSOJOH Q-Value — Expected Reward for taking an action at a state Action with Max Q-Value — Optimal Policy !
10. 2-FBSOJOH s’ s’ s a a’ a’ s’ a’ Q*(s,a) = sum(P(s,a,s’)(R(s,a,s’) + r(max(Q(s’,a’)|a’)|s’)))
11. %ZOBNJD
    1SPHSBNNJOH
     3FBMMZ
12. 10.%1 Life isn’t always good — We don’t know P(s,a,s’) and R(s,a,s’) — Partially observed MDP Needs model-free methods QAQ
13. 2-FBSOJOH Average each observation into previous Q-values Q(s,a) = Q(s,a) + k(R(s,a,s’)+r(max(Q(s’,a’)|a’))) with k << 1 Gain observation by epsilon-greedy method — for probability p act randomly and for (1-p) act on-policy
14. 1SPCMFNT Too much states Orz — Apply feature selection — Approximate Q-Values (LR, NN…) — No end-to-end approaches QAQ
15. 1MBZJOH
    5FUSJT
16. 1MBZJOH
    "UBSJ
    XJUI
    %FFQ
    3FJOGPSDFNFOU
    -FBSOJOH (Google) DeepMind NIPS 2013, Nature 2015 Implemented in Torch http://arxiv.org/abs/ 1312.5602 https://youtu.be/ cjpEIotvwFY
17. %FFQ
    2-FBSOJOH 1. Use artificial neural networks to approximate Q-values 2. Experience Replay — Model Free — Feature Selection Done — OwO
18. %FFQ
    2-FBSOJOH “Second, learning directly from consecutive samples is inefficient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates. Third, when learning on-policy the current parameters determine the next data sample that the parameters are trained on. ”
19. //
    BOE
    2WBMVFT Input — State Output — Q-values for each action Q(s,a) = Q(s,a) + k(R(s,a,s’)+r(max(Q(s’,a’)|a’))) Training (X,Y) = (state, R(s,a,s’)+r(max(Q(s’,a’)|a’))) — Neural Dynamic Programing @@
20. 1MBZJOH
    "UBSJ
    XJUI
    %FFQ
    3FJOGPSDFNFOU
    -FBSOJOH Game (86x86) CNN FC(256) Output(4-18)
21. 1MBZJOH
    "UBSJ
    XJUI
    %FFQ
    3FJOGPSDFNFOU
    -FBSOJOH “The leftmost two plots in figure 2 show how the average total reward evolves during training on the games Seaquest and Breakout. Both averaged reward plots are indeed quite noisy, giving one the impression that the learning algorithm is not making steady progress. ”
22. &YQFSJNFOU 1010! P(19,3) tiles 2100 board states 300 possible actions Orz
23. "SDIJUFDUVSF Input (109) FC Hidden Layer (128) FC Hidden Layer (128) Output (300)
24. %JFSFODFT Not frame based CNN-less Different reward scaling 4-18 actions vs 300 actions Valid actions not consistent
25. *NQMFNFOUBUJPO Theano~! Emulator MLP with RMSprop (without dropout) Deep Q-Learning https://github.com/Newmu/Theano-Tutorials https://github.com/tewei/MLatINFOR/blob/master/ DeepQLearning.ipynb
26. 3FTVMUT Average training score: 125 Average testing score: 147
27. 8JUIPVU
    &YQFSJFODF
    3FQMBZ
    (VFTT
    XIBU
28. .BHJD Average testing score: 224 Best testing score: 600
29. $PNQBSJTPO Random moves — average 230 points QAQ Deep Q-Learning agent tends to put tiles on the sides Good results == NaN Deep Q-Learning Failed or my implementation failed Orz
30. %JTDVTTJPO What caused training to stagnate? How do architecture and hyper parameters affect performance? Explore actions? Multi-Agent Games?
31. 'JOJTIFE_'JOBMMZ 2"
    5JNF