Deep NLP: 딥러닝을 이용한 자연어처리

머신러닝, 딥러닝 2학기 수업분량을 2시간으로 요약
VCNC 개발팀 워크샵 (2017/12/20)

Transcript

1. Deep NLP
   ٩۞׬ਸ ੉ਊೠ ੗োয୊ܻ
   ѐߊ੗ܳ ਤೠ ӝࣿ ইਓۄੋҗ ৘ઁ ઺बਵ۽
   ӣ࢚਋ @ VCNC
   kevin@between.us
   

   View Slide

2. Why??
   • ਋ܻо ೞҊ ੓ח ݽ߄ੌ জ ࠺ૉפझח അ੤о
   ੿੼
   • 5֙, 10֙ ٍীח যڄө?
   • ݠन۞׬਷ ਋ܻ ࢤഝী ௾ ߸ചܳ оઉৢ ࣻ
   ੓ח ӝࣿ ઺ ೞա
   • 100֙ оח ӝস੉ غ۰ݶ ഥࢎب, ࢎۈب ߸
   न੉ ೙ਃ
   • ׮ٜ प۱җ ҃೷ ੓ਵ޲۽ ޤٚ ٮۄ੟ইࢲ ੜ
   ೡ ࣻ ੓ѷ૑݅, ޷ܻ ળ࠺ೞҊ ҙब о૑ݶ જ
   ਸ Ѫ
   

   View Slide

3. ݠन۞׬
   • ஹೊఠ җ೟੄ ೠ ࠙ঠ۽, যڃ ۽૒ਸ ೐۽Ӓې߁ ೞ૑ ঋҊب ஹೊ
   ఠо ੘স੉ա ૑धਸ ߓ਎ ࣻ ੓ѱ ೞח ࠙ঠ
   • ؀ࠗ࠙ Data-driven prediction ഑਷ decision ਸ ࣻ೯ೞח ঌҊ
   ્ܻٜ
   • ؘ੉ఠ ࠙ࢳ ࠙ঠীࢲب օܻ ॳ੉חؘ, ࠂ੟ೠ ݽ؛ਸ ࠙ࢳоо ૒੽
   ٜ݅૑ ঋҊ ؘ੉ఠܳ ా೧ ೟णदெ যځೠ ࢎपਸ ৘ஏೞѢա, ऀѹ
   ૓ ੋࢎ੉౟ܳ ঌইղח ੌ ١ਸ ࣻ೯
   

   View Slide

4. ٩۞׬ (Deep Learning)
   • ݠन۞׬੄ ೞਤ ࠙ঠ ઺ ೞա, ౠ൤ Neural Network (ੋҕ न҃
   ݎ) ҅ৌ ঌҊ્ܻҗ োҙ੓਺
   • Deep Neural Network ੉ۄҊ ࠗܰӝب ೣ
   • ୭Ӕ ಩ߊ੸ਵ۽ ҙबਸ ߉Ҋ ੓ח ࠙ঠ۽, ӝઓ ݠन۞׬ ߑߨۿ੄
   ೠ҅ܳ ӓࠂೞח ઺
   

   View Slide

5. ٩۞׬ (Deep Learning)
   • Rawೠ input (Ӗ੗, ܻࣗ, ױ
   য ١١) ਸ ੑ۱߉਺
   • ઺р઺р ৈ۞ ҅கਵ۽ ؘ੉ఠ
   ٜਸ ઑ೤ೞৈ ಴അ
   • ҅கਸ ੼੼ ऺই ৢ۰оݴ ࢚
   ਤ ѐ֛ਸ ಴അೡ ࣻ ੓ѱ ؽ
   

   View Slide

6. ӝઓ ݠन۞׬ ߑߨٜ੄ ೠ҅
   • അ੤੄ ݠन۞׬ ߑߨٜ਷ ؀ࠗ࠙ Data-Driven ߑߨٜ
   • যڃ ੌਸ द೯ೞח ۽૒ਸ ೐۽Ӓې߁ೞחѪ੉ ইפۄ, ೟णೡ ࣻ ੓
   ח ҳઑܳ ٜ݅য֬Ҋ ؘ੉ఠܳ ઁҕ, ஹೊఠо ঌইࢲ ೟ण
   • যڃ ؘ੉ఠܳ, যڃ ҳઑ۽ ೟णदఃוջо ઺ਃ!
   • ౠ൤ ؘ੉ఠ ࠗ࠙੄ ઺ਃࢿ੉ ௼Ҋ ؘ੉ఠܳ যڌѱ оҕ೧ࢲ ֍וջ
   о ݽ؛੄ ࢿמਸ ઝ਋ೣ -> Feature Engineering
   

   View Slide

7. Feature Engineering
   • ؘ੉ఠܳ оҕ೧ࢲ ஹೊఠܳ ೟णदఆ featureܳ ݅٘۰ݶ, ೧׼ ࠙
   ঠী ؀ೠ ੹ޙ੸ੋ ૑ध (domain knowledge)੉ ೙ਃೞݴ दрҗ
   ࠺ਊ੉ ݆੉ ٘ח ੘স੐
   • ਕ௼೒۽਋
   - Feature engineering -> Machine training -> Evaluate result ->
   Idea -> Feature engineering -> …
   

   View Slide

8. ٩۞׬਷ ޤо ׮ܲо?
   • ٩۞׬ (Deep Neural Network)
   • ಴അ۱੉ જ਷ (ࠂ੟ೠ ղਊٜਸ ೟णೡ ࣻ ੓ח) ֎౟ਕ௼ܳ ৈ۞க
   ऺইࢲ, ֎౟ਕ௼о featureٜਸ झझ۽ ٜ݅যղҊ ೟णೡ ࣻ ੓ب
   ۾ ೞ੗!
   

   View Slide

9. ঱যܳ ਤೠ ٩۞׬
   • ٩۞׬ ݽ؛ਸ ഝਊೠ ਺ࢿੋध
   • ӝઓ੄ ਺ࢿੋध ߑߨ ؀࠺ ࢿמ੉ ࠺ড੸ਵ۽ ೱ࢚ؽ
   

   View Slide

10. ੉޷૑ܳ ਤೠ ٩۞׬
    • ୭Ӕ ٩۞׬ਸ ഝਊ೧ о੢ ݆੉ োҳо غҊ ੓ח ࠙ঠ
    • ImageNet ؀ഥীࢲ જ਷ ࢿ੸ਸ ղݶࢲ ݆਷ ҙबਸ ঳਺
    

    View Slide

11. ੗োয ୊ܻীࢲ੄ ٩۞׬ ਽ਊ
    • ఫझ౟੄ х੿࠙ࢳ (ӛ੿/ࠗ੿)
    • ੋҕ૑מ ࠺ࢲ
    • ૕੄਽׹
    • ੗ز ߣ৉
    

    View Slide

12. ٩۞׬ - х੿࠙ࢳ
    • ੉੹੄ ߑߨ
    - ৈ۞ ױযٜ੉ ӛ੿/ࠗ੿੄ х੿ਸ о૑ח૑ DBҳ୷
    - ޙ੢ী ೧׼ ױযٜ੉ ঴݃ա ١੢ೞח૑ ଺ӝ
    - ੤޷о ੓׮
    - ੤޷о হ׮
    - “੤޷” ۄח ױয݅ਵ۽ח ౸ױೡ ࣻ হ਺
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ޙߨ੉ա ੄޷੸ੋ Ѫө૑ Ҋ۰ೞৈ ౸߹ оמ
    

    View Slide

13. ٩۞׬ - ૕੄਽׹
    • ੉੹੄ ߑߨ
    - ݆ࣻ਷ ૕ޙ-׹߸ DBܳ ҳ୷ೞҊ, ࢜۽ ٜযয়ח ૕ޙਸ ੉੹੄ ૕׹ ࣇ
    ীࢲ ଺਺
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ৈ۞ ૕ޙী ؀ೠ ׹੉ vector ഋधਵ۽ ݽ؛ী ֣ইٜয੓਺
    

    View Slide

14. ٩۞׬ - ੗زߣ৉
    • ੉੹੄ ߑߨ
    - ই઱ ࠂ੟ೠ ৈ۞ ҅க੄ ߣ৉ ݽ؛
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ߣ৉ೡ ޙ੢ਸ vector۽ ಴അೞҊ, ੉ܳ ߸ജೞח ݽ؛ਸ ೟णदఇ
    

    View Slide

15. View Slide

16. ݠन۞׬ ѐਃ
    

    View Slide

17. ݠन۞׬ (Machine Learning)
    • “The field of study that gives computers the ability to learn
    without being explicitly programmed.” - Arthur Samuel
    • “A computer program is said to learn from experience E
    with respect to some class of tasks T and performance
    measure P, if its performance at tasks in T, as measured by
    P, improves with experience E.” - Tom Mitchell
    • ৘: ߄قѱ੐ਸ فח ஹೊఠ
    - E = ߄قѱ੐ਸ فח ҃೷஖
    - T = ߄قѱ੐ਸ فח ೯ਤ
    - P = ஹೊఠо ׮਺౸੄ ߄قਸ ੉ӡ ഛܫ
    

    View Slide

18. ݠन۞׬ (Machine Learning)
    • Supervised Learning (૑ب ೟ण)
    - Regression - োࣘ੸ੋ чਸ ৘ஏ
    • ৘: ࠗز࢑ оѺ ৘ஏ
    - Classification - ࠛোࣘ੸ੋ ч ৘ஏ
    • ৘: ࢎ૓ਸࠁҊ ࢎޛ ݏ୶ӝ
    - Regressionޙઁب Classification ޙઁ۽ ߄Չࣻب ੓׮!
    • ৘: ࠗز࢑ оѺਸ ࠁҊ ࢓૑ উ࢓૑ Ѿ੿
    • Unsupervised Learning (࠺૑ب ೟ण)
    - Clustering
    • Reinforcement Learning (ъച ೟ण)
    - ѥחߨਸ झझ۽ ߓ਋ח ۽ࠈ
    

    View Slide

19. Linear Regression
    

    View Slide

20. Linear Regression
    • ૘੄ և੉ ؘ੉ఠ۽ ࠗز࢑ оѺਸ ৘ஏ೧ ࠁ੗
    • 30 ಣ = 3র 5000݅ਗ
    • 50ಣ = 5র 5000݅ਗ
    • 10ಣ = 1র 5000݅ਗ
    • 20ಣ = ?
    

    View Slide

21. Linear Regression
    • y = a * x + b
    - x: և੉(ಣ)
    - y: оѺ
    - h(x) ৬ э੉ ಴അೞӝب ೣ (hypothesis, оࢸ)
    - ৬ э੉ ಴അ
    • Training dataܳ learning algorithmী ੸ਊ, h ܳ ҳೣ!
    • h ܳ ੉ਊೞৈ x ী ؀ೠ Ѿҗч (y) ܳ ৘ஏ оמ
    

    View Slide

22. Cost Function
    • যڃ о੿ hܳ ࣁਛਸٸ, Ӓ о੿੉ ঴݃ա ౣ۷חо(error) ܳ ࠺ਊ
    (cost)۽ р઱ೞৈ ࣻधਸ ࣁ਑
    • ੉ۧѱ ੿੄ೠ Costܳ ୭ࣗചೞח ߑೱਵ۽ ౵ۄݫఠ a,bٜਸ ઑ੿
    ೞݶ ખ؊ ৢ߄ܲ о੿ h ܳ о૕ ࣻ ੓׮!
    

    View Slide

23. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • पઁ ؘ੉ఠח h(x) = 1*x ੋؘ, h(x) = 0.5*x ۽ ৘ஏೠ ҃਋
    • য়ରч (Error, Cost):
    0 1 2 3
    3
    2
    1
    0
    

    View Slide

24. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • ੄ 2ରߑ੿ध!!
    

    View Slide

25. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • , ী ؀೧ࢲח ইې৬ э਷ ੑ୓੸ੋ ݽন੉ ػ׮
    • যڌѱ ୭੷੼ਸ ଺ਸѪੋо?
    

    View Slide

26. Gradient Descent
    • য়ରчਸ যڌѱ ୭ࣗചೡѪੋо?
    - Cost functionਸ ୭ࣗച
    - Cost function ١Ҋࢶ੄ Ҏ૞ӝ ଺ӝ
    • ੐੄੄ ૑੼ীࢲ द੘ೞৈ, ӝ਎ӝо ծই૑ח ૑੼ଃਵ۽ ҅ࣘ ੉زೞ੗
    • ӝ਎ӝ = Cost function੄ ޷࠙ч
    

    View Slide

27. Gradient Descent
    • ੉ җ੿ਸ ࣻ۴ೡٸө૑ ߈ࠂ
    • a: learning rate
    

    View Slide

28. Cost Function੄ ޷࠙ч
    

    View Slide

29. Cost Function੄ ޷࠙ч
    

    View Slide

30. Linear Regression੄ Gradient Descent
    

    View Slide

31. Linear Regression੄ Gradient Descent
    • Bowl-shaped convex function
    

    View Slide

32. Linear Regression੄ Gradient Descent
    • Iterationਸ ా೧ ୭੸੼ਸ ଺ইх
    (for fixed , this is a function of x) (function of the parameters )
    

    View Slide

33. Linear Regression੄ Gradient Descent
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    

    View Slide

34. Linear Regression੄ Gradient Descent
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    

    View Slide

35. Multi Variable Linear Regression
    • ࠗز࢑ оѺਸ ৘ஏೞחؘח ಣࣻ ݈Ҋب ׮নೠ ੗ܐܳ ଵҊೡ ࣻ ੓
    ਸѪ
    - ߑ੄ іࣻ, ળҕ֙ب, ߓۆ׮ іࣻ ١١..
    • ݽ؛੉ ׮೦ध੄ ഋకо ؽ
    • Feature scaling, feature ઑ೤, ઺ࠂ feature ઁѢ ١ ݆਷ ੘স੉
    ೙ਃೞѱ ؽ
    

    View Slide

36. Logistic Regression
    

    View Slide

37. Classification: Logistic Regression
    • োࣘ੸ੋ чਸ ৘ஏೞחѪ੉ ইצ, ࠛোࣘ੸ੋ чਸ ৘ஏ
    - झಅੋ૑ ইצ૑
    - ঐੋ૑ ইצ૑
    - ੉ ࢎ૓੉ Ҋন੉ੋ૑, ъই૑ੋ૑
    Tumor Size
    Tumor Size
    (Yes) 1
    (No) 0
    

    View Slide

38. Classification: Logistic Regression
    • Sigmoid (Logistic function) ਸ ࢎਊೞৈ և਷ ߧਤ੄ чਸ, 0~1
    ࢎ੉੄ чਵ۽ ߸ജ
    • ݅ড 0.5ܳ ӝળਵ۽ ೠ׮ݶ, 0.5 ੉࢚:1, 0.5 ޷݅: 0
    

    View Slide

39. Logistic Cost Function
    • y = 0 ੌٸ৬ y = 1 ੌٸо ׮ܴ
    • Gradient Descentܳ ഝਊೞח ߑߨ਷ Linear Regressionҗ زੌ!
    

    View Slide

40. Regularization
    • Overfitting (җ୭੸ച) ޙઁ
    - അपࣁ҅੄ पઁ ݽ؛ࠁ׮, ೟णदఅ ݽ؛੉ ࠂ੟بо ؊ ֫ইઉࢲ ઱য૓ ೟णࣇী݅
    җبೞѱ ୭੸ചػ ࢚ട
    - training set ੉ ইצ ࢜۽਍ ޙઁܳ ઁ؀۽ ৘ஏೡ ࣻ হѱ ػ׮
    • Feature ੄ іࣻܳ ઴੉Ѣա, Regularizationਵ۽ ೧Ѿ
    • Regularization
    - Cost functionী ੄ب੸ਵ۽ ࢚ࣻ೦ਸ ୶оदெ ೟णਸ ߑ೧ೣ
    - җبೞѱ ೟णغחѪਸ ߑ૑ೞח ബҗ
    

    View Slide

41. Neural Network
    

    View Slide

42. Neural Network
    • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ
    • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺
    • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ
    • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն
    • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ
    - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण
    

    View Slide

43. Neural Network
    • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ
    • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺
    • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ
    • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն
    • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ
    - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण
    

    View Slide

44. Neural Network
    

    View Slide

45. Neural Network
    

    View Slide

46. Neural Network Playground
    • http://playground.tensorflow.org/
    • 0 hidden layer, 2 neurons (Logistic Regression)
    - ױࣽೠ 1ର ഋక੄ ؘ੉ఠ߆ী ೟णदః૑ ޅೣ
    • 2 hidden layer, 2 neurons
    - Ѿҗо যڌѱ ׮ܲо?
    - ৈ۞க ऺইب Ѿҗо ࠺तೣ
    • neuronіࣻܳ טܻݶ ખ؊ ࠂ੟ೠ ഋకܳ ಴അೡ ࣻ ੓׮
    • Featureܳ טܻݶ?
    • Learning rate੉ ௼ݶ?
    • Deep ೠ ֎౟ਖীࢲ activation function?
    • Regularization
    

    View Slide

47. Neural Network - Cost Function
    Layer 1 Layer 2 Layer 3 Layer 4
    Logistic regression:
    Neural network:
    • Logistic Regression੄ ഛ੢౸!
    

    View Slide

48. Neural Network - Back Propagation
    • Forward propagation
    Layer 1 Layer 2 Layer 3 Layer 4
    

    View Slide

49. Neural Network - Back Propagation
    • Back propagation
    Layer 1 Layer 2 Layer 3 Layer 4
    Intuition: “error” of node in layer .
    For each output unit (layer L = 4)
    

    View Slide

50. Neural Network - ੿ܻ
    • ֎౟ਕ௼ ҳઑܳ ੿ೠ׮
    - Input feature੄ іࣻ, Output class੄ іࣻ, hidden layer੄ іࣻ ߂
    ରਗ
    • weightܳ ےؒೞѱ ୡӝച
    • forward propagation, cost function, back propagation ਸ ҳ
    അ
    • ݽٚ ؘ੉ఠٜী ؀ೞৈ, back propagationਸ ా೧ ҳೠ gradient
    ٜਸ ݽ؛ী ੸ਊ
    • TensorFlowо ׮ ঌইࢲ ೧઻ਃ
    

    View Slide

51. (੤޷۽) ੋр੄ ֱ vs. ׏ۡ֎౟ਕ௼
    • ੋр੄ ֱ
    - ড ୌর ѐ੄ ׏۠, ࣻߔ~ࣻୌઑ ѐ੄ दշझ۽ ੉ܖয૗
    - ੋрҗ زޛ੄ ֱח ׮׮੊ࢶ
    • ׏ۡ֎౟ਕ௼
    - ੋҕ੸ੋ ׏ۡ֎౟ਕ௼੄ ࢎ੉ૉח ࢚؀੸ਵ۽ ੘਺
    - ࢎ੉ૉо ௿ࣻ۾ ೟ण੉ য۰ਕ૗
    • AlphaGo
    - ৈ۞ѐ੄ ׏ۡ ֎౟ਕ௼੄ ઑ೤. (13 layer੄ Convolutional neural
    network ١)
    - Input layer: 48ѐ੄ 19x19 ߄ق౸ = 17328 ׏۠
    - ࣻߔ݅ѐ੄ ׏۠
    

    View Slide

52. ঱য ݽ؛җ
    RNN
    (Recurrent Neural Networks)
    

    View Slide

53. RNN?
    • !?!?!?
    • RNN੉ ޤӡې, ੉۠Ѫਸ
    ೡࣻ੓חо
    • RNNਸ Ҿӓ੄ ੋҕ न҃
    ݎ ҳઑۄҊ ઱੢ೞח ࢎ
    ۈٜب ੓਺
    

    View Slide

54. RNN (Recurrent Neural Networks)
    • ݆਷ NLP ੘সী ই઱ જ਷ ࢿמਸ ղષ
    • ѐ֛੉ য۵ӝب ೞҊ ઁ؀۽ ࢸݺೞҊ ੓ח ੗ܐо হ਺
    - ઁ؀۽ ੉೧ೞח ࢎۈب ੜ হחѪ э਺
    • য়טب ઁ؀۽ ࢸݺೡ ࣻ ੓ਸ૑ ഛप൤ ݽܰѷ਺
    

    View Slide

55. ঱য ݽ؛ (Language Model)
    • োࣘػ ױযٜ੄ ١੢ഛܫਸ ৘ஏೞח ݽ؛
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - ___ী ٜযт ױযח?
    - ݡ঻׮, ݡח׮, ૑঻׮,…
    - p(ݡ঻׮) = 0.25, p(ݡח׮) = 0.15, p(૑঻׮) = 0.05, ..
    - p(੗زର) = 0.000001
    • ޙ੢੄ ࣽࢲա ױযࢶఖ ١ਸ ৘ஏೡ ࣻ ੓ӝী, ੗زߣ৉ ١ ݆਷
    NLP Taskী ਬਊೞ׮
    

    View Slide

56. ঱য ݽ؛ (Language Model)
    • ഛܫ੸ ঱য ݽ؛ਸ ݅٘۰ݶ, ੉੹ী աয়ח ױযܳ ݆੉ ଵઑೡࣻ۾
    ੿ഛೞ׮!
    - 2-gram, 3-gram, 4-gram, .. n-gram
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - ߏਸ ____ ߓо Ҋ౵ࢲ աח ߏਸ ____
    • n-gramਸ טܾࣻ۾ ݫݽܻо ষ୒աѱ ೙ਃೞ׮!
    - ੉ ޙઁܳ ೧Ѿ೧઱חѪ੉ RNN(Recurrent Neural Networks)
    

    View Slide

57. RNN (Recurrent Neural Networks)
    • दрী ٮۄ (഑਷ ؘ੉ఠ੄ ૓೯ী ٮۄ) ҅ࣘ সؘ੉౟غח ࢚కчਸ р૒ೞҊ ੓ח
    ֎౟ਕ௼
    • ױࣽೠ ࢚కо ইפۄ, ೟णਸ ૑ࣘೡࣻ۾ ੑ۱ਸ ୊ܻೞח ҭ੢൤ ࠂ੟ೠ ۽૒੉ ֣
    ইٜѱؽ
    • ࢜۽ աৢ ױযܳ ৘ஏೞחؘী ੉੹੄ ݽٚ ױযٜ੄ ੿ࠁܳ ଵઑೞѱ ؽ
    

    View Slide

58. RNN (Recurrent Neural Networks)
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - 4ѐ੄ ױযܳ ଵઑ - RNNীࢲח 4 layer neural network੉ ࢤӣ
    

    View Slide

59. RNN (Recurrent Neural Networks)
    • RNNਸ ೟णदఃחѪ਷ য۵׮!
    • Vanishing Gradient ޙઁ
    - ցޖ য়ې੹੄ ੿ࠁө૑ ଵઑೞ۰׮ࠁפ, য٣ࢲ ੜޅ೮঻ח૑ ೟णೞӝ
    য۵ѱ ؽ
    • Exploding Gradient ޙઁ
    • ReLU / Clipping ١ ৈ۞о૑ ప௼ץਵ۽ ӓࠂ!
    

    View Slide

60. RNN (Recurrent Neural Networks)
    • ؊ ੗ࣁೠ ೟ण੗ܐ
    - http://www.wildml.com/2015/09/recurrent-neural-networks-
    tutorial-part-1-introduction-to-rnns/
    

    View Slide

61. LSTM (Long Short-Term Memory)
    • RNNҗ ਬࢎೠ ҳઑ੄ ֎౟ਕ௼
    • RNN਷ ߈ࠂ ҳઑ۽ ੋ೧ ӝর۱ਸ(memory) о૑ѱ غ঻਺
    • दр୷ਵ۽ ಽযࠁݶ, ৈ۞ѐ੄ ֎౟ਕ௼о ੓Ҋ п੗о ੗न੄ ׮਺
    ֎౟ਕ௼ী ݫࣁ૑ܳ ࠁղחѪҗ э਷ ਗܻ
    • োࣘػ ؘ੉ఠ৬ Ө਷ োҙ
    - ঱য ݽ؛, ߣ৉, ਺ࢿੋध ١١
    RNNҗ ੉ܳ दр୷ਵ۽ ಽ੉ೠ Ӓܿ
    

    View Slide

62. LSTM (Long Short-Term Memory)
    • RNN਷ ୭Ӕ ݻѐ ױয ੿ب੄ ӝর۱ ੿ب݅ਸ о૗
    • LSTM: ؊਌ ӟ ӝর۱ਸ о૑ӝ ਤೠ ҳઑ
    

    View Slide

63. LSTM (Long Short-Term Memory)
    • Cell State: п ױ҅݃׮ ই઱ ੸਷ ࣻ੿ਸ Ѣ஖ݴ ੿ࠁܳ য়ۖزউ
    ࠁઓೠ׮.
    

    View Slide

64. LSTM (Long Short-Term Memory)
    • Forget gate: ੉੹ cell state઺ী ࠛ೙ਃೠ ੿ࠁܳ ઁѢ
    • Input gate: ࢜۽਍ ੿ࠁܳ cell stateী ӝর
    

    View Slide

65. LSTM (Long Short-Term Memory)
    • Forget gate৬ input gate੄ ઑ೤ਵ۽ cell state ܳ সؘ੉౟
    • Output: cell state੄ ੌࠗ࠙ਵ۽ чਸ ୹۱
    

    View Slide

66. GRU (Gated Recurrent Unit)
    • LSTMҗ Ѣ੄ ਬࢎೠ ҳઑ, ؊ рױೠ ҳഅ
    - Forget gate ৬ input gate ܳ ೞա۽ ೤ஜ
    

    View Slide

67. Attention Mechanism
    • ੋр੄ ੋ૑җ੿ਸ ࢤп೧ࠁݶ, ݽٚ ࣽрী ݽٚࠗ࠙ী न҃ਸ ॳח
    Ѫ੉ ইפۄ, ౠ੿ ࣽрী ౠ੿ ࢎޛ੉ա ઱ઁী ૘઺ೞח ݽणਸ ࠁ
    ੐
    • RNNэ਷ ݽ؛੄ ӡ੉о ӡয૕ࣻ۾ ೞա੄ Hidden Stateী ݽٚ
    ղਊਸ ӝরೞӝо য۰ਕ૗
    • Hidden stateٜਸ ੷੢ೞҊ ੓׮о, ੉੹੄ stateٜਸ ഝਊ
    

    View Slide

68. ؊ ࠂ੟ೠ ֎౟ਕ௼
    

    View Slide

69. Dynamic Memory Networks
    • যڃ Ӗਸ ੍Ҋ, Ӓ Ӗী ؀ೠ ޛ਺ী ׹ਸ ೞחѪ
    - ࣻמद೷ ঱য৔৉җ ࠺तೣ
    - औ૑ ঋ਷ җઁ੉׮!
    

    View Slide

70. Dynamic Memory Networks
    • ӝמ߹۽ RNNਸ ઑ೤
    • End-to-End ೟ण
    • ֤ޙਸ ੍੗!
    - https://arxiv.org/abs/1506.07285
    

    View Slide

71. Dynamic Memory Networks
    • Input Module
    - RNN (GRU), п ޙ੢ٜ੄ hidden stateܳ ੷੢
    • Question Module
    - рױೠ RNN (GRU)
    • Episodic Memory Module
    - ੉ঠӝী ؀ೠ ੿ࠁܳ ҙ੢ೞח ݽٕ
    - ৈ۞ க੄ RNN (GRU) ۽ ҳࢿ
    

    View Slide

72. Dynamic Memory Networks
    • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!
    

    View Slide

73. Dynamic Memory Networks
    • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!
    

    View Slide

74. Dynamic Memory Networks
    

    View Slide

75. Dynamic Memory Networks
    

    View Slide

76. Dynamic Memory Networks
    

    View Slide

77. Dynamic Memory Networks
    • Attention + Memoryܳ ా೧ ૕ޙী ׹ೞח מ۱ਸ ഛ੢
    • ӝמ߹۽ RNNਵ۽ ੉ܖয૓ ߹ب੄ Module, ੑ۱ ഋకо ׮ܲ ݽ
    ٕ۽ ߸҃ೞחѪب оמೣ
    • ੉޷૑ ੑ۱ ݽٕҗ Ѿ೤غݶ ֥ۄ਍ Ѿҗܳ ࠁৈષ
    

    View Slide

78. NMT (Neural Machine Translation)
    • ࠂ੟ೠ ֎౟ਕ௼ҳઑܳ ഝਊೠ ੗زߣ৉ ֎౟ਕ௼
    

    View Slide

79. Google NMT
    • https://translate.google.com
    • https://research.googleblog.com/2016/09/a-neural-network-for-
    machine.html
    

    View Slide

80. ٩۞׬ ੸ਊী ؀ೠ ࢤп
    • ٩۞׬ = ই੉٣য + ؘ੉ఠ + ݽ؛ ೟ण
    - ݽ؛ ೟ण - ੼੼ ੷۴೧૗
    - ই੉٣য - ੼੼ ؊ ݆਷ ࢎۈٜ੉ ઙࢎೞݶࢲ ই੉٣যب ݆ই૕ Ѫ
    - ؘ੉ఠ - ݠन۞׬, ٩۞׬ द؀ীࢲ о੢ ൞ࣗೠ ੗ਗ
    • ؀ӏݽ ؘ੉ఠܳ ഝਊೠ ݠन۞׬, ؊ ࠂ੟ೠ ઑ೤੄ ݽ؛, ࠙ঠ߹ ௼
    ۽झ ݽ؛ ١ ই૒ ߊ੹ оמࢿ੉ ޖҾޖ૓ೣ
    • ҕࠗೞҊ োҳ೤द׮!
    

    View Slide

81. More..?
    • Join VCNC and join #study_ml
    ੷൞ח ঱ઁա ࠺౟ਦ ࢲ࠺झܳ ೣԋ ٜ݅ݴ ӝࣿ੸ੋ ޙઁܳ ೣԋ ಽযաт מ۱੓ח ѐߊ੗ܳ ݽदҊ ੓णפ׮.
    ঱ઁٚ ࠗ׸হ੉ jobs@vcnc.co.kr۽ ੉ݫੌਸ ઱दӝ ߄ۉפ׮!
    

    View Slide

82. Thank you!
    

    View Slide