Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep NLP: 딥러닝을 이용한 자연어처리

VCNC
December 20, 2017

Deep NLP: 딥러닝을 이용한 자연어처리

머신러닝, 딥러닝 2학기 수업분량을 2시간으로 요약
VCNC 개발팀 워크샵 (2017/12/20)

VCNC

December 20, 2017
Tweet

More Decks by VCNC

Other Decks in Technology

Transcript

  1. Deep NLP
    ٩۞׬ਸ ੉ਊೠ ੗োয୊ܻ
    ѐߊ੗ܳ ਤೠ ӝࣿ ইਓۄੋҗ ৘ઁ ઺बਵ۽
    ӣ࢚਋ @ VCNC
    [email protected]

    View full-size slide

  2. Why??
    • ਋ܻо ೞҊ ੓ח ݽ߄ੌ জ ࠺ૉפझח അ੤о
    ੿੼
    • 5֙, 10֙ ٍীח যڄө?
    • ݠन۞׬਷ ਋ܻ ࢤഝী ௾ ߸ചܳ оઉৢ ࣻ
    ੓ח ӝࣿ ઺ ೞա
    • 100֙ оח ӝস੉ غ۰ݶ ഥࢎب, ࢎۈب ߸
    न੉ ೙ਃ
    • ׮ٜ प۱җ ҃೷ ੓ਵ޲۽ ޤٚ ٮۄ੟ইࢲ ੜ
    ೡ ࣻ ੓ѷ૑݅, ޷ܻ ળ࠺ೞҊ ҙब о૑ݶ જ
    ਸ Ѫ

    View full-size slide

  3. ݠन۞׬
    • ஹೊఠ җ೟੄ ೠ ࠙ঠ۽, যڃ ۽૒ਸ ೐۽Ӓې߁ ೞ૑ ঋҊب ஹೊ
    ఠо ੘স੉ա ૑धਸ ߓ਎ ࣻ ੓ѱ ೞח ࠙ঠ
    • ؀ࠗ࠙ Data-driven prediction ഑਷ decision ਸ ࣻ೯ೞח ঌҊ
    ્ܻٜ
    • ؘ੉ఠ ࠙ࢳ ࠙ঠীࢲب օܻ ॳ੉חؘ, ࠂ੟ೠ ݽ؛ਸ ࠙ࢳоо ૒੽
    ٜ݅૑ ঋҊ ؘ੉ఠܳ ా೧ ೟णदெ যځೠ ࢎपਸ ৘ஏೞѢա, ऀѹ
    ૓ ੋࢎ੉౟ܳ ঌইղח ੌ ١ਸ ࣻ೯

    View full-size slide

  4. ٩۞׬ (Deep Learning)
    • ݠन۞׬੄ ೞਤ ࠙ঠ ઺ ೞա, ౠ൤ Neural Network (ੋҕ न҃
    ݎ) ҅ৌ ঌҊ્ܻҗ োҙ੓਺
    • Deep Neural Network ੉ۄҊ ࠗܰӝب ೣ
    • ୭Ӕ ಩ߊ੸ਵ۽ ҙबਸ ߉Ҋ ੓ח ࠙ঠ۽, ӝઓ ݠन۞׬ ߑߨۿ੄
    ೠ҅ܳ ӓࠂೞח ઺

    View full-size slide

  5. ٩۞׬ (Deep Learning)
    • Rawೠ input (Ӗ੗, ܻࣗ, ױ
    য ١١) ਸ ੑ۱߉਺
    • ઺р઺р ৈ۞ ҅கਵ۽ ؘ੉ఠ
    ٜਸ ઑ೤ೞৈ ಴അ
    • ҅கਸ ੼੼ ऺই ৢ۰оݴ ࢚
    ਤ ѐ֛ਸ ಴അೡ ࣻ ੓ѱ ؽ

    View full-size slide

  6. ӝઓ ݠन۞׬ ߑߨٜ੄ ೠ҅
    • അ੤੄ ݠन۞׬ ߑߨٜ਷ ؀ࠗ࠙ Data-Driven ߑߨٜ
    • যڃ ੌਸ द೯ೞח ۽૒ਸ ೐۽Ӓې߁ೞחѪ੉ ইפۄ, ೟णೡ ࣻ ੓
    ח ҳઑܳ ٜ݅য֬Ҋ ؘ੉ఠܳ ઁҕ, ஹೊఠо ঌইࢲ ೟ण
    • যڃ ؘ੉ఠܳ, যڃ ҳઑ۽ ೟णदఃוջо ઺ਃ!
    • ౠ൤ ؘ੉ఠ ࠗ࠙੄ ઺ਃࢿ੉ ௼Ҋ ؘ੉ఠܳ যڌѱ оҕ೧ࢲ ֍וջ
    о ݽ؛੄ ࢿמਸ ઝ਋ೣ -> Feature Engineering

    View full-size slide

  7. Feature Engineering
    • ؘ੉ఠܳ оҕ೧ࢲ ஹೊఠܳ ೟णदఆ featureܳ ݅٘۰ݶ, ೧׼ ࠙
    ঠী ؀ೠ ੹ޙ੸ੋ ૑ध (domain knowledge)੉ ೙ਃೞݴ दрҗ
    ࠺ਊ੉ ݆੉ ٘ח ੘স੐
    • ਕ௼೒۽਋
    - Feature engineering -> Machine training -> Evaluate result ->
    Idea -> Feature engineering -> …

    View full-size slide

  8. ٩۞׬਷ ޤо ׮ܲо?
    • ٩۞׬ (Deep Neural Network)
    • ಴അ۱੉ જ਷ (ࠂ੟ೠ ղਊٜਸ ೟णೡ ࣻ ੓ח) ֎౟ਕ௼ܳ ৈ۞க
    ऺইࢲ, ֎౟ਕ௼о featureٜਸ झझ۽ ٜ݅যղҊ ೟णೡ ࣻ ੓ب
    ۾ ೞ੗!

    View full-size slide

  9. ঱যܳ ਤೠ ٩۞׬
    • ٩۞׬ ݽ؛ਸ ഝਊೠ ਺ࢿੋध
    • ӝઓ੄ ਺ࢿੋध ߑߨ ؀࠺ ࢿמ੉ ࠺ড੸ਵ۽ ೱ࢚ؽ

    View full-size slide

  10. ੉޷૑ܳ ਤೠ ٩۞׬
    • ୭Ӕ ٩۞׬ਸ ഝਊ೧ о੢ ݆੉ োҳо غҊ ੓ח ࠙ঠ
    • ImageNet ؀ഥীࢲ જ਷ ࢿ੸ਸ ղݶࢲ ݆਷ ҙबਸ ঳਺

    View full-size slide

  11. ੗োয ୊ܻীࢲ੄ ٩۞׬ ਽ਊ
    • ఫझ౟੄ х੿࠙ࢳ (ӛ੿/ࠗ੿)
    • ੋҕ૑מ ࠺ࢲ
    • ૕੄਽׹
    • ੗ز ߣ৉

    View full-size slide

  12. ٩۞׬ - х੿࠙ࢳ
    • ੉੹੄ ߑߨ
    - ৈ۞ ױযٜ੉ ӛ੿/ࠗ੿੄ х੿ਸ о૑ח૑ DBҳ୷
    - ޙ੢ী ೧׼ ױযٜ੉ ঴݃ա ١੢ೞח૑ ଺ӝ
    - ੤޷о ੓׮
    - ੤޷о হ׮
    - “੤޷” ۄח ױয݅ਵ۽ח ౸ױೡ ࣻ হ਺
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ޙߨ੉ա ੄޷੸ੋ Ѫө૑ Ҋ۰ೞৈ ౸߹ оמ

    View full-size slide

  13. ٩۞׬ - ૕੄਽׹
    • ੉੹੄ ߑߨ
    - ݆ࣻ਷ ૕ޙ-׹߸ DBܳ ҳ୷ೞҊ, ࢜۽ ٜযয়ח ૕ޙਸ ੉੹੄ ૕׹ ࣇ
    ীࢲ ଺਺
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ৈ۞ ૕ޙী ؀ೠ ׹੉ vector ഋधਵ۽ ݽ؛ী ֣ইٜয੓਺

    View full-size slide

  14. ٩۞׬ - ੗زߣ৉
    • ੉੹੄ ߑߨ
    - ই઱ ࠂ੟ೠ ৈ۞ ҅க੄ ߣ৉ ݽ؛
    • ٩۞׬ਸ ഝਊೠ ߑߨ
    - ߣ৉ೡ ޙ੢ਸ vector۽ ಴അೞҊ, ੉ܳ ߸ജೞח ݽ؛ਸ ೟णदఇ

    View full-size slide

  15. ݠन۞׬ ѐਃ

    View full-size slide

  16. ݠन۞׬ (Machine Learning)
    • “The field of study that gives computers the ability to learn
    without being explicitly programmed.” - Arthur Samuel
    • “A computer program is said to learn from experience E
    with respect to some class of tasks T and performance
    measure P, if its performance at tasks in T, as measured by
    P, improves with experience E.” - Tom Mitchell
    • ৘: ߄قѱ੐ਸ فח ஹೊఠ
    - E = ߄قѱ੐ਸ فח ҃೷஖
    - T = ߄قѱ੐ਸ فח ೯ਤ
    - P = ஹೊఠо ׮਺౸੄ ߄قਸ ੉ӡ ഛܫ

    View full-size slide

  17. ݠन۞׬ (Machine Learning)
    • Supervised Learning (૑ب ೟ण)
    - Regression - োࣘ੸ੋ чਸ ৘ஏ
    • ৘: ࠗز࢑ оѺ ৘ஏ
    - Classification - ࠛোࣘ੸ੋ ч ৘ஏ
    • ৘: ࢎ૓ਸࠁҊ ࢎޛ ݏ୶ӝ
    - Regressionޙઁب Classification ޙઁ۽ ߄Չࣻب ੓׮!
    • ৘: ࠗز࢑ оѺਸ ࠁҊ ࢓૑ উ࢓૑ Ѿ੿
    • Unsupervised Learning (࠺૑ب ೟ण)
    - Clustering
    • Reinforcement Learning (ъച ೟ण)
    - ѥחߨਸ झझ۽ ߓ਋ח ۽ࠈ

    View full-size slide

  18. Linear Regression

    View full-size slide

  19. Linear Regression
    • ૘੄ և੉ ؘ੉ఠ۽ ࠗز࢑ оѺਸ ৘ஏ೧ ࠁ੗
    • 30 ಣ = 3র 5000݅ਗ
    • 50ಣ = 5র 5000݅ਗ
    • 10ಣ = 1র 5000݅ਗ
    • 20ಣ = ?

    View full-size slide

  20. Linear Regression
    • y = a * x + b
    - x: և੉(ಣ)
    - y: оѺ
    - h(x) ৬ э੉ ಴അೞӝب ೣ (hypothesis, оࢸ)
    - ৬ э੉ ಴അ
    • Training dataܳ learning algorithmী ੸ਊ, h ܳ ҳೣ!
    • h ܳ ੉ਊೞৈ x ী ؀ೠ Ѿҗч (y) ܳ ৘ஏ оמ

    View full-size slide

  21. Cost Function
    • যڃ о੿ hܳ ࣁਛਸٸ, Ӓ о੿੉ ঴݃ա ౣ۷חо(error) ܳ ࠺ਊ
    (cost)۽ р઱ೞৈ ࣻधਸ ࣁ਑
    • ੉ۧѱ ੿੄ೠ Costܳ ୭ࣗചೞח ߑೱਵ۽ ౵ۄݫఠ a,bٜਸ ઑ੿
    ೞݶ ખ؊ ৢ߄ܲ о੿ h ܳ о૕ ࣻ ੓׮!

    View full-size slide

  22. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • पઁ ؘ੉ఠח h(x) = 1*x ੋؘ, h(x) = 0.5*x ۽ ৘ஏೠ ҃਋
    • য়ରч (Error, Cost):
    0 1 2 3
    3
    2
    1
    0

    View full-size slide

  23. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • ੄ 2ରߑ੿ध!!

    View full-size slide

  24. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function
    • , ী ؀೧ࢲח ইې৬ э਷ ੑ୓੸ੋ ݽন੉ ػ׮
    • যڌѱ ୭੷੼ਸ ଺ਸѪੋо?

    View full-size slide

  25. Gradient Descent
    • য়ରчਸ যڌѱ ୭ࣗചೡѪੋо?
    - Cost functionਸ ୭ࣗച
    - Cost function ١Ҋࢶ੄ Ҏ૞ӝ ଺ӝ
    • ੐੄੄ ૑੼ীࢲ द੘ೞৈ, ӝ਎ӝо ծই૑ח ૑੼ଃਵ۽ ҅ࣘ ੉زೞ੗
    • ӝ਎ӝ = Cost function੄ ޷࠙ч

    View full-size slide

  26. Gradient Descent
    • ੉ җ੿ਸ ࣻ۴ೡٸө૑ ߈ࠂ
    • a: learning rate

    View full-size slide

  27. Cost Function੄ ޷࠙ч

    View full-size slide

  28. Cost Function੄ ޷࠙ч

    View full-size slide

  29. Linear Regression੄ Gradient Descent

    View full-size slide

  30. Linear Regression੄ Gradient Descent
    • Bowl-shaped convex function

    View full-size slide

  31. Linear Regression੄ Gradient Descent
    • Iterationਸ ా೧ ୭੸੼ਸ ଺ইх
    (for fixed , this is a function of x) (function of the parameters )

    View full-size slide

  32. Linear Regression੄ Gradient Descent
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )

    View full-size slide

  33. Linear Regression੄ Gradient Descent
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )
    (for fixed , this is a function of x) (function of the parameters )

    View full-size slide

  34. Multi Variable Linear Regression
    • ࠗز࢑ оѺਸ ৘ஏೞחؘח ಣࣻ ݈Ҋب ׮নೠ ੗ܐܳ ଵҊೡ ࣻ ੓
    ਸѪ
    - ߑ੄ іࣻ, ળҕ֙ب, ߓۆ׮ іࣻ ١١..
    • ݽ؛੉ ׮೦ध੄ ഋకо ؽ
    • Feature scaling, feature ઑ೤, ઺ࠂ feature ઁѢ ١ ݆਷ ੘স੉
    ೙ਃೞѱ ؽ

    View full-size slide

  35. Logistic Regression

    View full-size slide

  36. Classification: Logistic Regression
    • োࣘ੸ੋ чਸ ৘ஏೞחѪ੉ ইצ, ࠛোࣘ੸ੋ чਸ ৘ஏ
    - झಅੋ૑ ইצ૑
    - ঐੋ૑ ইצ૑
    - ੉ ࢎ૓੉ Ҋন੉ੋ૑, ъই૑ੋ૑
    Tumor Size
    Tumor Size
    (Yes) 1
    (No) 0

    View full-size slide

  37. Classification: Logistic Regression
    • Sigmoid (Logistic function) ਸ ࢎਊೞৈ և਷ ߧਤ੄ чਸ, 0~1
    ࢎ੉੄ чਵ۽ ߸ജ
    • ݅ড 0.5ܳ ӝળਵ۽ ೠ׮ݶ, 0.5 ੉࢚:1, 0.5 ޷݅: 0

    View full-size slide

  38. Logistic Cost Function
    • y = 0 ੌٸ৬ y = 1 ੌٸо ׮ܴ
    • Gradient Descentܳ ഝਊೞח ߑߨ਷ Linear Regressionҗ زੌ!

    View full-size slide

  39. Regularization
    • Overfitting (җ୭੸ച) ޙઁ
    - അपࣁ҅੄ पઁ ݽ؛ࠁ׮, ೟णदఅ ݽ؛੉ ࠂ੟بо ؊ ֫ইઉࢲ ઱য૓ ೟णࣇী݅
    җبೞѱ ୭੸ചػ ࢚ട
    - training set ੉ ইצ ࢜۽਍ ޙઁܳ ઁ؀۽ ৘ஏೡ ࣻ হѱ ػ׮
    • Feature ੄ іࣻܳ ઴੉Ѣա, Regularizationਵ۽ ೧Ѿ
    • Regularization
    - Cost functionী ੄ب੸ਵ۽ ࢚ࣻ೦ਸ ୶оदெ ೟णਸ ߑ೧ೣ
    - җبೞѱ ೟णغחѪਸ ߑ૑ೞח ബҗ

    View full-size slide

  40. Neural Network

    View full-size slide

  41. Neural Network
    • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ
    • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺
    • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ
    • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն
    • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ
    - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण

    View full-size slide

  42. Neural Network
    • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ
    • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺
    • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ
    • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն
    • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ
    - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण

    View full-size slide

  43. Neural Network

    View full-size slide

  44. Neural Network

    View full-size slide

  45. Neural Network Playground
    • http://playground.tensorflow.org/
    • 0 hidden layer, 2 neurons (Logistic Regression)
    - ױࣽೠ 1ର ഋక੄ ؘ੉ఠ߆ী ೟णदః૑ ޅೣ
    • 2 hidden layer, 2 neurons
    - Ѿҗо যڌѱ ׮ܲо?
    - ৈ۞க ऺইب Ѿҗо ࠺तೣ
    • neuronіࣻܳ טܻݶ ખ؊ ࠂ੟ೠ ഋకܳ ಴അೡ ࣻ ੓׮
    • Featureܳ טܻݶ?
    • Learning rate੉ ௼ݶ?
    • Deep ೠ ֎౟ਖীࢲ activation function?
    • Regularization

    View full-size slide

  46. Neural Network - Cost Function
    Layer 1 Layer 2 Layer 3 Layer 4
    Logistic regression:
    Neural network:
    • Logistic Regression੄ ഛ੢౸!

    View full-size slide

  47. Neural Network - Back Propagation
    • Forward propagation
    Layer 1 Layer 2 Layer 3 Layer 4

    View full-size slide

  48. Neural Network - Back Propagation
    • Back propagation
    Layer 1 Layer 2 Layer 3 Layer 4
    Intuition: “error” of node in layer .
    For each output unit (layer L = 4)

    View full-size slide

  49. Neural Network - ੿ܻ
    • ֎౟ਕ௼ ҳઑܳ ੿ೠ׮
    - Input feature੄ іࣻ, Output class੄ іࣻ, hidden layer੄ іࣻ ߂
    ରਗ
    • weightܳ ےؒೞѱ ୡӝച
    • forward propagation, cost function, back propagation ਸ ҳ

    • ݽٚ ؘ੉ఠٜী ؀ೞৈ, back propagationਸ ా೧ ҳೠ gradient
    ٜਸ ݽ؛ী ੸ਊ
    • TensorFlowо ׮ ঌইࢲ ೧઻ਃ

    View full-size slide

  50. (੤޷۽) ੋр੄ ֱ vs. ׏ۡ֎౟ਕ௼
    • ੋр੄ ֱ
    - ড ୌর ѐ੄ ׏۠, ࣻߔ~ࣻୌઑ ѐ੄ दշझ۽ ੉ܖয૗
    - ੋрҗ زޛ੄ ֱח ׮׮੊ࢶ
    • ׏ۡ֎౟ਕ௼
    - ੋҕ੸ੋ ׏ۡ֎౟ਕ௼੄ ࢎ੉ૉח ࢚؀੸ਵ۽ ੘਺
    - ࢎ੉ૉо ௿ࣻ۾ ೟ण੉ য۰ਕ૗
    • AlphaGo
    - ৈ۞ѐ੄ ׏ۡ ֎౟ਕ௼੄ ઑ೤. (13 layer੄ Convolutional neural
    network ١)
    - Input layer: 48ѐ੄ 19x19 ߄ق౸ = 17328 ׏۠
    - ࣻߔ݅ѐ੄ ׏۠

    View full-size slide

  51. ঱য ݽ؛җ
    RNN
    (Recurrent Neural Networks)

    View full-size slide

  52. RNN?
    • !?!?!?
    • RNN੉ ޤӡې, ੉۠Ѫਸ
    ೡࣻ੓חо
    • RNNਸ Ҿӓ੄ ੋҕ न҃
    ݎ ҳઑۄҊ ઱੢ೞח ࢎ
    ۈٜب ੓਺

    View full-size slide

  53. RNN (Recurrent Neural Networks)
    • ݆਷ NLP ੘সী ই઱ જ਷ ࢿמਸ ղષ
    • ѐ֛੉ য۵ӝب ೞҊ ઁ؀۽ ࢸݺೞҊ ੓ח ੗ܐо হ਺
    - ઁ؀۽ ੉೧ೞח ࢎۈب ੜ হחѪ э਺
    • য়טب ઁ؀۽ ࢸݺೡ ࣻ ੓ਸ૑ ഛप൤ ݽܰѷ਺

    View full-size slide

  54. ঱য ݽ؛ (Language Model)
    • োࣘػ ױযٜ੄ ١੢ഛܫਸ ৘ஏೞח ݽ؛
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - ___ী ٜযт ױযח?
    - ݡ঻׮, ݡח׮, ૑঻׮,…
    - p(ݡ঻׮) = 0.25, p(ݡח׮) = 0.15, p(૑঻׮) = 0.05, ..
    - p(੗زର) = 0.000001
    • ޙ੢੄ ࣽࢲա ױযࢶఖ ١ਸ ৘ஏೡ ࣻ ੓ӝী, ੗زߣ৉ ١ ݆਷
    NLP Taskী ਬਊೞ׮

    View full-size slide

  55. ঱য ݽ؛ (Language Model)
    • ഛܫ੸ ঱য ݽ؛ਸ ݅٘۰ݶ, ੉੹ী աয়ח ױযܳ ݆੉ ଵઑೡࣻ۾
    ੿ഛೞ׮!
    - 2-gram, 3-gram, 4-gram, .. n-gram
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - ߏਸ ____ <-> ߓо Ҋ౵ࢲ աח ߏਸ ____
    • n-gramਸ טܾࣻ۾ ݫݽܻо ষ୒աѱ ೙ਃೞ׮!
    - ੉ ޙઁܳ ೧Ѿ೧઱חѪ੉ RNN(Recurrent Neural Networks)

    View full-size slide

  56. RNN (Recurrent Neural Networks)
    • दрী ٮۄ (഑਷ ؘ੉ఠ੄ ૓೯ী ٮۄ) ҅ࣘ সؘ੉౟غח ࢚కчਸ р૒ೞҊ ੓ח
    ֎౟ਕ௼
    • ױࣽೠ ࢚కо ইפۄ, ೟णਸ ૑ࣘೡࣻ۾ ੑ۱ਸ ୊ܻೞח ҭ੢൤ ࠂ੟ೠ ۽૒੉ ֣
    ইٜѱؽ
    • ࢜۽ աৢ ױযܳ ৘ஏೞחؘী ੉੹੄ ݽٚ ױযٜ੄ ੿ࠁܳ ଵઑೞѱ ؽ

    View full-size slide

  57. RNN (Recurrent Neural Networks)
    • ߓо Ҋ౵ࢲ աח ߏਸ ____
    - 4ѐ੄ ױযܳ ଵઑ - RNNীࢲח 4 layer neural network੉ ࢤӣ

    View full-size slide

  58. RNN (Recurrent Neural Networks)
    • RNNਸ ೟णदఃחѪ਷ য۵׮!
    • Vanishing Gradient ޙઁ
    - ցޖ য়ې੹੄ ੿ࠁө૑ ଵઑೞ۰׮ࠁפ, য٣ࢲ ੜޅ೮঻ח૑ ೟णೞӝ
    য۵ѱ ؽ
    • Exploding Gradient ޙઁ
    • ReLU / Clipping ١ ৈ۞о૑ ప௼ץਵ۽ ӓࠂ!

    View full-size slide

  59. RNN (Recurrent Neural Networks)
    • ؊ ੗ࣁೠ ೟ण੗ܐ
    - http://www.wildml.com/2015/09/recurrent-neural-networks-
    tutorial-part-1-introduction-to-rnns/

    View full-size slide

  60. LSTM (Long Short-Term Memory)
    • RNNҗ ਬࢎೠ ҳઑ੄ ֎౟ਕ௼
    • RNN਷ ߈ࠂ ҳઑ۽ ੋ೧ ӝর۱ਸ(memory) о૑ѱ غ঻਺
    • दр୷ਵ۽ ಽযࠁݶ, ৈ۞ѐ੄ ֎౟ਕ௼о ੓Ҋ п੗о ੗न੄ ׮਺
    ֎౟ਕ௼ী ݫࣁ૑ܳ ࠁղחѪҗ э਷ ਗܻ
    • োࣘػ ؘ੉ఠ৬ Ө਷ োҙ
    - ঱য ݽ؛, ߣ৉, ਺ࢿੋध ١١
    RNNҗ ੉ܳ दр୷ਵ۽ ಽ੉ೠ Ӓܿ

    View full-size slide

  61. LSTM (Long Short-Term Memory)
    • RNN਷ ୭Ӕ ݻѐ ױয ੿ب੄ ӝর۱ ੿ب݅ਸ о૗
    • LSTM: ؊਌ ӟ ӝর۱ਸ о૑ӝ ਤೠ ҳઑ

    View full-size slide

  62. LSTM (Long Short-Term Memory)
    • Cell State: п ױ҅݃׮ ই઱ ੸਷ ࣻ੿ਸ Ѣ஖ݴ ੿ࠁܳ য়ۖزউ
    ࠁઓೠ׮.

    View full-size slide

  63. LSTM (Long Short-Term Memory)
    • Forget gate: ੉੹ cell state઺ী ࠛ೙ਃೠ ੿ࠁܳ ઁѢ
    • Input gate: ࢜۽਍ ੿ࠁܳ cell stateী ӝর

    View full-size slide

  64. LSTM (Long Short-Term Memory)
    • Forget gate৬ input gate੄ ઑ೤ਵ۽ cell state ܳ সؘ੉౟
    • Output: cell state੄ ੌࠗ࠙ਵ۽ чਸ ୹۱

    View full-size slide

  65. GRU (Gated Recurrent Unit)
    • LSTMҗ Ѣ੄ ਬࢎೠ ҳઑ, ؊ рױೠ ҳഅ
    - Forget gate ৬ input gate ܳ ೞա۽ ೤ஜ

    View full-size slide

  66. Attention Mechanism
    • ੋр੄ ੋ૑җ੿ਸ ࢤп೧ࠁݶ, ݽٚ ࣽрী ݽٚࠗ࠙ী न҃ਸ ॳח
    Ѫ੉ ইפۄ, ౠ੿ ࣽрী ౠ੿ ࢎޛ੉ա ઱ઁী ૘઺ೞח ݽणਸ ࠁ

    • RNNэ਷ ݽ؛੄ ӡ੉о ӡয૕ࣻ۾ ೞա੄ Hidden Stateী ݽٚ
    ղਊਸ ӝরೞӝо য۰ਕ૗
    • Hidden stateٜਸ ੷੢ೞҊ ੓׮о, ੉੹੄ stateٜਸ ഝਊ

    View full-size slide

  67. ؊ ࠂ੟ೠ ֎౟ਕ௼

    View full-size slide

  68. Dynamic Memory Networks
    • যڃ Ӗਸ ੍Ҋ, Ӓ Ӗী ؀ೠ ޛ਺ী ׹ਸ ೞחѪ
    - ࣻמद೷ ঱য৔৉җ ࠺तೣ
    - औ૑ ঋ਷ җઁ੉׮!

    View full-size slide

  69. Dynamic Memory Networks
    • ӝמ߹۽ RNNਸ ઑ೤
    • End-to-End ೟ण
    • ֤ޙਸ ੍੗!
    - https://arxiv.org/abs/1506.07285

    View full-size slide

  70. Dynamic Memory Networks
    • Input Module
    - RNN (GRU), п ޙ੢ٜ੄ hidden stateܳ ੷੢
    • Question Module
    - рױೠ RNN (GRU)
    • Episodic Memory Module
    - ੉ঠӝী ؀ೠ ੿ࠁܳ ҙ੢ೞח ݽٕ
    - ৈ۞ க੄ RNN (GRU) ۽ ҳࢿ

    View full-size slide

  71. Dynamic Memory Networks
    • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!

    View full-size slide

  72. Dynamic Memory Networks
    • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!

    View full-size slide

  73. Dynamic Memory Networks

    View full-size slide

  74. Dynamic Memory Networks

    View full-size slide

  75. Dynamic Memory Networks

    View full-size slide

  76. Dynamic Memory Networks
    • Attention + Memoryܳ ా೧ ૕ޙী ׹ೞח מ۱ਸ ഛ੢
    • ӝמ߹۽ RNNਵ۽ ੉ܖয૓ ߹ب੄ Module, ੑ۱ ഋకо ׮ܲ ݽ
    ٕ۽ ߸҃ೞחѪب оמೣ
    • ੉޷૑ ੑ۱ ݽٕҗ Ѿ೤غݶ ֥ۄ਍ Ѿҗܳ ࠁৈષ

    View full-size slide

  77. NMT (Neural Machine Translation)
    • ࠂ੟ೠ ֎౟ਕ௼ҳઑܳ ഝਊೠ ੗زߣ৉ ֎౟ਕ௼

    View full-size slide

  78. Google NMT
    • https://translate.google.com
    • https://research.googleblog.com/2016/09/a-neural-network-for-
    machine.html

    View full-size slide

  79. ٩۞׬ ੸ਊী ؀ೠ ࢤп
    • ٩۞׬ = ই੉٣য + ؘ੉ఠ + ݽ؛ ೟ण
    - ݽ؛ ೟ण - ੼੼ ੷۴೧૗
    - ই੉٣য - ੼੼ ؊ ݆਷ ࢎۈٜ੉ ઙࢎೞݶࢲ ই੉٣যب ݆ই૕ Ѫ
    - ؘ੉ఠ - ݠन۞׬, ٩۞׬ द؀ীࢲ о੢ ൞ࣗೠ ੗ਗ
    • ؀ӏݽ ؘ੉ఠܳ ഝਊೠ ݠन۞׬, ؊ ࠂ੟ೠ ઑ೤੄ ݽ؛, ࠙ঠ߹ ௼
    ۽झ ݽ؛ ١ ই૒ ߊ੹ оמࢿ੉ ޖҾޖ૓ೣ
    • ҕࠗೞҊ োҳ೤द׮!

    View full-size slide

  80. More..?
    • Join VCNC and join #study_ml
    ੷൞ח ঱ઁա ࠺౟ਦ ࢲ࠺झܳ ೣԋ ٜ݅ݴ ӝࣿ੸ੋ ޙઁܳ ೣԋ ಽযաт מ۱੓ח ѐߊ੗ܳ ݽदҊ ੓णפ׮.
    ঱ઁٚ ࠗ׸হ੉ [email protected]۽ ੉ݫੌਸ ઱दӝ ߄ۉפ׮!

    View full-size slide