Share
Embed
Deep NLP: 딥러닝을 이용한 자연어처리
머신러닝, 딥러닝 2학기 수업분량을 2시간으로 요약
VCNC 개발팀 워크샵 (2017/12/20)
VCNC
December 20, 2017
Transcript
82 slides
-
Deep NLP ٩۞ਸ ਊೠ োযܻ ѐߊܳ ਤೠ ӝࣿ ইਓۄੋҗ ઁ
बਵ۽ ӣ࢚ @ VCNC [email protected] -
Why?? • ܻо ೞҊ ח ݽ߄ੌ জ ࠺ૉפझח അо
• 5֙, 10֙ ٍীח যڄө? • ݠन۞ ܻ ࢤഝী ߸ചܳ оઉৢ ࣻ ח ӝࣿ ೞա • 100֙ оח ӝস غ۰ݶ ഥࢎب, ࢎۈب ߸ न ਃ • ٜ प۱җ ҃ ਵ۽ ޤٚ ٮۄইࢲ ੜ ೡ ࣻ ѷ݅, ܻ ળ࠺ೞҊ ҙब оݶ જ ਸ Ѫ -
ݠन۞ • ஹೊఠ җ ೠ ࠙ঠ۽, যڃ ۽ਸ ۽Ӓې߁ ೞ
ঋҊب ஹೊ ఠо সա धਸ ߓ ࣻ ѱ ೞח ࠙ঠ • ࠗ࠙ Data-driven prediction decision ਸ ࣻ೯ೞח ঌҊ ્ܻٜ • ؘఠ ࠙ࢳ ࠙ঠীࢲب օܻ ॳחؘ, ࠂೠ ݽ؛ਸ ࠙ࢳоо ٜ݅ ঋҊ ؘఠܳ ా೧ णदெ যځೠ ࢎपਸ ஏೞѢա, ऀѹ ੋࢎܳ ঌইղח ੌ ١ਸ ࣻ೯ -
٩۞ (Deep Learning) • ݠन۞ ೞਤ ࠙ঠ ೞա, ౠ
Neural Network (ੋҕ न҃ ݎ) ҅ৌ ঌҊ્ܻҗ োҙ • Deep Neural Network ۄҊ ࠗܰӝب ೣ • ୭Ӕ ߊਵ۽ ҙबਸ ߉Ҋ ח ࠙ঠ۽, ӝઓ ݠन۞ ߑߨۿ ೠ҅ܳ ӓࠂೞח -
٩۞ (Deep Learning) • Rawೠ input (Ӗ, ܻࣗ, ױ য
١١) ਸ ੑ۱߉ • рр ৈ۞ ҅கਵ۽ ؘఠ ٜਸ ઑೞৈ അ • ҅கਸ ऺই ৢ۰оݴ ࢚ ਤ ѐ֛ਸ അೡ ࣻ ѱ ؽ -
ӝઓ ݠन۞ ߑߨٜ ೠ҅ • അ ݠन۞ ߑߨٜ ࠗ࠙ Data-Driven
ߑߨٜ • যڃ ੌਸ द೯ೞח ۽ਸ ۽Ӓې߁ೞחѪ ইפۄ, णೡ ࣻ ח ҳઑܳ ٜ݅য֬Ҋ ؘఠܳ ઁҕ, ஹೊఠо ঌইࢲ ण • যڃ ؘఠܳ, যڃ ҳઑ۽ णदఃוջо ਃ! • ౠ ؘఠ ࠗ࠙ ਃࢿ Ҋ ؘఠܳ যڌѱ оҕ೧ࢲ ֍וջ о ݽ؛ ࢿמਸ ઝೣ -> Feature Engineering -
Feature Engineering • ؘఠܳ оҕ೧ࢲ ஹೊఠܳ णदఆ featureܳ ݅٘۰ݶ, ೧
࠙ ঠী ೠ ޙੋ ध (domain knowledge) ਃೞݴ दрҗ ࠺ਊ ݆ ٘ח স • ਕ۽ - Feature engineering -> Machine training -> Evaluate result -> Idea -> Feature engineering -> … -
٩۞ ޤо ܲо? • ٩۞ (Deep Neural Network) • അ۱
જ (ࠂೠ ղਊٜਸ णೡ ࣻ ח) ֎ਕܳ ৈ۞க ऺইࢲ, ֎ਕо featureٜਸ झझ۽ ٜ݅যղҊ णೡ ࣻ ب ۾ ೞ! -
যܳ ਤೠ ٩۞ • ٩۞ ݽ؛ਸ ഝਊೠ ࢿੋध • ӝઓ
ࢿੋध ߑߨ ࠺ ࢿמ ࠺ডਵ۽ ೱ࢚ؽ -
ܳ ਤೠ ٩۞ • ୭Ӕ ٩۞ਸ ഝਊ೧ о ݆ োҳо
غҊ ח ࠙ঠ • ImageNet ഥীࢲ જ ࢿਸ ղݶࢲ ݆ ҙबਸ -
োয ܻীࢲ ٩۞ ਊ • ఫझ х࠙ࢳ (ӛ/ࠗ) • ੋҕמ
࠺ࢲ • • ز ߣ -
٩۞ - х࠙ࢳ • ߑߨ - ৈ۞ ױযٜ ӛ/ࠗ
хਸ оח DBҳ୷ - ޙী ೧ ױযٜ ݃ա ١ೞח ӝ - о - о হ - “” ۄח ױয݅ਵ۽ח ౸ױೡ ࣻ হ • ٩۞ਸ ഝਊೠ ߑߨ - ޙߨա ੋ Ѫө Ҋ۰ೞৈ ౸߹ оמ -
٩۞ - • ߑߨ - ݆ࣻ ޙ-߸ DBܳ
ҳ୷ೞҊ, ࢜۽ ٜযয়ח ޙਸ ࣇ ীࢲ • ٩۞ਸ ഝਊೠ ߑߨ - ৈ۞ ޙী ೠ vector ഋधਵ۽ ݽ؛ী ֣ইٜয -
٩۞ - زߣ • ߑߨ - ই ࠂೠ ৈ۞
҅க ߣ ݽ؛ • ٩۞ਸ ഝਊೠ ߑߨ - ߣೡ ޙਸ vector۽ അೞҊ, ܳ ߸ജೞח ݽ؛ਸ णदఇ -
ݠन۞ (Machine Learning) • “The field of study that gives
computers the ability to learn without being explicitly programmed.” - Arthur Samuel • “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” - Tom Mitchell • : ߄قѱਸ فח ஹೊఠ - E = ߄قѱਸ فח ҃ - T = ߄قѱਸ فח ೯ਤ - P = ஹೊఠо ౸ ߄قਸ ӡ ഛܫ -
ݠन۞ (Machine Learning) • Supervised Learning (ب ण) - Regression
- োࣘੋ чਸ ஏ • : ࠗز оѺ ஏ - Classification - ࠛোࣘੋ ч ஏ • : ࢎਸࠁҊ ࢎޛ ݏ୶ӝ - Regressionޙઁب Classification ޙઁ۽ ߄Չࣻب ! • : ࠗز оѺਸ ࠁҊ উ Ѿ • Unsupervised Learning (࠺ب ण) - Clustering • Reinforcement Learning (ъച ण) - ѥחߨਸ झझ۽ ߓח ۽ࠈ -
Linear Regression • և ؘఠ۽ ࠗز оѺਸ ஏ೧ ࠁ
• 30 ಣ = 3র 5000݅ਗ • 50ಣ = 5র 5000݅ਗ • 10ಣ = 1র 5000݅ਗ • 20ಣ = ? -
Linear Regression • y = a * x + b
- x: և(ಣ) - y: оѺ - h(x) ৬ э അೞӝب ೣ (hypothesis, оࢸ) - ৬ э അ • Training dataܳ learning algorithmী ਊ, h ܳ ҳೣ! • h ܳ ਊೞৈ x ী ೠ Ѿҗч (y) ܳ ஏ оמ -
Cost Function • যڃ о hܳ ࣁਛਸٸ, Ӓ о ݃ա
ౣ۷חо(error) ܳ ࠺ਊ (cost)۽ рೞৈ ࣻधਸ ࣁ • ۧѱ ೠ Costܳ ୭ࣗചೞח ߑೱਵ۽ ۄݫఠ a,bٜਸ ઑ ೞݶ ખ؊ ৢ߄ܲ о h ܳ о ࣻ ! -
Linear Regression(ױੌ ߸ࣻ) Cost Function • पઁ ؘఠח h(x) =
1*x ੋؘ, h(x) = 0.5*x ۽ ஏೠ ҃ • য়ରч (Error, Cost): 0 1 2 3 3 2 1 0 -
Linear Regression(ױੌ ߸ࣻ) Cost Function • , ী ೧ࢲח ইې৬
э ੑੋ ݽন ػ • যڌѱ ୭ਸ ਸѪੋо? -
Gradient Descent • য়ରчਸ যڌѱ ୭ࣗചೡѪੋо? - Cost functionਸ ୭ࣗച
- Cost function ١Ҋࢶ Ҏӝ ӝ • ীࢲ दೞৈ, ӝӝо ծইח ଃਵ۽ ҅ࣘ زೞ • ӝӝ = Cost function ࠙ч -
Linear Regression Gradient Descent • Iterationਸ ా೧ ୭ਸ ইх (for
fixed , this is a function of x) (function of the parameters ) -
Linear Regression Gradient Descent (for fixed , this is a
function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) -
Linear Regression Gradient Descent (for fixed , this is a
function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) -
Multi Variable Linear Regression • ࠗز оѺਸ ஏೞחؘח ಣࣻ ݈Ҋب
নೠ ܐܳ ଵҊೡ ࣻ ਸѪ - ߑ іࣻ, ળҕ֙ب, ߓۆ іࣻ ١١.. • ݽ؛ ೦ध ഋకо ؽ • Feature scaling, feature ઑ, ࠂ feature ઁѢ ١ ݆ স ਃೞѱ ؽ -
Classification: Logistic Regression • োࣘੋ чਸ ஏೞחѪ ইצ, ࠛোࣘੋ чਸ
ஏ - झಅੋ ইצ - ঐੋ ইצ - ࢎ Ҋনੋ, ъইੋ Tumor Size Tumor Size (Yes) 1 (No) 0 -
Classification: Logistic Regression • Sigmoid (Logistic function) ਸ ࢎਊೞৈ և
ߧਤ чਸ, 0~1 ࢎ чਵ۽ ߸ജ • ݅ড 0.5ܳ ӝળਵ۽ ೠݶ, 0.5 ࢚:1, 0.5 ݅: 0 -
Logistic Cost Function • y = 0 ੌٸ৬ y =
1 ੌٸо ܴ • Gradient Descentܳ ഝਊೞח ߑߨ Linear Regressionҗ زੌ! -
Regularization • Overfitting (җ୭ച) ޙઁ - അपࣁ҅ पઁ ݽ؛ࠁ, णदఅ
ݽ؛ ࠂبо ؊ ֫ইઉࢲ য णࣇী݅ җبೞѱ ୭ചػ ࢚ട - training set ইצ ࢜۽ ޙઁܳ ઁ۽ ஏೡ ࣻ হѱ ػ • Feature іࣻܳ Ѣա, Regularizationਵ۽ ೧Ѿ • Regularization - Cost functionী بਵ۽ ࢚ࣻ೦ਸ ୶оदெ णਸ ߑ೧ೣ - җبೞѱ णغחѪਸ ߑೞח ബҗ -
Neural Network • ੋр ֱ ҳઑܳ ٮۄೠ ҳઑ • Dendrite:
ܲ ۠ਵ۽ࠗఠ नഐܳ ߉ • Cell body: ݽٚ ੑ۱ਸ ઙೣ • Axon: ੑ۱ ҅ਸ ֈਵݶ, ܲ ۠ਵ۽ नഐܳ ࠁն • Synapses: ܲ ۠җ োѾؽ. োѾ ъبী ٮۄ नഐ ࣁӝо ߄Պ - नഐо ъ೧Ѣա, ড೧חѪ = ण -
Neural Network • ੋр ֱ ҳઑܳ ٮۄೠ ҳઑ • Dendrite:
ܲ ۠ਵ۽ࠗఠ नഐܳ ߉ • Cell body: ݽٚ ੑ۱ਸ ઙೣ • Axon: ੑ۱ ҅ਸ ֈਵݶ, ܲ ۠ਵ۽ नഐܳ ࠁն • Synapses: ܲ ۠җ োѾؽ. োѾ ъبী ٮۄ नഐ ࣁӝо ߄Պ - नഐо ъ೧Ѣա, ড೧חѪ = ण -
Neural Network Playground • http://playground.tensorflow.org/ • 0 hidden layer, 2
neurons (Logistic Regression) - ױࣽೠ 1ର ഋక ؘఠ߆ী णदః ޅೣ • 2 hidden layer, 2 neurons - Ѿҗо যڌѱ ܲо? - ৈ۞க ऺইب Ѿҗо ࠺तೣ • neuronіࣻܳ טܻݶ ખ؊ ࠂೠ ഋకܳ അೡ ࣻ • Featureܳ טܻݶ? • Learning rate ݶ? • Deep ೠ ֎ਖীࢲ activation function? • Regularization -
Neural Network - Cost Function Layer 1 Layer 2 Layer
3 Layer 4 Logistic regression: Neural network: • Logistic Regression ഛ౸! -
Neural Network - Back Propagation • Forward propagation Layer 1
Layer 2 Layer 3 Layer 4 -
Neural Network - Back Propagation • Back propagation Layer 1
Layer 2 Layer 3 Layer 4 Intuition: “error” of node in layer . For each output unit (layer L = 4) -
Neural Network - ܻ • ֎ਕ ҳઑܳ ೠ - Input
feature іࣻ, Output class іࣻ, hidden layer іࣻ ߂ ରਗ • weightܳ ےؒೞѱ ୡӝച • forward propagation, cost function, back propagation ਸ ҳ അ • ݽٚ ؘఠٜী ೞৈ, back propagationਸ ా೧ ҳೠ gradient ٜਸ ݽ؛ী ਊ • TensorFlowо ঌইࢲ ೧ਃ -
(۽) ੋр ֱ vs. ۡ֎ਕ • ੋр ֱ - ড
ୌর ѐ ۠, ࣻߔ~ࣻୌઑ ѐ दշझ۽ ܖয - ੋрҗ زޛ ֱח ࢶ • ۡ֎ਕ - ੋҕੋ ۡ֎ਕ ࢎૉח ࢚ਵ۽ - ࢎૉо ࣻ۾ ण য۰ਕ • AlphaGo - ৈ۞ѐ ۡ ֎ਕ ઑ. (13 layer Convolutional neural network ١) - Input layer: 48ѐ 19x19 ߄ق౸ = 17328 ۠ - ࣻߔ݅ѐ ۠ -
RNN? • !?!?!? • RNN ޤӡې, ۠Ѫਸ ೡࣻחо • RNNਸ
Ҿӓ ੋҕ न҃ ݎ ҳઑۄҊ ೞח ࢎ ۈٜب -
RNN (Recurrent Neural Networks) • ݆ NLP সী ই જ
ࢿמਸ ղષ • ѐ֛ য۵ӝب ೞҊ ઁ۽ ࢸݺೞҊ ח ܐо হ - ઁ۽ ೧ೞח ࢎۈب ੜ হחѪ э • য়טب ઁ۽ ࢸݺೡ ࣻ ਸ ഛप ݽܰѷ -
য ݽ؛ (Language Model) • োࣘػ ױযٜ ١ഛܫਸ ஏೞח ݽ؛
• ߓо Ҋࢲ աח ߏਸ ____ - ___ী ٜযт ױযח? - ݡ, ݡח, ,… - p(ݡ) = 0.25, p(ݡח) = 0.15, p() = 0.05, .. - p(زର) = 0.000001 • ޙ ࣽࢲա ױযࢶఖ ١ਸ ஏೡ ࣻ ӝী, زߣ ١ ݆ NLP Taskী ਬਊೞ -
য ݽ؛ (Language Model) • ഛܫ য ݽ؛ਸ ݅٘۰ݶ, ী
աয়ח ױযܳ ݆ ଵઑೡࣻ۾ ഛೞ! - 2-gram, 3-gram, 4-gram, .. n-gram • ߓо Ҋࢲ աח ߏਸ ____ - ߏਸ ____ <-> ߓо Ҋࢲ աח ߏਸ ____ • n-gramਸ טܾࣻ۾ ݫݽܻо ষաѱ ਃೞ! - ޙઁܳ ೧Ѿ೧חѪ RNN(Recurrent Neural Networks) -
RNN (Recurrent Neural Networks) • दрী ٮۄ ( ؘఠ ೯ী
ٮۄ) ҅ࣘ সؘغח ࢚కчਸ рೞҊ ח ֎ਕ • ױࣽೠ ࢚కо ইפۄ, णਸ ࣘೡࣻ۾ ੑ۱ਸ ܻೞח ҭ ࠂೠ ۽ ֣ ইٜѱؽ • ࢜۽ աৢ ױযܳ ஏೞחؘী ݽٚ ױযٜ ࠁܳ ଵઑೞѱ ؽ -
RNN (Recurrent Neural Networks) • ߓо Ҋࢲ աח ߏਸ ____
- 4ѐ ױযܳ ଵઑ - RNNীࢲח 4 layer neural network ࢤӣ -
RNN (Recurrent Neural Networks) • RNNਸ णदఃחѪ য۵! • Vanishing
Gradient ޙઁ - ցޖ য়ې ࠁө ଵઑೞ۰ࠁפ, য٣ࢲ ੜޅ೮ח णೞӝ য۵ѱ ؽ • Exploding Gradient ޙઁ • ReLU / Clipping ١ ৈ۞о పץਵ۽ ӓࠂ! -
RNN (Recurrent Neural Networks) • ؊ ࣁೠ णܐ - http://www.wildml.com/2015/09/recurrent-neural-networks-
tutorial-part-1-introduction-to-rnns/ -
LSTM (Long Short-Term Memory) • RNNҗ ਬࢎೠ ҳઑ ֎ਕ •
RNN ߈ࠂ ҳઑ۽ ੋ೧ ӝর۱ਸ(memory) оѱ غ • दр୷ਵ۽ ಽযࠁݶ, ৈ۞ѐ ֎ਕо Ҋ по न ֎ਕী ݫࣁܳ ࠁղחѪҗ э ਗܻ • োࣘػ ؘఠ৬ Ө োҙ - য ݽ؛, ߣ, ࢿੋध ١١ RNNҗ ܳ दр୷ਵ۽ ಽೠ Ӓܿ -
LSTM (Long Short-Term Memory) • RNN ୭Ӕ ݻѐ ױয ب
ӝর۱ ب݅ਸ о • LSTM: ؊ ӟ ӝর۱ਸ оӝ ਤೠ ҳઑ -
LSTM (Long Short-Term Memory) • Cell State: п ױ҅݃ ই
ࣻਸ Ѣݴ ࠁܳ য়ۖزউ ࠁઓೠ. -
LSTM (Long Short-Term Memory) • Forget gate: cell stateী
ࠛਃೠ ࠁܳ ઁѢ • Input gate: ࢜۽ ࠁܳ cell stateী ӝর -
LSTM (Long Short-Term Memory) • Forget gate৬ input gate ઑਵ۽
cell state ܳ সؘ • Output: cell state ੌࠗ࠙ਵ۽ чਸ ۱ -
GRU (Gated Recurrent Unit) • LSTMҗ Ѣ ਬࢎೠ ҳઑ, ؊
рױೠ ҳഅ - Forget gate ৬ input gate ܳ ೞա۽ ஜ -
Attention Mechanism • ੋр ੋҗਸ ࢤп೧ࠁݶ, ݽٚ ࣽрী ݽٚࠗ࠙ী न҃ਸ
ॳח Ѫ ইפۄ, ౠ ࣽрী ౠ ࢎޛա ઁী ೞח ݽणਸ ࠁ • RNNэ ݽ؛ ӡо ӡযࣻ۾ ೞա Hidden Stateী ݽٚ ղਊਸ ӝরೞӝо য۰ਕ • Hidden stateٜਸ ೞҊ о, stateٜਸ ഝਊ -
Dynamic Memory Networks • যڃ Ӗਸ ੍Ҋ, Ӓ Ӗী ೠ
ޛী ਸ ೞחѪ - ࣻמद যҗ ࠺तೣ - औ ঋ җઁ! -
Dynamic Memory Networks • ӝמ߹۽ RNNਸ ઑ • End-to-End ण
• ֤ޙਸ ੍! - https://arxiv.org/abs/1506.07285 -
Dynamic Memory Networks • Input Module - RNN (GRU), п
ޙٜ hidden stateܳ • Question Module - рױೠ RNN (GRU) • Episodic Memory Module - ঠӝী ೠ ࠁܳ ҙೞח ݽٕ - ৈ۞ க RNN (GRU) ۽ ҳࢿ -
Dynamic Memory Networks • Attention + Memoryܳ ా೧ ޙী ೞח
מ۱ਸ ഛ • ӝמ߹۽ RNNਵ۽ ܖয ߹ب Module, ੑ۱ ഋకо ܲ ݽ ٕ۽ ߸҃ೞחѪب оמೣ • ੑ۱ ݽٕҗ Ѿغݶ ֥ۄ Ѿҗܳ ࠁৈષ -
٩۞ ਊী ೠ ࢤп • ٩۞ = ই٣য + ؘఠ
+ ݽ؛ ण - ݽ؛ ण - ۴೧ - ই٣য - ؊ ݆ ࢎۈٜ ઙࢎೞݶࢲ ই٣যب ݆ই Ѫ - ؘఠ - ݠन۞, ٩۞ दীࢲ о ൞ࣗೠ ਗ • ӏݽ ؘఠܳ ഝਊೠ ݠन۞, ؊ ࠂೠ ઑ ݽ؛, ࠙ঠ߹ ۽झ ݽ؛ ١ ই ߊ оמࢿ ޖҾޖೣ • ҕࠗೞҊ োҳद! -
More..? • Join VCNC and join #study_ml ൞ח ઁա ࠺ਦ
ࢲ࠺झܳ ೣԋ ٜ݅ݴ ӝࣿੋ ޙઁܳ ೣԋ ಽযաт מ۱ח ѐߊܳ ݽदҊ णפ. ઁٚ ࠗহ [email protected]۽ ݫੌਸ दӝ ߄ۉפ!