Distillation Overview and Recent Research ߔ৔޹ (ML Engineer, Pingpong)

ݾର ݾର 1. Overview 1. What is Distillation? 2. Distilling the Knowledge in a Neural Network (Hinton et al., 2015) 2. Recent Research 1. Transformer to CNN:Label-scarce distillation for efficient text classification (Chia et al., 2018 NIPS Workshop) 2. BAM!:Born-Again Multi-Task Networks for Natural Language Understanding (Clark et al., 2019 arXiv) 3. Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation(Turc et al., 2019 arXiv) 4. Patient Knowledge Distillation for BERT Model Compression (Sun et al., 2019 EMNLP)

Overview “Distillation” Overview

What is Distillation? Overview

Overview What is Distillation? Teacher Model Many parameter - Large Model Well-trained Student Model Fewer parameter - Small Model Knowledge

Overview • Original Classification • One-hot label: ੿׹ classী 1੄ labelਸ ࠗৈ Ã “Hard target” • Loss: Cross-Entropy • Distillation • Continuous label: Model੄ Outputਸ label۽ ੉ਊ Ã “Soft target” • Loss: Cross-Entropy, K-L Divergence, MSE One-hot vs Continuous 0 0.175 0.35 0.525 0.7 Class 1 Class 2 Class 3 Class 4 0.1 0.05 0.7 0.15 Probability 0 0.25 0.5 0.75 1 Class 1 Class 2 Class 3 Class 4 0 0 1 0 Probability

Overview Dataset • Labelled Dataset • ࢎۈ੉ ૒੽ labelling à ݆਷ ন ഛࠁ x • Supervised Learningਸ ਤ೧ࢲ ٜ݅য૓ (input, label) हਵ۽ ҳ ࢿػ ؘ੉ఠࣇ • Unlabelled Dataset • Labelling੉ غয ੓૑ ঋ਷ ؘ੉ఠࣇ ex) Pingpong corpus • Unsupervised Learningਸ ਤ೧ࢲ ੉ਊؽ ex) Word2Vec, Autoencoder, BERT pre-training ١ • ࣻ૘ೞӝ ए਑ Model • Teacher Model • ૑धਸ ੹ࣻೞӝ ਤೠ ݽ؛ (о੢ ੿ഛೠ ݽ؛) • ੌ߈੸ਵ۽ ݆਷ ౵ۄݫఠܳ о૑Ҋ, о੢ ࢿמ੉ જ਷ ҳઑ੄ ݽ؛ ਸ ੉ਊೣ (ঔ࢚࠶ب ੗઱ ੉ਊೣ) • Student Model • ૑धਸ ੹ࣻ߉ӝ ਤೠ ݽ؛ • ੌ߈੸ਵ۽ ࢲࡂ੉ оמೠ(memory, latency)ܳ о૓ ݽ؛ਸ ੉ਊ ೣ (੸਷ ౵ۄݫఠ ࣻ ߂ ߽۳ചо ੜ غয ࣘبо ࡅܲ ҳઑ - CNN) à ࢎप memory/latencyח ࢚؀੸੐ (severীࢲ ࡅܰ׮Ҋ mobileীࢲ ࡅܰ૓ ঋ਺) Main Concept

Overview Unlabelled Data ࣻ૘੉ए਑ ন੉݆਺ 8JLJ (PPHMF*NBHF١ Labelled Data ࣻ૘੉য۰਑ ন੉੸਺ /-* 454١ Main Concept Machine Learning Data driven approach

Overview Unlabelled Data ࣻ૘੉ए਑ ন੉݆਺ 8JLJ (PPHMF*NBHF١ Labelled Data ࣻ૘੉য۰਑ ন੉੸਺ /-* 454١ Transfer Data 6OMBCFMMFE$PSQVT۽ࠗఠ୶୹ ন੉࢚؀੸ਵ۽݆਺ -BCFMFE৬%JTUSJCVUJPO੉׮ܳࣻ੓਺ ؼࣻ੓ਵݶ࠺तೠ%JTUSJCVUJPOਵ۽ Teacher Model Main Concept

Overview Teacher Training ݆਷౵ۄݫఠ MBUFODZNFNPSZन҃Y о੢ࢿמ੉જ਷ 405" ݽ؛ 0OFIPUMBCFM۽೟ण Make Transfer Data 5FBDIFSݽ؛۽ࠗఠࢤࢿ ੿ഛೞ૑חঋ਺ /-*١਷ٜ݅ӝয۰਑ Student Training ੸਷౵ۄݫఠ ੸੺ೠNFNPSZMBUFODZ 5FBDIFS੄0VUQVUਸ੉ਊೞৈ೟ण Labeled Data Unlabeled Data Transfer Data Process

Distilling the Knowledge in a Neural Network Hinton et al., 2015 Overview

Overview Distilling the Knowledge in a Neural Network • Neural Networkী ੓যࢲ Distillationਸ ୊਺ ࣗѐೣ • Neural Networkח ੿׹ Class੄ ഛܫਸ о੢ ௼ѱ ࠗৈೞب۾ ೟ण • ੿׹੉ ইצ Classীب ޷ࣁೠ ഛܫਸ ࠗৈ • ؀ࠗ࠙ ݒ਋ ੘਷ ഛܫ੉૑݅ ݻݻ਷ ׮ܲ Ѫٜ ࠁ׮ ௼׮. • ࢚؀੸ੋ ഛܫ ࠙ನ Ã “Model੉ ੌ߈ചೞח ҃ೱࢿ”

Overview Distilling the Knowledge in a Neural Network • Neural Networkী ੓যࢲ Distillationਸ ୊਺ ࣗѐೣ • Neural Networkח ੿׹ Class੄ ഛܫਸ о੢ ௼ѱ ࠗৈೞب۾ ೟ण • ੿׹੉ ইצ Classীب ޷ࣁೠ ഛܫਸ ࠗৈ • ؀ࠗ࠙ ݒ਋ ੘਷ ഛܫ੉૑݅ ݻݻ਷ ׮ܲ Ѫٜ ࠁ׮ ௼׮. • ࢚؀੸ੋ ഛܫ ࠙ನ Ã “Model੉ ੌ߈ചೞח ҃ೱࢿ” 0 20 40 60 80 Car Garbage Truck Bus Carrot 0.1 4 7 75 Probability Classifier Knowledge!

Overview Distilling Method H(p, q) = − ∑ x p(x)log q(x) Label(After Softmax) Model Output(After Softmax) Cross-Entropy Mean Squared Error “Teacher੄ Output”ਸ “Label”۽ ೟ण! H(p, q) = ∑ i (pi − qi )2 Label(Logit) Model Output(Logit) “Teacher੄ Output” + “True label”ਸ زदী ੉ਊೞӝب ೣ

Overview Main Idea • Temperature Term • ੜ ೟णػ ݽ؛ à ੿׹ী “High Confidence(≈ 1)” • Soft target੉ ੄޷о হਸ ࣻ ੓਺ • Softmax function ੉੹ী Smoothing à ੸׼ೠ “Soft target” H(p, q) = − ∑ x p(x)log q(x) Label Model Output

Overview Main Idea • Temperature Term • ੜ ೟णػ ݽ؛ à ੿׹ী “High Confidence(≈ 1)” • Soft target੉ ੄޷о হਸ ࣻ ੓਺ • Softmax function ੉੹ী Smoothing à ੸׼ೠ “Soft target” qi = exp(zi ) ∑ j exp(zj ) qi = exp(zi /T) ∑ j exp(zj /T) T : Temperature 0 0.175 0.35 0.525 0.7 Class 1 Class 2 Class 3 Class 4 0.1 0.05 0.7 0.15 Probability 0 0.225 0.45 0.675 0.9 Class 1 Class 2 Class 3 Class 4 0.01 0.02 0.9 0.07 Probability H(p, q) = − ∑ x p(x)log q(x) Label Model Output

Overview Experiments Meaningful Experiment • MNIST • 0~9੄ Ӓܿ Ã ं੗ ৘ஏ • Experiment • ੿׹੉ “3”ੋ exampleਸ ೟ण ࣇীࢲ ઁ৻ • “3”਷ “2”ա “8” ١੄ Distillation labelীࢲ݅ ١੢ • Test Accuracy: 877/1010 = 86%

Recent Research Distillation on “NLP”! Recent Research

Transformer to CNN: Label-scarce distillation for efficient text classification Chia et al., 2018 NIPS workshop Recent Research

Recent Research Main Idea • Transformer(GPT) Ã Simple CNN

Recent Research Experiments Environment • 3-Architecture: Bi-LSTM, KimCNN(Char-CNN), BlendCNN Text Classification Task AG News(4 classes), DBpedia(10 classes), Yahoo Answers(10 classes) ~x300 speed up!!

Result Recent Research Environment • 3-Architecture: Bi-LSTM, KimCNN(Char-CNN), BlendCNN Text Classification Task AG News(4 classes), DBpedia(10 classes), Yahoo Answers(10 classes) ~x300 speed up!! Labelled Data Transfer Data Experiments

Recent Research Main Contribution 
 BERT Distillation੄ оמࢿ ഛੋ ! (प೷੄ ֤ܻࢿ਷…)

BAM!:Born-Again Multi-Task Networks for Natural Language Understanding Clark et al., 2019 arXiv Recent Research

Recent Research Main Idea • Multi-Task Learning + Distillation

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence Loss Input Batch(Multiple Task, Input) Training Phase

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence(Task 1) Loss: Task 1 Loss Input Batch(Multiple Task, Input) Training Phase

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence(Task 2) Loss: Task 1 Loss + Task 2 Loss Input Batch(Multiple Task, Input) Training Phase

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence(Task 3) Loss: Task 1 Loss + Task 2 Loss + Task 3 Loss Input Batch(Multiple Task, Input) Training Phase

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence Loss: Task 1 Loss + Task 2 Loss + Task 3 Loss Input Batch(Multiple Task, Input) Optimize Training Phase

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier Inference Phase BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence Task 1 Classifier Task 1 Classifier Task 1 Classifier Input Batch(Multiple Task, Input) Task 1 Output Task 2 Output Task 3 Output

Recent Research Main Idea • Multi-Task Learning (with BERT) • ೞա੄ Model۽ Multi Task • ҕా੄ Encoder + п Task߹ Classifier • ੉੼ • Single Sentence à One-time Inference! • Intent, DA, Sentiment ١ “োҙࢿ” ੓ח Task • Training Robustness Inference Phase BERT Encoder Task 1 Classifier Task 2 Classifier Task 3 Classifier Input sentence Task 1 Classifier Task 1 Classifier Task 1 Classifier Input Batch(Multiple Task, Input) Task 1 Output Task 2 Output Task 3 Output

Recent Research Main Idea • Teacher Annealing Loss function e.g. cross entropy, MSE L(θ) = ∑ τ∈T ∑ (xi T ,yi T )∈DT l(fT (xi T , θT ), fT (xi T , θ))

Recent Research Main Idea • Teacher Annealing Task Task Dataset L(θ) = ∑ τ∈T ∑ (xi T ,yi T )∈DT l(fT (xi T , θT ), fT (xi T , θ))

L(θ) = ∑ τ∈T ∑ (xi T ,yi T )∈DT l(fT (xi T , θT ), fT (xi T , θ)) Recent Research Main Idea • Teacher Annealing Teacher Result(Label) Student Result Same Architecture

Recent Research Main Idea • Teacher Annealing L(θ) = ∑ τ∈T ∑ (xi T ,yi T )∈DT l(fT (xi T , θT ), fT (xi T , θ)) l(λyi T + (1 − λ)fT (xi T , θT ), fT (xi T , θ)) Teacher Annealing lambdaী ٮۄ True label: Teacher Output੄ ࠺ਯ Ѿ੿ Label True label੉ ೙ਃ! Ã Transfer Dataset ࢎਊࠛо

Recent Research Main Idea • Model & Training Parameter • Large LR :1e-4 • Task Weighted Sampling(Multi-task) • Layer-wise LR

Recent Research Experiments Environment • GLUE • NLU Benchmark • Experiment • Multi-task + Distillationਸ ׮নೞѱ ઑ೤ೞৈ प೷ ૓೯ • Ablation Studyܳ ా೧ Teacher Annealing੄ ബҗ ૐݺ

Recent Research Main Contribution 
 Multitask Learning + Teacher Annealing

Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation Turc et al., 2019 arXiv Recent Research

Recent Research Main Idea • Various Size of BERT Distillation (Google ౵ਕ..) • BERT Ã BERT Distillation • ׮নೠ Hidden Size, Layer depth۽ Distillation (24о૑ ݽ؛) Memory Speed

Recent Research Main Idea • BERTী ݏ୸ Distillation ೟ण ߑध 1. Student Model Pre-training(Initialization) 2. Distillation using Transfer Dataset 3. Fine-Tuning using Labelled Dataset

Recent Research Experiments Environment • Text Classification Task(Single, Pair) • SST, Book Review(Single), MNLI, RTE(Pair) • Experiment • 4о૑ Method۽ प೷ ૓೯ • #1:Basic Training: Pre-training হ੉ ߄۽ ೟ण • #2:Distillation: Distillationਵ۽݅ ೟ण • #3:Pre-training + Fine-Tuning: ੌ߈੸ੋ BERT ೟णߨ • #4:Pre-training + Distillation + (Fine-Tuning)

Recent Research Experiments Environment • Text Classification Task(Single, Pair) • SST, Book Review(Single), MNLI, RTE(Pair) • Experiment • 4о૑ Method۽ प೷ ૓೯ • #1:Basic Training: Pre-training হ੉ ߄۽ ೟ण • #2:Distillation: Distillationਵ۽݅ ೟ण • #3:Pre-training + Fine-Tuning: ੌ߈੸ੋ BERT ೟णߨ • #4:Pre-training + Distillation + (Fine-Tuning) ݽٚ Taskীࢲ #4о જ਷ ࢿמ

Recent Research Experiments Environment • Text Classification Task(Single, Pair) • SST, Book Review(Single), MNLI, RTE(Pair) • Experiment • 4о૑ Method۽ प೷ ૓೯ • #1:Basic Training: Pre-training হ੉ ߄۽ ೟ण • #2:Distillation: Distillationਵ۽݅ ೟ण • #3:Pre-training + Fine-Tuning: ੌ߈੸ੋ BERT ೟णߨ • #4:Pre-training + Distillation + (Fine-Tuning) - Layer ࣻী ٮܲ Ѿҗ - Hidden Sizeী ٮܲ Ѿҗ 8-layerө૑ח Hidden size << Layer ࣻ Intermediate layer = Hidden size * 4 Multi-head = Hidden size / 64

Recent Research Main Contribution 
 BERT Ã BERT Distillation Relation Between Model Size and Performance

Patient Knowledge Distillation for BERT Model Compression Sun et al., 2019 EMNLP Recent Research

Recent Research Main Idea • Patient Distillation • Output + Intermediate Layerܳ Teacherܳ ٮܰب۾ • Intermediate Layer੄ [CLS]ష௾ ੉ਊ • [CLS] ష௾਷ classifier ݆਷ ੿ࠁܳ ׸Ҋ ੓׮Ҋ о੿ • ױ, Teacher৬ Student੄ Hidden sizeо زੌ೧ঠೣ • ࣘب ஏݶী ੓যࢲח ௾ ઁডઑѤ੐ • Teacher੄ ೞਤ kѐ੄ layer۽ initialization

Recent Research Experiments Environment • Dataset • SST, MRPC, QQP, MNLI, QNLI. RTE • Experiment • 2о૑ ݽ؛(3-layer, 6-layer) ߂ 3о૑ ߑߨ(FT, KD, PKD)۽ ૓೯ • #1:FT(Fine-Tuning): ೞਤ k-layer۽ initialize റ Fine-tuning • #2:KD(Knowledge Distillation): ੌ߈੸ੋ KD(output݅ ੉ਊ) • #3:PKD(Patient Knowledge Distillation): Intermediate layer + output ੉ਊ

Recent Research Experiments Environment • Dataset • SST, MRPC, QQP, MNLI, QNLI. RTE • Experiment • 2о૑ ݽ؛(3-layer, 6-layer) ߂ 3о૑ ߑߨ(FT, KD, PKD)۽ ૓೯ • #1:FT(Fine-Tuning): ೞਤ k-layer۽ initialize റ Fine-tuning • #2:KD(Knowledge Distillation): ੌ߈੸ੋ KD(output݅ ੉ਊ) • #3:PKD(Patient Knowledge Distillation): Intermediate layer + output ੉ਊ ੉੹ ֤ޙҗ ࠺Ү೮ਸ ٸ, ੹୓੸ਵ۽ ઑӘঀ ծ਷ ࣻ஖

Recent Research Main Contribution 
 Patient Distillation ੉ۄח ࢜۽਍ Distillation Method੄ ઁद

хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽ োۅ ઱ࣁਃ! ߔ৔޹ (Machine Learning Engineer, Pingpong) Email.yeongmin.baek@scatterlab.co.kr Facebook. bym0313