Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
The Natural Language Decathlon: Multitask Learn...
Search
Scatter Lab Inc.
July 10, 2019
Research
950
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
The Natural Language Decathlon: Multitask Learning as Question Answering
Scatter Lab Inc.
July 10, 2019
Transcript
30 slides
스캐터랩(ScatterLab) ੌ࢚ച ੋҕמ Scatterlab ML Technical Seminar Session 2 (QA):
백영민 The Natural Language Decathlon: Multitask Learning as Question Answering McCann et al. Salesforce Research Machine Learning Engineer
#1. Concept
!3 Multitask Learning ৈ۞ о taskܳ э ण೧ࠁ!
• Method: ౠ objectiveܳ оҊ णػ model/representationਸ ܲ downstream taskী
ਊೞח Ѫ • ੌ߈ਵ۽ Language Modeling١ Natural Language ߈ੋ ౠࢿਸ णೡ ࣻ ח objective ਊ • : • Random initializeীࢲ दೞח Ѫ ࠁ જ Ѿҗܳ ࠁҊ, ࡅܲ ࣻ۴ਸ оמೞѱ ೧ષ • ߑध • Representation: Word2Vec, Glove ١ fixed representation, ULMFit, ELMO ١ intermediate representation(context aware)ਸ downstream taskীࢲ ਊೞח Ѫ(߹ب model ઓ) • Model: BERT, GPT١ pre-trainingী ਊ೮؍ modelਸ downstream taskীࢲ “fine-tuning” !4 #1 Concept Transfer Learning
• Method: ৈ۞ taskٜਸ ೞա ݽ؛۽ زदী णदఃח Ѫ •
Chunking, POS tagging, NER, SRL, dependency parsing, NLI ١ NLP taskٜਸ زੌೠ ݽ؛۽ زदী ण • : • ৈ۞ taskܳ زदী modelingೡ ࣻ • ੜ णغݶ ౠ taskী ೠػ Ѫ ইצ “General Representation”ਸ ਸ ࣻ • Zero-shot Learning, Meta-Learning ١ ਊ оמࢿ • ୭Ӕ ഝߊ োҳо ܖযҊ ח ࠙ঠ • ই singletask learningী ࠺Үؼ݅ೠ ࢿמਸ ࠁৈҊ ঋ݅, challengingೠ োҳٜ ݆ ܖযҊ . • Image classification + NLP • ࣗѐೡ ֤ޙ “MQAN” singletask learningী Ӓա݃ ࠺Үؼ݅ೠ ࢿמਸ ࠁ(BERT ֤ޙ ੑפ) !5 #1 Concept Multitask Learning
#2. Method
!7 Approach ݽٚ taskܳ QAഋधী ݏࠁ!
• য questionী ೠ ਸ “contextղীࢲ” ח ޙઁܳ ಿ (
Context ղࠗী Ҋ о) • Ex) SQuAD, RACE… • ࠁా द index৬ indexܳ ח ߑध !8 #2 Approach Question Answering
!9 #2 Approach Question Answering Idea: ݽٚ taskٜਸ “QAഋध”ਵ۽ ٜ݅যࢲ
“Multitask Learning”ਸ ೧ࠁ! QA Translation Summary NLI Sentiment Analysis
• ୨ 10ѐ taskܳ Multitask Learning !10 #2 Approach Decathlon
!11 Model Architecture Multitask Question Answering Network(MQAN)
!12 #2 Architecture Overview
!13 #2 Architecture I/O • Input: • Q: Question Sentences
• C: Context Sentences • A: Answer Sentences(for generation - autoregressive) • Output: • General QA: Contextীࢲ द, indexܳ • Q(Question Sentence) + C(Context Sentences) + Outer Vocabulary(Generation) ী ࢶఖ
!14 #2 Architecture Feature - Input Representation
!15 #2 Architecture Feature - Alignment <dummy dataܳ ֍ח ਬ>
!16 #2 Architecture Feature - Dual Coattention
!17 #2 Architecture Feature - Compression & Self-Attention
!18 #2 Architecture Feature - Answer Representation
!19 #2 Architecture Feature - Answer Representation
!20 #2 Architecture Feature - Answer
!21 Training Strategy Curriculum learning
!22 #3 Traning Strategy Multitask Learning Strategy • Round-robin Algorithm
• CPU scheduling ߑߨ ೞա۽ ஹೊఠ ਗਸ ࢎਊೡ ࣻ ח ӝഥܳ “۽ࣁझٜীѱ ҕ”ೞѱ ࠗৈ • п ۽ࣁझী ੌदрਸ ೡ, ೡػ दр աݶ Ӓ ۽ࣁझ ਫ਼द ࠁܨ, ܲ ۽ࣁझীѱ ӝഥܳ ષ • п Taskী ੌߓܳ ೡ, ೡػ ߓо աݶ Ӓ Taskਸ ਫ਼द ࠁܨ, ܲ Taskীѱ ӝഥܳ ષ • Fully Joint • п Task ࣽࢲܳ ҊೞҊ, round-robin algorithmਸ ా೧ batchܳ sampling • Single-task trainingীࢲ iterationਵ۽ب ࣻ۴೮؍ taskٜ ੜ غ݅ աݠח single-task݅ ण೮ ਸ ٸ݅ఀ ࢿמਸ ࠁৈ ޅೣ
!23 #3 Traning Strategy Multitask Learning Strategy • Curriculum Strategy
• Curriculumਸ ٜ݅যࢲ learningदெࠁ! • First Phase:࠺Ү ए taskٜਸ ݢ ण -> Second Phase:য۰ taskٜਸ ण • First Phase(SST, QA-SRL, QA-ZRE, WOZ, WikiSQL, MWSC) -> Second Phase(Others)
!24 #3 Traning Strategy Multitask Learning Strategy • Curriculum Strategy
• Curriculumਸ ٜ݅যࢲ learningदெࠁ! • First Phase:࠺Ү ए taskٜਸ ݢ ण -> Second Phase:য۰ taskٜਸ ण • First Phase(SST, QA-SRL, QA-ZRE, WOZ, WikiSQL, MWSC) -> Second Phase(Others) • Anti-Curriculum Strategy • Curriculumী “߈(Anti)ೞח” ۚ - Curriculum Learning Bengio et al. [2009] • ए taskח ܲ taskٜী بਸ ࣻ ח ਬਊೠ representationਸ णೡ ࣻ হ! • First Phase:য۰ taskٜਸ ݢ ण -> Second Phase:ए taskٜਸ ण • First Phase(SQuAD, IWSLT, CNN/DM, MNLI) -> Second Phase(Others)
#3. Result
!26 Result ־о־о ੜ೮ա?
!27 #1single vs multitask Single vs Multitask Training
!28 #2 curriculum Training Strategy
!29 #2 curriculum Pointer weight distribution
!30 Q & A хࢎפ
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
山中市政が招いた組織崩壊:よこはま安心ボックスを例に
sishi
0
370
論文紹介:Doc-to-LoRA: Learning to Instantly Internalize Contexts
yukako_nakano
0
240
マニアックセッション 〜次世代のメールプロトコルの斜め読み(2026年版)
hirachan
0
520
Evaluation génomique des femelles laitières croisées : comprendre la méthode pour accompagner les éleveurs à son utilisation
institutdelelevage
PRO
0
230
L’albédo:levier d’atténuation du changement climatique dans les systèmes fourragers
institutdelelevage
PRO
0
170
SPACE 2026 - Renforcer l’attractivité des métiers des productions animales dans les établissements d'enseignement agricole
institutdelelevage
PRO
0
120
ros2-perf-multihost: 分散システムにおける客観的なアーキテクチャ評価フレームワーク
takasehideki
0
350
Who’s Behind the Divide?
sirmatrixpage
1
1.6k
生成AI時代の開発方法論と、メールサーバー運用の新しいかたち
hirachan
0
230
[最先端NLP勉強会2026] Checklists Are Better Than Reward Models For Aligning Language Models
nzw0301
1
350
J-STAGEの現況と全文XML登載必須化について
xspa2012
0
410
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
institutdelelevage
PRO
0
160
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
39
3.5k
Navigating Team Friction
lara
192
16k
Heart Work Chapter 1 - Part 1
lfama
PRO
11
37k
Between Models and Reality
mayunak
4
590
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
chikuwait
0
950
So, you think you're a good person
axbom
PRO
2
2.3k
Stop Working from a Prison Cell
hatefulcrawdad
274
21k
Abbi's Birthday
coloredviolet
5
11k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
4.2k
Joys of Absence: A Defence of Solitary Play
codingconduct
1
580
A better future with KSS
kneath
240
18k
Navigating the moral maze — ethical principles for Al-driven product design
skipperchong
2
690