Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Knowledge Distillation for BERT
Search
Scatter Lab Inc.
April 17, 2020
Research
0
2.1k
Knowledge Distillation for BERT
Scatter Lab Inc.
April 17, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
3.1k
Adversarial Filters of Dataset Biases
scatterlab
0
2.1k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.1k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.2k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.1k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.1k
What Can Neural Networks Reason About?
scatterlab
0
2.1k
Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer
scatterlab
0
2k
Other Decks in Research
See All in Research
Embodied AIについて / About Embodied AI
nttcom
1
510
Target trial emulationの概要
shuntaros
2
1.1k
20240209 データを肴に熊本の交通を考える会「車1割削減、渋滞半減、公共交通2倍」をめざし世界に学ぼう
trafficbrain
0
780
自己教師あり学習による事前学習(CVIMチュートリアル)
naok615
2
1.4k
VAR モデルによる OSS プロジェクト同士が生存性に与える 影響の分析
noppoman
0
130
People Driven Transformation / 人が起点の、社会の変え方
dmattsun
0
150
東工大Swallowプロジェクトにおける大規模日本語Webコーパスの構築
aya_se
12
6.3k
言語間転移学習で大規模言語モデルを賢くする
ikuyamada
6
1.8k
プロシェアリング白書2024_PROSHARING_REPORT_2024
circulation
0
600
Breaking Tradeoffs: Extremely Scalable Multi-Agent Pathfinding Algorithms
kei18
0
140
20240127_熊本から今いちど真面目に都市交通~めざせ「車1割削減、渋滞半減、公共交通2倍」~ 全国路面電車サミット2024宇都宮
trafficbrain
1
650
DeepCrysTet: A Deep Learning Approach Using Tetrahedral Mesh for Predicting Properties of Crystalline Materials
tsurubee
0
360
Featured
See All Featured
Creatively Recalculating Your Daily Design Routine
revolveconf
209
11k
Code Review Best Practice
trishagee
54
15k
Happy Clients
brianwarren
91
6.4k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.6k
The Invisible Side of Design
smashingmag
294
49k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
6
990
Product Roadmaps are Hard
iamctodd
43
9.7k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
1
1.3k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Rails Girls Zürich Keynote
gr2m
91
13k
Building Better People: How to give real-time feedback that sticks.
wjessup
354
18k
Transcript
,OPXMFEHF%JTUJMMBUJPOGPS#&35 .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS
ݾର ݾର 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
• &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗܰѱ1BUJFOUܳਊೣ • 4PGU5BSHFU݅ਊೞঋҊܲۨযীࢲبणೞب۾ೣ
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO णۚ • ࣁоMPTTܳઙೣ • )BSEUBSHFUীೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFMPVUQVU ীೠ$SPTT
&OUSPQZ-PTT • рۨযٜ<$-4>షSFQSFTFOBUJPOٜীೠ OPSNBMJ[FE.FBO4RVBSFE-PTT
ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲLѐۨয۽4UVEFOUܳୡӝച •
1,%4LJQFWFSZLMBZFSীࢲण • 1,%-BTUMBTULMBZFSীࢲण
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘఠ݆ਸٸੜؽ
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
• /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFSઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS
DIFBQFSBOEMJHIUFS %JTUJM#&35
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS णۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •
.BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐۨযೞաঀஂೣ
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
• 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ইࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑоמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ • णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI$SPTT&OUSPQZ҅ • 6ODFSUBJOUZѐ֛بੑ
• п-BZFS#SBODI߹۽6ODFSUBJOUZ҅ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
%ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
• )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ • 8JEUIण • 5FBDIFS4PGU5BSHFUҗ$SPTTFOUSPQZ ߬٬ۨয )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPOਊ
• %FQUIण • NPE E N@E ੋۨযܳESPQ • FYNVMUJUQMJFSח ਸESPQ • MPTTחXJEUI৬эؘXFJHIUחܴ • ݄݃ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS &NBJMVLKBF!TDBUUFSMBCDPLS