Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Knowledge Distillation for BERT
Search
Scatter Lab Inc.
April 17, 2020
Research
0
2.3k
Knowledge Distillation for BERT
Scatter Lab Inc.
April 17, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
1.7k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4k
Adversarial Filters of Dataset Biases
scatterlab
0
2.2k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.4k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.3k
What Can Neural Networks Reason About?
scatterlab
0
2.2k
Other Decks in Research
See All in Research
実行環境に中立なWebAssemblyライブマイグレーション機構/techtalk-2025spring
chikuwait
0
220
3D Gaussian Splattingによる高効率な新規視点合成技術とその応用
muskie82
5
2.5k
Type Theory as a Formal Basis of Natural Language Semantics
daikimatsuoka
1
220
データサイエンティストの就労意識~2015→2024 一般(個人)会員アンケートより
datascientistsociety
PRO
0
660
公立高校入試等に対する受入保留アルゴリズム(DA)導入の提言
shunyanoda
0
5.7k
チャッドローン:LLMによる画像認識を用いた自律型ドローンシステムの開発と実験 / ec75-morisaki
yumulab
1
440
SSII2025 [SS2] 横浜DeNAベイスターズの躍進を支えたAIプロダクト
ssii
PRO
7
3.4k
なめらかなシステムと運用維持の終わらぬ未来 / dicomo2025_coherently_fittable_system
monochromegane
0
270
Vision And Languageモデルにおける異なるドメインでの継続事前学習が性能に与える影響の検証 / YANS2024
sansan_randd
1
110
90 分で学ぶ P 対 NP 問題
e869120
17
7.5k
Mathematics in the Age of AI and the 4 Generation University
hachama
0
160
Large Language Model Agent: A Survey on Methodology, Applications and Challenges
shunk031
12
8.1k
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
The Invisible Side of Design
smashingmag
299
51k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.6k
Side Projects
sachag
455
42k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Testing 201, or: Great Expectations
jmmastey
42
7.5k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
Typedesign – Prime Four
hannesfritz
42
2.7k
A better future with KSS
kneath
239
17k
Code Reviewing Like a Champion
maltzj
524
40k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Transcript
,OPXMFEHF%JTUJMMBUJPOGPS#&35 .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS
ݾର ݾର 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
• &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗܰѱ1BUJFOUܳਊೣ • 4PGU5BSHFU݅ਊೞঋҊܲۨযীࢲبणೞب۾ೣ
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO णۚ • ࣁоMPTTܳઙೣ • )BSEUBSHFUীೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFMPVUQVU ীೠ$SPTT
&OUSPQZ-PTT • рۨযٜ<$-4>షSFQSFTFOBUJPOٜীೠ OPSNBMJ[FE.FBO4RVBSFE-PTT
ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲLѐۨয۽4UVEFOUܳୡӝച •
1,%4LJQFWFSZLMBZFSীࢲण • 1,%-BTUMBTULMBZFSীࢲण
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘఠ݆ਸٸੜؽ
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
• /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFSઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS
DIFBQFSBOEMJHIUFS %JTUJM#&35
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS णۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •
.BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐۨযೞաঀஂೣ
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
• 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ইࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑоמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ • णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI$SPTT&OUSPQZ҅ • 6ODFSUBJOUZѐ֛بੑ
• п-BZFS#SBODI߹۽6ODFSUBJOUZ҅ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
%ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
• )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ • 8JEUIण • 5FBDIFS4PGU5BSHFUҗ$SPTTFOUSPQZ ߬٬ۨয )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPOਊ
• %FQUIण • NPE E N@E ੋۨযܳESPQ • FYNVMUJUQMJFSח ਸESPQ • MPTTחXJEUI৬эؘXFJHIUחܴ • ݄݃ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS &NBJMVLKBF!TDBUUFSMBCDPLS