Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Knowledge Distillation for BERT
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Scatter Lab Inc.
April 17, 2020
Research
2.3k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Knowledge Distillation for BERT
Scatter Lab Inc.
April 17, 2020
Transcript
33 slides
,OPXMFEHF%JTUJMMBUJPOGPS#&35 .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS
ݾର ݾର 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
• &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗܰѱ1BUJFOUܳਊೣ • 4PGU5BSHFU݅ਊೞঋҊܲۨযীࢲبणೞب۾ೣ
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO णۚ • ࣁоMPTTܳઙೣ • )BSEUBSHFUীೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFMPVUQVU ীೠ$SPTT
&OUSPQZ-PTT • рۨযٜ<$-4>షSFQSFTFOBUJPOٜীೠ OPSNBMJ[FE.FBO4RVBSFE-PTT
ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲLѐۨয۽4UVEFOUܳୡӝച •
1,%4LJQFWFSZLMBZFSীࢲण • 1,%-BTUMBTULMBZFSীࢲण
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘఠ݆ਸٸੜؽ
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
• /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFSઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS
DIFBQFSBOEMJHIUFS %JTUJM#&35
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS णۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •
.BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐۨযೞաঀஂೣ
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
• 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ইࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑоמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ • णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI$SPTT&OUSPQZ҅ • 6ODFSUBJOUZѐ֛بੑ
• п-BZFS#SBODI߹۽6ODFSUBJOUZ҅ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
%ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
• )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ • 8JEUIण • 5FBDIFS4PGU5BSHFUҗ$SPTTFOUSPQZ ߬٬ۨয )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPOਊ
• %FQUIण • NPE E N@E ੋۨযܳESPQ • FYNVMUJUQMJFSח ਸESPQ • MPTTחXJEUI৬эؘXFJHIUחܴ • ݄݃ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS &NBJMVLKBF!TDBUUFSMBCDPLS
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Other Decks in Research
See All in Research
[CV勉強会@関東 CVPR2026] PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow / kantocv 67th CVPR 2026
shunk031
0
410
Hitting the buffers
beale
0
410
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
institutdelelevage
PRO
0
120
湯村研究室の紹介2026 / yumulab2026
yumulab
PRO
0
230
オンライン凸最適化の基礎とゲーム理論への応用 (IBIS 2026 チュートリアル)
taira_tsuchiya
2
1.4k
Adapter les bergeries aux fortes chaleurs
institutdelelevage
PRO
0
180
SAM3を用いたコマ・吹き出しの 領域検出と分割構造からの読み順推定
kzmssk
0
230
MIRU2026 チュートリアル講演2:三次元データ処理の動向
nnchiba
8
5.6k
Être éleveur dans un monde en transition : Comprendre les changements pour mieux les accompagner
institutdelelevage
PRO
0
160
[WebDB2026]セレンディピティ指向推薦システム再考 ―セレンディピティの原義・類型・発生過程に基づく設計指針―
recsyslab
PRO
0
230
The Conductor Pattern -Multi-Granularity Feedback for Creative Agents-
shibuiwilliam
2
120
ICAITD 2026 Oral Presentation
rikkabotan7
0
150
Featured
See All Featured
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
The browser strikes back
jonoalderson
0
2k
GraphQLの誤解/rethinking-graphql
sonatard
75
12k
Build The Right Thing And Hit Your Dates
maggiecrowley
39
3.5k
Designing Experiences People Love
moore
143
24k
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
sarafernandez
0
330
From Legacy to Launchpad: Building Startup-Ready Communities
dugsong
0
380
Documentation Writing (for coders)
carmenintech
77
5.6k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
konakalab
0
670
The Art of Programming - Codeland 2020
erikaheidi
57
15k
How to build an LLM SEO readiness audit: a practical framework
nmsamuel
2
990
Speed Design
sergeychernyshev
33
2.2k