Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Knowledge Distillation for BERT
Search
Scatter Lab Inc.
April 17, 2020
Research
0
2.3k
Knowledge Distillation for BERT
Scatter Lab Inc.
April 17, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
1.8k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.1k
Adversarial Filters of Dataset Biases
scatterlab
0
2.2k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.5k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.3k
What Can Neural Networks Reason About?
scatterlab
0
2.2k
Other Decks in Research
See All in Research
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
tomoyanonymous
0
160
IMC の細かすぎる話 2025
smly
2
630
20250624_熊本経済同友会6月例会講演
trafficbrain
1
620
数理最適化と機械学習の融合
mickey_kubo
16
9.3k
VectorLLM: Human-like Extraction of Structured Building Contours via Multimodal LLMs
satai
4
210
とあるSREの博士「過程」 / A Certain SRE’s Ph.D. Journey
yuukit
10
4.2k
時系列データに対する解釈可能な 決定木クラスタリング
mickey_kubo
2
940
GPUを利用したStein Particle Filterによる点群6自由度モンテカルロSLAM
takuminakao
0
270
利用シーンを意識した推薦システム〜SpotifyとAmazonの事例から〜
kuri8ive
1
250
電力システム最適化入門
mickey_kubo
1
930
Mechanistic Interpretability:解釈可能性研究の新たな潮流
koshiro_aoki
1
410
[CV勉強会@関東 CVPR2025] VLM自動運転model S4-Driver
shinkyoto
2
480
Featured
See All Featured
Unsuck your backbone
ammeep
671
58k
BBQ
matthewcrist
89
9.8k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
A Tale of Four Properties
chriscoyier
160
23k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
The Invisible Side of Design
smashingmag
301
51k
Gamification - CAS2011
davidbonilla
81
5.4k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
3k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Transcript
,OPXMFEHF%JTUJMMBUJPOGPS#&35 .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS
ݾର ݾର 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
• &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗܰѱ1BUJFOUܳਊೣ • 4PGU5BSHFU݅ਊೞঋҊܲۨযীࢲبणೞب۾ೣ
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO णۚ • ࣁоMPTTܳઙೣ • )BSEUBSHFUীೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFMPVUQVU ীೠ$SPTT
&OUSPQZ-PTT • рۨযٜ<$-4>షSFQSFTFOBUJPOٜীೠ OPSNBMJ[FE.FBO4RVBSFE-PTT
ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲLѐۨয۽4UVEFOUܳୡӝച •
1,%4LJQFWFSZLMBZFSীࢲण • 1,%-BTUMBTULMBZFSীࢲण
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘఠ݆ਸٸੜؽ
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
• /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFSઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS
DIFBQFSBOEMJHIUFS %JTUJM#&35
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS णۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •
.BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐۨযೞաঀஂೣ
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
• 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ইࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑоמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ • णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI$SPTT&OUSPQZ҅ • 6ODFSUBJOUZѐ֛بੑ
• п-BZFS#SBODI߹۽6ODFSUBJOUZ҅ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
%ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
• )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ • 8JEUIण • 5FBDIFS4PGU5BSHFUҗ$SPTTFOUSPQZ ߬٬ۨয )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPOਊ
• %FQUIण • NPE E N@E ੋۨযܳESPQ • FYNVMUJUQMJFSח ਸESPQ • MPTTחXJEUI৬эؘXFJHIUחܴ • ݄݃ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS &NBJMVLKBF!TDBUUFSMBCDPLS