Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Knowledge Distillation for BERT
Search
Scatter Lab Inc.
April 17, 2020
Research
0
2.3k
Knowledge Distillation for BERT
Scatter Lab Inc.
April 17, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
1.7k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4k
Adversarial Filters of Dataset Biases
scatterlab
0
2.2k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.5k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.3k
What Can Neural Networks Reason About?
scatterlab
0
2.2k
Other Decks in Research
See All in Research
SSII2025 [SS1] レンズレスカメラ
ssii
PRO
2
980
20250624_熊本経済同友会6月例会講演
trafficbrain
1
420
Ad-DS Paper Circle #1
ykaneko1992
0
5.6k
Google Agent Development Kit (ADK) 入門 🚀
mickey_kubo
2
1.2k
Mathematics in the Age of AI and the 4 Generation University
hachama
0
170
Agentic AIとMCPを利用したサービス作成入門
mickey_kubo
0
300
Cross-Media Information Spaces and Architectures
signer
PRO
0
230
Looking for Escorts in Sydney?
lunsophia
1
120
Principled AI ~深層学習時代における課題解決の方法論~
taniai
3
1.2k
Pix2Poly: A Sequence Prediction Method for End-to-end Polygonal Building Footprint Extraction from Remote Sensing Imagery
satai
3
500
生成的推薦の人気バイアスの分析:暗記の観点から / JSAI2025
upura
0
200
LLM-as-a-Judge: 文章をLLMで評価する@教育機関DXシンポ
k141303
3
830
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
40
1.9k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
BBQ
matthewcrist
89
9.7k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Writing Fast Ruby
sferik
628
62k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
The Language of Interfaces
destraynor
158
25k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Transcript
,OPXMFEHF%JTUJMMBUJPOGPS#&35 .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS
ݾର ݾର 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
• &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗܰѱ1BUJFOUܳਊೣ • 4PGU5BSHFU݅ਊೞঋҊܲۨযীࢲبणೞب۾ೣ
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO
1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO णۚ • ࣁоMPTTܳઙೣ • )BSEUBSHFUীೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFMPVUQVU ীೠ$SPTT
&OUSPQZ-PTT • рۨযٜ<$-4>షSFQSFTFOBUJPOٜীೠ OPSNBMJ[FE.FBO4RVBSFE-PTT
ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲLѐۨয۽4UVEFOUܳୡӝച •
1,%4LJQFWFSZLMBZFSীࢲण • 1,%-BTUMBTULMBZFSীࢲण
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘఠ݆ਸٸੜؽ
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
• /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFSઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS
DIFBQFSBOEMJHIUFS %JTUJM#&35
%JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS णۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •
.BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐۨযೞաঀஂೣ
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS
'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
• 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ইࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑоמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ
'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF णۚ • णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI$SPTT&OUSPQZ҅ • 6ODFSUBJOUZѐ֛بੑ
• п-BZFS#SBODI߹۽6ODFSUBJOUZ҅ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF
%ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
• )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ
%ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI णۚ • 8JEUIण • 5FBDIFS4PGU5BSHFUҗ$SPTTFOUSPQZ ߬٬ۨয )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPOਊ
• %FQUIण • NPE E N@E ੋۨযܳESPQ • FYNVMUJUQMJFSח ਸESPQ • MPTTחXJEUI৬эؘXFJHIUחܴ • ݄݃ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS &NBJMVLKBF!TDBUUFSMBCDPLS