Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc. Scatter Lab Inc.
April 17, 2020
Research
0
2.3k

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

April 17, 2020
Tweet

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
1.8k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.3k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.5k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
Collective Predictive Coding and World Models in LLMs: A System 0/1/2/3 Perspective on Hierarchical Physical AI (IEEE SII 2026 Plenary Talk)
Avatar for Tadahiro Taniguchi tanichu
1
270
2025-11-21-DA-10th-satellite
Avatar for Yuka Egusa yegusa
0
130
【SIGGRAPH Asia 2025】Lo-Fi Photograph with Lo-Fi Communication
Avatar for Chinatsu Ozawa toremolo72
0
120
2026 東京科学大 情報通信系 研究室紹介 (大岡山)
Avatar for Tokyo Tech ICT Dept. icttitech
0
400
存立危機事態の再検討
Avatar for Ken Jimbo jimboken
0
250
SREはサイバネティクスの夢をみるか？ / Do SREs Dream of Cybernetics?
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
3
420
社内データ分析AIエージェントを できるだけ使いやすくする工夫
Avatar for Yusuke Fukasawa fufufukakaka
1
950
「なんとなく」の顧客理解から脱却する ──顧客の解像度を武器にするインサイトマネジメント
Avatar for 田島佳穂 tajima_kaho
10
6.4k
Ankylosing Spondylitis
Avatar for Charles Holtzkampf ankh2054
0
140
AIスパコン「さくらONE」の
オブザーバビリティ / Observability for AI Supercomputer SAKURAONE
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
2
1.2k
ドメイン知識がない領域での自然言語処理の始め方
Avatar for Hayahide Yamagishi hargon24
1
250
Attaques quantiques sur Bitcoin : comment se protéger ?
Avatar for Renaud Lifchitz rlifchitz
0
150

Featured

See All Featured
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
62k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
380
The Impact of AI in SEO - AI Overviews June 2024 Edition
Avatar for Aleyda Solis aleyda
5
760
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
370
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
Avatar for Sara Fernández Carmona sarafernandez
0
130
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.1k
Color Theory Basics | Prateek | Gurzu
Avatar for Gurzu gurzu
0
220
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
470
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
1
140
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.4k

Transcript

  1. ,OPXMFEHF%JTUJMMBUJPOGPS#&35 ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS

  2. ݾର ݾର  1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO  %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS 

    'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF  %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  3. 1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  4. • &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35঑୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗ׮ܰѱ1BUJFOUܳ੸ਊೣ • 4PGU5BSHFU݅੉ਊೞ૑ঋҊ׮ܲۨ੉যীࢲب೟णೞب۾ೣ

    1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO

  5. 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ೟ण੹ۚ • ࣁо૑MPTTܳઙ೤ೣ • )BSEUBSHFUী؀ೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFM੄PVUQVU ী؀ೠ$SPTT

    &OUSPQZ-PTT • ઺рۨ੉যٜ੄<$-4>ష௾੄SFQSFTFOBUJPOٜী؀ೠ OPSNBMJ[FE.FBO4RVBSFE-PTT

  6. ೟ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲ୐Lѐ੄ۨ੉য۽4UVEFOUܳୡӝച •

    1,%4LJQFWFSZLMBZFSীࢲ೟ण • 1,%-BTUMBTULMBZFSীࢲ೟ण

  7. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘ੉ఠ݆ਸٸੜؽ

  8. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  9. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  10. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  11. • /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFS࢏ઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS

    DIFBQFSBOEMJHIUFS %JTUJM#&35

  12. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS ೟ण੹ۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •

    .BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐ੄ۨ੉য׼ೞաঀஂೣ

  13. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  14. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  15. 'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  16. • 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ই੘ࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑ੺оמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35

  17. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ

  18. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ • ೟णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI੄$SPTT&OUSPQZ҅࢑ • 6ODFSUBJOUZѐ֛بੑ

    • п-BZFS੄#SBODI߹۽6ODFSUBJOUZ҅࢑ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫׮

  19. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  20. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  21. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  22. %ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  23. • )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35

  24. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  25. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  26. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  27. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ • 8JEUI೟ण • 5FBDIFS੄4PGU5BSHFUҗ$SPTTFOUSPQZ ੐߬٬ۨ੉য )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPO੸ਊ

    • %FQUI೟ण • NPE E  N@E ੋۨ੉যܳESPQ • FYNVMUJUQMJFSח  ਸESPQ • MPTTחXJEUI৬э਷ؘXFJHIUח׮ܴ • ݃૑݄ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH

  28. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  29. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  30. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  31. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  32. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  33. хࢎ೤פ׮✌ ୶о૕ޙژחҾӘೠ੼੉੓׮ݶ঱ઁٚইېোۅ୊۽োۅ઱ࣁਃ ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS  &NBJMVLKBF!TDBUUFSMBCDPLS