Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc. Scatter Lab Inc.
April 17, 2020
Research
0
2.3k

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

April 17, 2020
Tweet

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
1.8k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.2k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.5k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
心理言語学の視点から再考する言語モデルの学習過程
Avatar for Masato Mita chemical_tree
2
690
Stealing LUKS Keys via TPM and UUID Spoofing in 10 Minutes - BSides 2025
Avatar for AnyKeyShik anykeyshik
0
160
CVPR2025論文紹介：Unboxed
Avatar for 村川卓也 murakawatakuya
0
210
SREのためのテレメトリー技術の探究 / Telemetry for SRE
Avatar for Yuuki Tsubouchi (yuuk1) yuukit
12
2k
Remote sensing × Multi-modal meta survey
Avatar for SatAI.challenge satai
4
590
超高速データサイエンス
Avatar for Yusuke Matsui matsui_528
1
220
財務諸表監査のための逐次検定
Avatar for MasaKat0 masakat0
0
200
まずはここから：Overleaf共同執筆・CopilotでAIコーディング入門・Codespacesで独立環境
Avatar for Yusuke Matsui matsui_528
2
770
言語モデルの地図：確率分布と 情報幾何による類似性の可視化
Avatar for Hidetoshi Shimodaira shimosan
8
2.1k
Minimax and Bayes Optimal Best-arm Identification: Adaptive Experimental Design for Treatment Choice
Avatar for MasaKat0 masakat0
0
190
CoRL2025速報
Avatar for robotpaper.challenge rpc
2
3.2k
投資戦略202508
Avatar for Pragmaworks pw
0
570

Featured

See All Featured
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
359
30k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
60
9.6k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
1
59
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
14k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
73
4.9k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
84
9.3k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
34
2.3k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
355
21k
Scaling GitHub
Avatar for Zach Holman holman
464
140k

Transcript

  1. ,OPXMFEHF%JTUJMMBUJPOGPS#&35 ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS

  2. ݾର ݾର  1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO  %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS 

    'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF  %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  3. 1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  4. • &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35঑୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗ׮ܰѱ1BUJFOUܳ੸ਊೣ • 4PGU5BSHFU݅੉ਊೞ૑ঋҊ׮ܲۨ੉যীࢲب೟णೞب۾ೣ

    1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO

  5. 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ೟ण੹ۚ • ࣁо૑MPTTܳઙ೤ೣ • )BSEUBSHFUী؀ೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFM੄PVUQVU ী؀ೠ$SPTT

    &OUSPQZ-PTT • ઺рۨ੉যٜ੄<$-4>ష௾੄SFQSFTFOBUJPOٜী؀ೠ OPSNBMJ[FE.FBO4RVBSFE-PTT

  6. ೟ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲ୐Lѐ੄ۨ੉য۽4UVEFOUܳୡӝച •

    1,%4LJQFWFSZLMBZFSীࢲ೟ण • 1,%-BTUMBTULMBZFSীࢲ೟ण

  7. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘ੉ఠ݆ਸٸੜؽ

  8. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  9. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  10. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  11. • /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFS࢏ઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS

    DIFBQFSBOEMJHIUFS %JTUJM#&35

  12. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS ೟ण੹ۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •

    .BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐ੄ۨ੉য׼ೞաঀஂೣ

  13. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  14. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  15. 'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  16. • 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ই੘ࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑ੺оמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35

  17. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ

  18. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ • ೟णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI੄$SPTT&OUSPQZ҅࢑ • 6ODFSUBJOUZѐ֛بੑ

    • п-BZFS੄#SBODI߹۽6ODFSUBJOUZ҅࢑ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫׮

  19. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  20. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  21. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  22. %ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  23. • )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35

  24. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  25. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  26. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  27. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ • 8JEUI೟ण • 5FBDIFS੄4PGU5BSHFUҗ$SPTTFOUSPQZ ੐߬٬ۨ੉য )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPO੸ਊ

    • %FQUI೟ण • NPE E  N@E ੋۨ੉যܳESPQ • FYNVMUJUQMJFSח  ਸESPQ • MPTTחXJEUI৬э਷ؘXFJHIUח׮ܴ • ݃૑݄ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH

  28. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  29. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  30. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  31. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  32. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  33. хࢎ೤פ׮✌ ୶о૕ޙژחҾӘೠ੼੉੓׮ݶ঱ઁٚইېোۅ୊۽োۅ઱ࣁਃ ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS  &NBJMVLKBF!TDBUUFSMBCDPLS