Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc. Scatter Lab Inc.
April 17, 2020
Research
2.3k
0

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

April 17, 2020

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.5k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
[最先端NLP勉強会2026] Checklists Are Better Than Reward Models For Aligning Language Models
Avatar for Kento Nozawa nzw0301
1
140
The story of RefactoringMiner. Slow research, long-term impact
Avatar for Nikolaos Tsantalis tsantalis
0
110
nlp2026 In-Context Learningに基づく経路案内のための地理的知識の活用方法に関する検討
Avatar for Takashi INUI takashiinui
0
130
NLP colloquium: AI Safety Survey
Avatar for Masahiro Kaneko kanekomasahiro
1
980
重要だけど測れていないもの：高齢者ケアの見えない課題
Avatar for テオリア・テクノロジーズ theoriatec2024
0
470
Scalable dynamic origin-destination demand estimation enhanced by high-resolution satellite imagery data
Avatar for SatAI.challenge satai
3
410
HackSick vol.7 LT資料【LLMアーキテクチャ入門・事前学習時の躓き所解説】 スパースなAttention・状態空間モデル
Avatar for Rikka Botan rikkabotan7
0
160
シングルチャネルマルチトーカー音声認識の進展
Avatar for Ryo Masumura ryomasumura
0
220
LLM の Attention 機構まとめ — 数式・計算量・メモリ
Avatar for プワー puwaer
8
2.5k
某助成金プロジェクト採択に向けて企業研究所のアウトリーチ専任者がやったこと
Avatar for afroscript afroscript
0
170
SOTAのさらに先へ：厳しい推論制約下での高性能モデルのPost-Training
Avatar for Hiroshi Y (RabotniKuma) analokmaus
0
1.4k
JICA QUEST 共創×革新プログラム Impact Report（海ノ向こうコーヒー）
Avatar for 坂ノ途中 ontheslope
0
410

Featured

See All Featured
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
3
1.1k
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.8k
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
780
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
410
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
2k
HTML-Aware ERB: The Path to Reactive Rendering @ RubyCon 2026, Rimini, Italy
Avatar for Marco Roth marcoroth
3
470
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
Avatar for Umar Saidu Auna auna
0
210
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.4k
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
280
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
1
2.1k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.5k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
528
40k

Transcript

  1. ,OPXMFEHF%JTUJMMBUJPOGPS#&35 ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS

  2. ݾର ݾର  1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO  %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS 

    'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF  %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  3. 1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  4. • &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35঑୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗ׮ܰѱ1BUJFOUܳ੸ਊೣ • 4PGU5BSHFU݅੉ਊೞ૑ঋҊ׮ܲۨ੉যীࢲب೟णೞب۾ೣ

    1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO

  5. 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ೟ण੹ۚ • ࣁо૑MPTTܳઙ೤ೣ • )BSEUBSHFUী؀ೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFM੄PVUQVU ী؀ೠ$SPTT

    &OUSPQZ-PTT • ઺рۨ੉যٜ੄<$-4>ష௾੄SFQSFTFOBUJPOٜী؀ೠ OPSNBMJ[FE.FBO4RVBSFE-PTT

  6. ೟ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲ୐Lѐ੄ۨ੉য۽4UVEFOUܳୡӝച •

    1,%4LJQFWFSZLMBZFSীࢲ೟ण • 1,%-BTUMBTULMBZFSীࢲ೟ण

  7. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘ੉ఠ݆ਸٸੜؽ

  8. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  9. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  10. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  11. • /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFS࢏ઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS

    DIFBQFSBOEMJHIUFS %JTUJM#&35

  12. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS ೟ण੹ۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •

    .BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐ੄ۨ੉য׼ೞաঀஂೣ

  13. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  14. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  15. 'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  16. • 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ই੘ࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑ੺оמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35

  17. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ

  18. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ • ೟णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI੄$SPTT&OUSPQZ҅࢑ • 6ODFSUBJOUZѐ֛بੑ

    • п-BZFS੄#SBODI߹۽6ODFSUBJOUZ҅࢑ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫׮

  19. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  20. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  21. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  22. %ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  23. • )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35

  24. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  25. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  26. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  27. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ • 8JEUI೟ण • 5FBDIFS੄4PGU5BSHFUҗ$SPTTFOUSPQZ ੐߬٬ۨ੉য )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPO੸ਊ

    • %FQUI೟ण • NPE E  N@E ੋۨ੉যܳESPQ • FYNVMUJUQMJFSח  ਸESPQ • MPTTחXJEUI৬э਷ؘXFJHIUח׮ܴ • ݃૑݄ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH

  28. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  29. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  30. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  31. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  32. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  33. хࢎ೤פ׮✌ ୶о૕ޙژחҾӘೠ੼੉੓׮ݶ঱ઁٚইېোۅ୊۽োۅ઱ࣁਃ ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS  &NBJMVLKBF!TDBUUFSMBCDPLS