Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Beyond Accuracy: Behavioral Testing of NLP Mode...
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Scatter Lab Inc.
July 31, 2020
Research
2.4k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Scatter Lab Inc.
July 31, 2020
Transcript
40 slides
#FZPOE"DDVSBDZ#FIBWJPSBM5FTUJOHPG/-1 .PEFMTXJUI$)&$,-*45 ߅ള .-3FTFBSDI4DJFOUJTU 1JOHQPOH
0WFSWJFX 0WFSWJFX "$- #FTU1BQFS .JDSPTPGU3FTFBSDI 6OJWPG8BTIJOHUPO 6OJWPG$BMJGPSOJB
*SWJOF $PEFIUUQTHJUIVCDPNNBSDPUDSDIFDLMJTU ೠਃড নೠ/-1ݽ؛ਸ4PGUXBSF5FTUJOHҙীࢲಣо೧ࠁ
ݾର ݾର *OUSPEVDUJPO $)&$,-*45 5FTUJOH405"XJUI$)&$,-*45 $PNNFSDJBM4ZTUFNBOE$)&$,-*45
$PODMVTJPO
*OUSPEVDUJPO *OUSPEVDUJPO
*OUSPEVDUJPO (FOFSBMJ[BUJPOPG/-1NPEFMT ↟ ੌ߈ച (FOFSBMJ[BUJPO ח٩۞ݽ؛୶ҳ೧ঠೞחоݾೞաੑפ ↟ ݽ؛णೠؘఠࡺ݅ইפۄ ীחೠߣبࠁঋؘఠ VOTFFOEBUB
ীࢲبੜ೧ঠפ ↟ ݽ؛ੌ߈ചػࢿמਸࡅܰѱঈೞӝਤ೧ ؘఠࣇਸള۲ಣоࣇਵ۽ҳ࠙ ↟ ୭Ӕীח)VNBOQFSGPNBODFܳڪযֈחݽ؛ٜبࣻઓ
*OUSPEVDUJPO (FOFSBMJ[BUJPOPG/-1NPEFMT ↟ ੌ߈ച (FOFSBMJ[BUJPO ח٩۞ݽ؛୶ҳ೧ঠೞחоݾೞաੑפ ↟ ݽ؛णೠؘఠࡺ݅ইפۄ ীחೠߣبࠁঋؘఠ VOTFFOEBUB
ীࢲبੜ೧ঠפ ↟ ݽ؛ੌ߈ചػࢿמਸࡅܰѱঈೞӝਤ೧ ؘఠࣇਸള۲ಣоࣇਵ۽ҳ࠙ ↟ ୭Ӕীח)VNBOQFSGPNBODFܳڪযֈחݽ؛ٜبࣻઓ ↟ ೞ݅ಣоؘఠীࢲੜೞחѪਵ۽࠙ೡө ↟ ٩۞ݽ؛ਵ۽যܳ೧ೞӝࠁח 5BTLTQFDJGJDೞҊ%BUBTQFDJGJDೠಁఢਸ৻ਛӝٸޙীࢿמ֫חبԲળઁӝؾפ ↟ ٮۄࢲഅۢMFBEFSCPBSE֫Ҕਸೱೠোҳߑೱ ஞݽ؛מ۱ਸPWFSFTUJNBUFೡࣻبणפ ↟ tղо݅ٚݽ؛SFBMXPSMEীࢲઁ۽زೞחഛੋೞ۰ݶ যڌѱ೧ঠೡө u
(PPHMF4FOUJNFOU"OBMZTJT"1* *OUSPEVDUJPO
(PPHMF4FOUJNFOU"OBMZTJT"1* *OUSPEVDUJPO ↟ য়טݡधաࢁঋওݶযب/FVUSBMա৬ঠೞ݅ ৈ/FHBUJWFੑפ
(PPHMF4FOUJNFOU"OBMZTJT"1* *OUSPEVDUJPO
(PPHMF4FOUJNFOU"OBMZTJT"1* *OUSPEVDUJPO ↟ /FVUSBMޙ/FHBUJPO/FVUSBMৈঠೞ݅ ъೠ/FHBUJWFоա৳णפ
1SFWJPVT8PSL *OUSPEVDUJPO ↟ ഛب৻নೠߑߨਵ۽ݽ؛מ۱ਸಣоೞחোҳоઓפ ↟ بਵ۽য়ఋٜܳ݅যࢲݽ؛3PCVTUOFTTܳಣоೞӝ ↟ ޙਸQBSBQISBTJOH೧ࢲ߸ೞחࠁӝ ↟ ౠݺࢎܳജ೧ࢲݽ؛'BJSOFTTܳಣоೞӝ
tղҊೱೠҴঠu tղҊೱࣗ۲ঠu tղҊೱইܻঠu ↟ ӝઓোҳೠ҅ ↟ ࠗ࠙োҳח2" TFOUJNFOUBOBMZTJT /-*١ѐ߹ੋUBTL࢚ܳਵ۽ೣ ↟ ݽ؛ীೠನҚੋಣоо٘ۄੋغӝয۰
$)&$,-*45 $)&$,-*45
ઁউೞחߑߨݽ؛ب4PGUXBSFܳ5FTUೞחѪۢಣо೧ࠁ $)&$,-*45 ↟ 4PGUXBSF&OHJOFFSJOH࠙ঠt#FIBWJPS5FTUJOHu BLB#MBDLCPY5FTUJOH җਬࢎೠߑߨਸઁউפ
ઁউೞחߑߨݽ؛ب4PGUXBSFܳ5FTUೞחѪۢಣо೧ࠁ ↟ 4PGUXBSF&OHJOFFSJOH࠙ঠt#FIBWJPS5FTUJOHu BLB#MBDLCPY5FTUJOH җਬࢎೠߑߨਸઁউפ ↟ ղࠗҳઑחഃҊ۰ೞঋҊ ٯੑ۱۱݅оҊݽ؛מ۱ $BQBCJMJUZ ਸಣоפ
8IBUUP5FTU ↟ 5BTLীҙ҅হы୶যঠೞחৈ۞-JOHVJTUJD$BQBCJMJUZ FH /FHBUJPO /&3 ܳݽ؛ઁ۽ы୶Ҋחಣоפ $)&$,-*45
ઁউೞחߑߨݽ؛ب4PGUXBSFܳ5FTUೞחѪۢಣо೧ࠁ ↟ 4PGUXBSF&OHJOFFSJOH࠙ঠt#FIBWJPS5FTUJOHu BLB#MBDLCPY5FTUJOH җਬࢎೠߑߨਸઁউפ ↟ ղࠗҳઑחഃҊ۰ೞঋҊ ٯੑ۱۱݅оҊݽ؛מ۱ $BQBCJMJUZ ਸಣоפ
8IBUUP5FTU ↟ 5BTLীҙ҅হы୶যঠೞחৈ۞-JOHVJTUJD$BQBCJMJUZ FH /FHBUJPO /&3 ܳݽ؛ઁ۽ы୶Ҋחಣоפ ↟ ೞա$BQBCJMJUZоGBJMVSF೮ਸٸ ܳTQFDJGJDCFIBWJPS۽ա־ӝਤ೧ѐ5FTU5ZQFਸبੑפ )PXUP5FTU ↟ .JOJNVN'VODUJPOBMJUZ5FTU .'5 ݽ؛۱࢚чҗزੌೠ 6OJU5FTU ↟ *OWBSJBODF5FTU */7 ݽ؛ੑ۱߄Շযب۱җزੌೠ 1FSUVSCBUJPO5FTU ↟ %JSFDUJPOBM&YQFDUBUJPO5FTU %&5 ݽ؛ੑ۱ਸ߄Բݶ۱࢚чҗزੌೠ 1FSUVSCBUJPO5FTU $)&$,-*45
ઁউೞחߑߨݽ؛ب4PGUXBSFܳ5FTUೞחѪۢಣо೧ࠁ ↟ 4PGUXBSF&OHJOFFSJOH࠙ঠt#FIBWJPS5FTUJOHu BLB#MBDLCPY5FTUJOH җਬࢎೠߑߨਸઁউפ ↟ ղࠗҳઑחഃҊ۰ೞঋҊ ٯੑ۱۱݅оҊݽ؛מ۱ $BQBCJMJUZ ਸಣоפ
8IBUUP5FTU ↟ 5BTLীҙ҅হы୶যঠೞחৈ۞-JOHVJTUJD$BQBCJMJUZ FH /FHBUJPO /&3 ܳݽ؛ઁ۽ы୶Ҋחಣоפ ↟ ೞա$BQBCJMJUZоGBJMVSF೮ਸٸ ܳTQFDJGJDCFIBWJPS۽ա־ӝਤ೧ѐ5FTU5ZQFਸبੑפ )PXUP5FTU ↟ .JOJNVN'VODUJPOBMJUZ5FTU .'5 ݽ؛۱࢚чҗزੌೠ 6OJU5FTU ↟ *OWBSJBODF5FTU */7 ݽ؛ੑ۱߄Շযب۱җزੌೠ 1FSUVSCBUJPO5FTU ↟ %JSFDUJPOBM&YQFDUBUJPO5FTU %&5 ݽ؛ੑ۱ਸ߄Բݶ۱࢚чҗزੌೠ 1FSUVSCBUJPO5FTU ↟ ݆పझ٘ܳࡅܰѱ݅٘חѪਸذӝਤ೧UFNQMBUF MFYJDPO QFSUVSCBUJPO WJTVBMJ[BUJPO .BTLFE-.١5PPMਸઁҕפ $)&$,-*45
&YBNQMF ↟ ಣоೞח$BQBCJMJUZ/&("5*0/ ↟ 5FTU5ZQF.'5 ۱࢚җزੌೠ $)&$,-*45
&YBNQMF ↟ ಣоೞח$BQBCJMJUZ/&3 ↟ 5FTU5ZQF*/7 ੑ۱׳ۄઉب۱زੌೠ $)&$,-*45
&YBNQMF ↟ ಣоೞח$BQBCJMJUZ7PDBCVMBSZ ↟ 5FTU5ZQF%*3 ੑ۱׳ۄٸ۱࢚җزੌೠ $)&$,-*45
&YBNQMF $)&$,-*45 ↟ ৬эపझоաݶ ਤۢݽ؛ыযמ۱ਸಣоೡࣻणפ
6TBHF&YBNQMFమ݁࠼ਸࣚਵ۽ਕࢲపझாझٜܳ݅ӝ $)&$,-*45
6TBHF&YBNQMFమ݁࠼ਸ-FYJDPOਵ۽ਕࢲపझாझٜܳ݅ӝ $)&$,-*45
6TBHF&YBNQMFమ݁࠼ਸ#&35.BTLFE-.ਵ۽ਕࢲపझாझٜܳ݅ӝ $)&$,-*45
6TBHF&YBNQMFమ݁࠼ਸ#&35.BTLFE-.ਵ۽ਕࢲపझாझٜܳ݅ӝ $)&$,-*45
5FTUJOH405"XJUI$)&$,-*45 5FTUJOH405"XJUI$)&$,-*45
405"ݽ؛ٜਸಣо೧ࠁ 5FTUJOH405"XJUI$)&$,-*45 ↟ Ӕؘࢎपখ5BTL݅ࠁݶ ޙઁоցޖएਕࠁੌࣻبणפ ↟ ѐ5BTLী೧೯פ ↟ 4FOUJNFOU"OBMZTJT ↟
%VQMJDBUFE2VFTUJPO%FUFDUJPO ↟ 3FBEJOH$PNQSFIFOTJPO ↟ ৈ۞ݽ؛ٜਸ࢚ਵ۽פ ↟ $PNNFSDJBMNPEFMT(PPHMF .4 "NB[PO ↟ 405"NPEFMT#&35 3P#&35B
4FOUJNFOU"OBMZTJT 5FTUJOH405"XJUI$)&$,-*45 ↟ /FVUSBM8PSEܳ߄Ծীب4FOUJNFOUѾҗо߄Շणפ
4FOUJNFOU"OBMZTJT 5FTUJOH405"XJUI$)&$,-*45 ↟ ࢎѤࢲࣽਸ߈ೞޅೣਸঌࣻणפ
4FOUJNFOU"OBMZTJT 5FTUJOH405"XJUI$)&$,-*45 ↟ ױࣽೠࠗޙীبपಁೠѾҗо݆णפ
4FOUJNFOU"OBMZTJT 5FTUJOH405"XJUI$)&$,-*45 ↟ খޙਸߣࠂೞחޙࠢਵݶஏѾҗо߄Շযঠೞ݅ Ӓ۞ޅפ
%VQMJDBUFE2VFTUJPO%FUFDUJPO 5FTUJOH405"XJUI$)&$,-*45
%VQMJDBUFE2VFTUJPO%FUFDUJPO 5FTUJOH405"XJUI$)&$,-*45 ↟ Ѿҗஏীҙ҅হחݺࢎоജغݶѾҗо߄Շযߡ݀פ
3FBEJOH$PNQSFIFOTJPO 5FTUJOH405"XJUI$)&$,-*45 ↟ IBNTUFSWFIJDMFۄחҙ҅ܳঈೞޅפ
3FBEJOH$PNQSFIFOTJPO 5FTUJOH405"XJUI$)&$,-*45 ↟ য়ఋ١QFSUVSCBUJPOীхೞѱ߈פ
$PNNFSDJBM4ZTUFNBOE $)&$,-*45 $PNNFSDJBM4ZTUFNBOE$)&$,-*45
$PNNFSDJBM4ZTUFNBOE$)&$,-*45 ↟ খীࢲࠅࣻ٠ *5ӝসࢲ࠺झա405"ݽ؛ۄبষ݆ߡӒоߊѼغणפ ↟ Ӓۢ$)&$,-*45оߡӒܳחؘ݃աبؼࣻਸө
$PNNFSDJBM4ZTUFNBOE$)&$,-*45 6TFSTUVEZ#&35۽݅ٚ%21ܳदрزউಣо೧ࠁ ↟ ೠਃড $)&$,-*45оਵݶ؊݆పझாझٜܳ݅Ҋ ؊݆ߡӒܳওणפ $BQBCJMJUZ$BQBCJMJUZ 5FNQMBUF
$PODMVTJPO $PODMVTJPO
$PODMVTJPO ↟ /-1ݽ؛ਸ4PGUXBSFۢಣо೧ঠೠ ↟ Ӓۧݶޖਸ যڌѱಣо೧ঠೡө ↟ ޖਸ-JOHVJTUJD$BQBCJMJUZ ↟
যڌѱ6OJU5FTU .'5 1FSUVSCBUJPOT */7 %*3 ↟ ܳذӝਤೠ.BTLFE-. दпച -FYJDPO١ਸઁҕפ ↟ $PEFIUUQTHJUIVCDPNNBSDPUDSDIFDLMJTU $PODMVTJPO
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ ߅ള .-3FTFBSDI4DJFOUJTU 1JOHQPOH &NBJMDIBFIVO!TDBUUFSMBCDPLS
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer
scatterlab
0
2.3k
Other Decks in Research
See All in Research
Model Predictive Control for Uncertain Systems: Robust & Data-Driven Designs
kohlerjohannes
0
230
[IEICE-Soceity-Conf2026] セマンティック通信を見据えたマルチスケールセンシングと 学習の融合によるデジタルツイン基盤 / Digital Twin Infrastructure via the Fusion of Multi-Scale Sensing and Learning toward Semantic Communications
pman0214
0
150
超効率化への挑戦:1bit LLMの現状と展望
yumaichikawa
0
1k
Détection des lésions podales par thermographie infrarouge chez les bovins et les ovins
institutdelelevage
PRO
0
130
生成AI時代の開発方法論と、メールサーバー運用の新しいかたち
hirachan
0
230
ros2-perf-multihost: 分散システムにおける客観的なアーキテクチャ評価フレームワーク
takasehideki
0
350
Du suivi individuel à la sélection animale : comment Digi4Live valorise les données des technologies numériques en élevage
institutdelelevage
PRO
0
120
Physical AIでモデリングはどう変わるか/How Physical AI Will Change Modeling
stktu
0
350
宛先から考えるデジタルアーカイブ:――ジャック・デリダの〈痕跡〉と三輪田米山日記
ktanishima
0
100
完全自律LLMエージェントでDIVER OSINT CTF 2026に挑戦してみた
analokmaus
0
830
ふとした出会いで生まれたSkillが、 社内利用1位になるまで
mikimhk
22
25k
Kunai: Toward a Verifier-Safe Layered DSL for Nested Encapsulation Packet Filtering on eBPF
takehaya
1
210
Featured
See All Featured
Building a A Zero-Code AI SEO Workflow
portentint
PRO
0
840
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4.6k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
25
2.2k
Test your architecture with Archunit
thirion
2
2.5k
The Spectacular Lies of Maps
axbom
PRO
1
1.1k
Optimising Largest Contentful Paint
csswizardry
37
4k
Practical Orchestrator
shlominoach
192
12k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
aleyda
2
12k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
mfonobong
2
640
Docker and Python
trallard
47
4.3k
Designing for Timeless Needs
cassininazir
1
570