Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Adversarial Filters of Dataset Biases
Search
Scatter Lab Inc.
September 04, 2020
Research
2.3k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Adversarial Filters of Dataset Biases
Scatter Lab Inc.
September 04, 2020
Transcript
22 slides
Adversarial Filters of Dataset Biases ࢿࠁ (ML Research Scientist, Pingpong)
ݾର ݾର 1. োҳ ߓ҃ 2. AFLite 1. द: WinoGrande
ؘఠࣇ 2. ੌ߈ചػ ঌҊ્ܻ 3. प 1. Synthetic Data 2. NLP 3. Vision
োҳ ߓ҃ োҳ ߓ҃
‘߮݃ ؘఠࣇীࢲ ֫ ࢿמਸ ׳ࢿ೮Ҋ ೧ ޙઁܳ ೧Ѿ೮Ҋ ݈ೡ ࣻ
ਸө?’ • In-distribution పझࣇীࢲח ੜೞ݅ Out-of-distribution adversarial sampleীח ডೠ അ࢚ • Input-Output рী ب ঋ Spurious correlation ࢤ҂ӝ ٸޙ • ܳ ೧Ѿೠ ؘఠࣇਸ ٜ݅যঠ दझమਸ ઁ۽ ಣоೡ ࣻ োҳ ߓ҃ High Performance = Problem Solved?
োҳо domain-specificೠ spurious ಁఢਸ ࠙ܨ ߂ ೞҊ ܳ ઁѢೞח
ߑध • োҳ domain-specificೠ धҗ ҙী ઓ • ঌҊ્ܻ ࢸ҅о Ҋ۰ೞ ޅೠ biasח ழߡ ࠛо োҳ ߓ҃ Previous Approaches
AFLite AFLite
• ޙীࢲ ݺࢎо оܻఃח ࢚ਸ ݏח ޙઁ • SOTA ഛب
ড 90% → ݽ؛ Spurious correlationਸ ਊೞח ѱ ইקө? • (3), (4)ח ߃ հ݈ җ ҙ۲ ਸ ഛܫ ֫ই Word association݅ਵ۽ ޙઁܳ ಽ ࣻ AFLite Winograd Schema Challenge (WSC)
• ࢎۈ ؘఠࣇਸ ٜ݅ݶ ۠ Annotation artifactী ೠ Biasܳ
ೖೞӝ য۰ • AFLite۽ ఠ݂ೠ WinoGrande ؘఠࣇ ݽ؛ ഛبب ծҊ ܲ ߮݃۽ Transferب ੜؽ AFLite WinoGrande Dataset
1. ؘఠ ੌࠗ݅ਵ۽ RoBERTa fine-tune 2. Splitਸ ׳ܻ ೞݶࢲ RoBERTa
feature۽ linear classifier ण 3. Split పझࣇীࢲ ߬٬݅ਵ۽ ਸ औѱ ਸ ࣻ ח పझ ೞҊ ੋझఢझ߹۽ ঔ࢚࠶ ࣇী ୶о 4. ৈ۞ linear classifierо ਸ ݏ൦ ࠺ਯ Thresholdܳ ֈח Ѫ Top-kѐܳ ୭ઙ ؘఠࣇীࢲ ઁ৻ 5. ઁ৻غח ѐࣻо kѐо উ غѢա ਗೞח ӝ ؘఠࣇ ؼ ٸ ө 2~4 ߈ࠂ AFLite AFLite in WinoGrande
• ױয ӓࢿ݅ਵ۽ ಽ ࣻ ח ޙઁܳ Ѧ۞ն • ח
ష ۨ߰ Biasۄӝࠁח ҳઑੋ Ѫ۽ lexical-level heuristicਵ۽ח Ѧ۞ղӝ ൨ٝ AFLite Filtered Examples
• AFLiteܳ ৈ۞ بݫੋਵ۽ ഛೞҊ model-agnosticೞѱ ੌ߈ച • Contributions: 1.
࢚݅ intractableೠ AFOptܳ AFLite۽ Ӕࢎೡ ࣻ ਸ ࠁੋ. (Skip) 2. Vision, NLP ࠙ঠ ৈ۞ ؘఠࣇীࢲ प೧ AFLite ਬബࢿਸ ّ߉ஜೠ. 3. Biasܳ হঙ ؘఠࣇਵ۽ णೠ ݽ؛ ੌ߈ചо ੜؽਸ पਵ۽ ࠁੋ. 4. AFLite۽ ఠ݂ೞݶ ؊ بੋ ߮݃ ؘఠࣇਸ ٜ݅ ࣻ ਸ ࠁੋ. AFLite Adversarial Filters of Dataset Biases
: any feature extractor : a family of classification models
Φ M AFLite AFLite (Generalized)
Experiments Experiments
Biasing Dataset • Class-specificೠ ੋҕ featureܳ ؘఠ 75%ী ੑ, աݠח
random feature ੑ • Biased sample ੌࠗח ۨ࠶ ߄Է Results • Linear classifier۽ب ֫ ࢿמ ׳ࢿ • AFLiteܳ ਊೞݶ ࢚धੋ ࢿמਵ۽ جই১ Experiments Synthetic Data
• प ࢚: SNLI annotation artifactܳ ೖೠ out-of-distribution ؘఠࣇ 3ઙ
• Non-entailment ޙઁ ਬഋ߹۽ Zero-shot పझ Experiments NLP: Out-of-distribution Generalization
AFLite۽ ఠ݂ೠ ؘఠࣇ ݽٚ ݽ؛ীࢲ ࢿמ ѱ ڄয Experiments In-distribution
Benchmark Re-estimation: SNLI
Experiments In-distribution Benchmark Re-estimation: MultiNLI & QNLI
• : ImageNet ؘఠࣇ 20%۽ णೠ EfficientNet-B7 feature • ImageNet-A۽
ಣоೞפ AFLite-filtered ؘఠࣇਵ۽ ण೮ਸ ٸ ࢿמ ؊ જ Φ Experiments Vision: Adversarial Image Classification
ImageNet dev setਸ ఠ݂ೞҊ ಣо೮ਸ ٸ ࢿמ ೞۅ ؊ ఀ
Experiments In-distribution Image Classification
ӝઓীب ࠁҊػ ౠ ನૉী ೠ Bias, ݽনࠁ х݅ਵ۽ ҳ࠙ೞח ޙઁ
١җ Ѿਸ эೣ Experiments Filtered Examples
• Adversarial Filtering SWAG: A Large-Scale Adversarial Dataset for Grounded
Commonsense Inference [EMNLP’18] HellaSwag: Can a Machine Really Finish Your Sentence? [ACL’19] • AFLite WinoGrande: An Adversarial Winograd Schema Challenge at Scale [arXiv’19] Adversarial Filters of Dataset Biases [ICML’20] References References
хࢎפ✌ ୶о ޙ ژח ҾӘೠ ݶ ઁٚ ইې োۅ۽
োۅ ࣁਃ! ࢿࠁ (ML Research Scientist, Pingpong)
[email protected]
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
zeta introduction
scatterlab
0
2.2k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
scatterlab
0
4.6k
Sparse, Dense, and Attentional Representations for Text Retrieval
scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
scatterlab
0
2.4k
What Can Neural Networks Reason About?
scatterlab
0
2.4k
Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer
scatterlab
0
2.3k
Other Decks in Research
See All in Research
Méthane 2030 : Développer des solutions de réduction des émissions de méthane entérique : 1ers résultats
institutdelelevage
PRO
0
120
幾何学的深層学習入門 ~3次元構造と対称性を学習するニューラルネットワーク~
taniai
3
1.1k
Prithvi-EO-2.0
satai
3
120
生成AIなんでも展示会vol6 LT登壇資料 NexteraBERT
rikkabotan7
0
220
J-STAGEの現況と全文XML登載必須化について
xspa2012
0
410
【令和7年度版】内航船 VS 外航船 早見表
anikisencyo
0
220
山中市政の総括:相反する二つの顔
sishi
0
240
AI創薬の壁
kanojikajino
1
1.3k
Physical AIでモデリングはどう変わるか/How Physical AI Will Change Modeling
stktu
0
350
実例から見るLLMのマンガ理解:実務VQAタスクによる長期的文脈と視覚情報の定性評価
kzmssk
0
230
[UAI2026, Tutorial Talk] Tensor Networks for Density Estimation: The merits of low-rank structures, tractable inference, and uncertainty quantification
gkazunii
0
130
[Research Paper Analysis] SigLIP2
auranightsky
0
160
Featured
See All Featured
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
akashhashmi
0
550
Prompt Engineering for Job Search
mfonobong
0
590
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
28
3.8k
Redefining SEO in the New Era of Traffic Generation
szymonslowik
1
530
Un-Boring Meetings
codingconduct
0
500
Primal Persuasion: How to Engage the Brain for Learning That Lasts
tmiket
0
520
jQuery: Nuts, Bolts and Bling
dougneiner
66
8.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
25
2.2k
Bootstrapping a Software Product
garrettdimon
PRO
306
120k
Building a Scalable Design System with Sketch
lauravandoore
464
34k
Claude Code のすすめ
schroneko
67
230k