Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Adversarial Filters of Dataset Biases
Scatter Lab Inc.
September 04, 2020
Research
0
1.1k
Adversarial Filters of Dataset Biases
Scatter Lab Inc.
September 04, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
scatterlab
0
1.3k
scatterlab
0
1.1k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.3k
scatterlab
0
1.3k
scatterlab
0
1.3k
Other Decks in Research
See All in Research
supikiti
0
220
kumagallium
1
100
ksudoh
0
150
yuukit
1
640
takuyaa
0
130
nzw0301
2
280
racterub
0
140
kayceesrk
0
1.1k
yuukit
0
1.1k
highlow
0
250
shinyorke
1
320
kayceesrk
2
3.4k
Featured
See All Featured
lynnandtonic
270
16k
notwaldorf
13
1.5k
paulrobertlloyd
71
1.4k
moore
125
21k
denniskardys
220
120k
erikaheidi
13
4.2k
bryan
100
11k
robhawkes
52
2.8k
scottboms
251
11k
mojombo
358
62k
hatefulcrawdad
257
17k
tenderlove
52
3.4k
Transcript
Adversarial Filters of Dataset Biases ࢿࠁ (ML Research Scientist, Pingpong)
ݾର ݾର 1. োҳ ߓ҃ 2. AFLite 1. द: WinoGrande
ؘఠࣇ 2. ੌ߈ചػ ঌҊ્ܻ 3. प 1. Synthetic Data 2. NLP 3. Vision
োҳ ߓ҃ োҳ ߓ҃
‘߮݃ ؘఠࣇীࢲ ֫ ࢿמਸ ׳ࢿ೮Ҋ ೧ ޙઁܳ ೧Ѿ೮Ҋ ݈ೡ ࣻ
ਸө?’ • In-distribution పझࣇীࢲח ੜೞ݅ Out-of-distribution adversarial sampleীח ডೠ അ࢚ • Input-Output рী ب ঋ Spurious correlation ࢤ҂ӝ ٸޙ • ܳ ೧Ѿೠ ؘఠࣇਸ ٜ݅যঠ दझమਸ ઁ۽ ಣоೡ ࣻ োҳ ߓ҃ High Performance = Problem Solved?
োҳо domain-specificೠ spurious ಁఢਸ ࠙ܨ ߂ ೞҊ ܳ ઁѢೞח
ߑध • োҳ domain-specificೠ धҗ ҙী ઓ • ঌҊ્ܻ ࢸ҅о Ҋ۰ೞ ޅೠ biasח ழߡ ࠛо োҳ ߓ҃ Previous Approaches
AFLite AFLite
• ޙীࢲ ݺࢎо оܻఃח ࢚ਸ ݏח ޙઁ • SOTA ഛب
ড 90% → ݽ؛ Spurious correlationਸ ਊೞח ѱ ইקө? • (3), (4)ח ߃ հ݈ җ ҙ۲ ਸ ഛܫ ֫ই Word association݅ਵ۽ ޙઁܳ ಽ ࣻ AFLite Winograd Schema Challenge (WSC)
• ࢎۈ ؘఠࣇਸ ٜ݅ݶ ۠ Annotation artifactী ೠ Biasܳ
ೖೞӝ য۰ • AFLite۽ ఠ݂ೠ WinoGrande ؘఠࣇ ݽ؛ ഛبب ծҊ ܲ ߮݃۽ Transferب ੜؽ AFLite WinoGrande Dataset
1. ؘఠ ੌࠗ݅ਵ۽ RoBERTa fine-tune 2. Splitਸ ׳ܻ ೞݶࢲ RoBERTa
feature۽ linear classifier ण 3. Split పझࣇীࢲ ߬٬݅ਵ۽ ਸ औѱ ਸ ࣻ ח పझ ೞҊ ੋझఢझ߹۽ ঔ࢚࠶ ࣇী ୶о 4. ৈ۞ linear classifierо ਸ ݏ൦ ࠺ਯ Thresholdܳ ֈח Ѫ Top-kѐܳ ୭ઙ ؘఠࣇীࢲ ઁ৻ 5. ઁ৻غח ѐࣻо kѐо উ غѢա ਗೞח ӝ ؘఠࣇ ؼ ٸ ө 2~4 ߈ࠂ AFLite AFLite in WinoGrande
• ױয ӓࢿ݅ਵ۽ ಽ ࣻ ח ޙઁܳ Ѧ۞ն • ח
ష ۨ߰ Biasۄӝࠁח ҳઑੋ Ѫ۽ lexical-level heuristicਵ۽ח Ѧ۞ղӝ ൨ٝ AFLite Filtered Examples
• AFLiteܳ ৈ۞ بݫੋਵ۽ ഛೞҊ model-agnosticೞѱ ੌ߈ച • Contributions: 1.
࢚݅ intractableೠ AFOptܳ AFLite۽ Ӕࢎೡ ࣻ ਸ ࠁੋ. (Skip) 2. Vision, NLP ࠙ঠ ৈ۞ ؘఠࣇীࢲ प೧ AFLite ਬബࢿਸ ّ߉ஜೠ. 3. Biasܳ হঙ ؘఠࣇਵ۽ णೠ ݽ؛ ੌ߈ചо ੜؽਸ पਵ۽ ࠁੋ. 4. AFLite۽ ఠ݂ೞݶ ؊ بੋ ߮݃ ؘఠࣇਸ ٜ݅ ࣻ ਸ ࠁੋ. AFLite Adversarial Filters of Dataset Biases
: any feature extractor : a family of classification models
Φ M AFLite AFLite (Generalized)
Experiments Experiments
Biasing Dataset • Class-specificೠ ੋҕ featureܳ ؘఠ 75%ী ੑ, աݠח
random feature ੑ • Biased sample ੌࠗח ۨ࠶ ߄Է Results • Linear classifier۽ب ֫ ࢿמ ׳ࢿ • AFLiteܳ ਊೞݶ ࢚धੋ ࢿמਵ۽ جই১ Experiments Synthetic Data
• प ࢚: SNLI annotation artifactܳ ೖೠ out-of-distribution ؘఠࣇ 3ઙ
• Non-entailment ޙઁ ਬഋ߹۽ Zero-shot పझ Experiments NLP: Out-of-distribution Generalization
AFLite۽ ఠ݂ೠ ؘఠࣇ ݽٚ ݽ؛ীࢲ ࢿמ ѱ ڄয Experiments In-distribution
Benchmark Re-estimation: SNLI
Experiments In-distribution Benchmark Re-estimation: MultiNLI & QNLI
• : ImageNet ؘఠࣇ 20%۽ णೠ EfficientNet-B7 feature • ImageNet-A۽
ಣоೞפ AFLite-filtered ؘఠࣇਵ۽ ण೮ਸ ٸ ࢿמ ؊ જ Φ Experiments Vision: Adversarial Image Classification
ImageNet dev setਸ ఠ݂ೞҊ ಣо೮ਸ ٸ ࢿמ ೞۅ ؊ ఀ
Experiments In-distribution Image Classification
ӝઓীب ࠁҊػ ౠ ನૉী ೠ Bias, ݽনࠁ х݅ਵ۽ ҳ࠙ೞח ޙઁ
١җ Ѿਸ эೣ Experiments Filtered Examples
• Adversarial Filtering SWAG: A Large-Scale Adversarial Dataset for Grounded
Commonsense Inference [EMNLP’18] HellaSwag: Can a Machine Really Finish Your Sentence? [ACL’19] • AFLite WinoGrande: An Adversarial Winograd Schema Challenge at Scale [arXiv’19] Adversarial Filters of Dataset Biases [ICML’20] References References
хࢎפ✌ ୶о ޙ ژח ҾӘೠ ݶ ઁٚ ইې োۅ۽
োۅ ࣁਃ! ࢿࠁ (ML Research Scientist, Pingpong) Email.seongbo@scatterlab.co.kr