Adversarial Filters of Dataset Biases

Transcript

1. Adversarial Filters of Dataset Biases ੢ࢿࠁ (ML Research Scientist, Pingpong)

2. ݾର ݾର 1. োҳ੸ ߓ҃ 2. AFLite 1. ৘द: WinoGrande

   ؘ੉ఠࣇ 2. ੌ߈ചػ ঌҊ્ܻ 3. प೷ 1. Synthetic Data 2. NLP 3. Vision

3. োҳ੸ ߓ҃ োҳ੸ ߓ҃

4. ‘߮஖݃௼ ؘ੉ఠࣇীࢲ ֫਷ ࢿמਸ ׳ࢿ೮׮Ҋ ೧׼ ޙઁܳ ೧Ѿ೮׮Ҋ ݈ೡ ࣻ

   ੓ਸө?’ • In-distribution పझ౟ࣇীࢲח ੜೞ૑݅ Out-of-distribution adversarial sampleীח ডೠ അ࢚ • Input-Output рী ੄ب஖ ঋ਷ Spurious correlation੉ ࢤ҂ӝ ٸޙ • ੉ܳ ೧Ѿೠ ؘ੉ఠࣇਸ ٜ݅যঠ दझమਸ ઁ؀۽ ಣоೡ ࣻ ੓਺ োҳ੸ ߓ҃ High Performance = Problem Solved?

5. োҳ੗о ૒੽ domain-specificೠ spurious ಁఢਸ ࠙ܨ ߂ ੿੄ೞҊ ੉ܳ ઁѢೞח

   ߑध • োҳ੗੄ domain-specificೠ ૑धҗ ૒ҙী ੄ઓ • ঌҊ્ܻ ࢸ҅੗о ޷୊ Ҋ۰ೞ૑ ޅೠ biasח ழߡ ࠛо োҳ੸ ߓ҃ Previous Approaches

6. AFLite AFLite

7. • ޙ੢ীࢲ ؀ݺࢎо оܻఃח ؀࢚ਸ ݏ൤ח ޙઁ • SOTA ੿ഛب

   ড 90% → ݽ؛੉ Spurious correlationਸ ੉ਊೞח ѱ ইקө? • (3), (4)ח ߃઴ ஘ հ݈੉ ੿׹җ ҙ۲ ੓ਸ ഛܫ੉ ֫ই Word association݅ਵ۽ ޙઁܳ ಽ ࣻ ੓਺ AFLite Winograd Schema Challenge (WSC)

8. • ࢎۈ੉ ૒੽ ؘ੉ఠࣇਸ ٜ݅ݶ ੉۠ Annotation artifactী ੄ೠ Biasܳ

   ೖೞӝ য۰਑ • AFLite۽ ೙ఠ݂ೠ WinoGrande ؘ੉ఠࣇ਷ ݽ؛ ੿ഛبب ծҊ ׮ܲ ߮஖݃௼۽ Transferب ੜؽ AFLite WinoGrande Dataset

9. 1. ؘ੉ఠ੄ ੌࠗ݅ਵ۽ RoBERTa fine-tune 2. Splitਸ ׳ܻ ೞݶࢲ RoBERTa

   feature۽ linear classifier ೟ण 3. Split పझ౟ࣇীࢲ ੐߬٬݅ਵ۽ ׹ਸ औѱ ଺ਸ ࣻ ੓ח૑ పझ ౟ೞҊ ੋझఢझ߹۽ ঔ࢚࠶ ࣇী ୶о 4. ৈ۞ linear classifierо ੿׹ਸ ݏ൦ ࠺ਯ੉ Thresholdܳ ֈח Ѫ ઺ Top-kѐܳ ୭ઙ ؘ੉ఠࣇীࢲ ઁ৻ 5. ઁ৻غח ѐࣻо kѐо উ غѢա ਗೞח ௼ӝ੄ ؘ੉ఠࣇ੉ ؼ ٸ ө૑ 2~4 ߈ࠂ AFLite AFLite in WinoGrande

10. • ױয੄ ӓࢿ݅ਵ۽ ಽ ࣻ ੓ח ޙઁܳ Ѧ۞ն • ੉ח

    ష௾ ۨ߰੄ Biasۄӝࠁ׮ח ҳઑ੸ੋ Ѫ੉޲۽ lexical-level heuristicਵ۽ח Ѧ۞ղӝ ൨ٝ AFLite Filtered Examples

11. • AFLiteܳ ৈ۞ بݫੋਵ۽ ഛ੢ೞҊ model-agnosticೞѱ ੌ߈ച • Contributions: 1.

    ੉࢚੸੉૑݅ intractableೠ AFOptܳ AFLite۽ Ӕࢎೡ ࣻ ੓਺ਸ ࠁੋ׮. (Skip) 2. Vision, NLP ࠙ঠ੄ ৈ۞ ؘ੉ఠࣇীࢲ प೷೧ AFLite੄ ਬബࢿਸ ّ߉ஜೠ׮. 3. Biasܳ হঙ ؘ੉ఠࣇਵ۽ ೟णೠ ݽ؛੉ ੌ߈ചо ੜؽਸ प೷੸ਵ۽ ࠁੋ׮. 4. AFLite۽ ೙ఠ݂ೞݶ ؊ ب੹੸ੋ ߮஖݃௼ ؘ੉ఠࣇਸ ٜ݅ ࣻ ੓਺ਸ ࠁੋ׮. AFLite Adversarial Filters of Dataset Biases

12. : any feature extractor : a family of classification models

    Φ M AFLite AFLite (Generalized)

13. Experiments Experiments

14. Biasing Dataset • Class-specificೠ ੋҕ featureܳ ؘ੉ఠ੄ 75%ী ઱ੑ, աݠ૑ח

    random feature ઱ੑ • Biased sample ઺ ੌࠗח ۨ੉࠶ ߄Է Results • Linear classifier۽ب ֫਷ ࢿמ ׳ࢿ • AFLiteܳ ੸ਊೞݶ ࢚ध੸ੋ ࢿמਵ۽ جই১ Experiments Synthetic Data

15. • प೷ ؀࢚: SNLI੄ annotation artifactܳ ೖೠ out-of-distribution ؘ੉ఠࣇ 3ઙ

    • Non-entailment ઺ ޙઁ ਬഋ߹۽ Zero-shot పझ౟ Experiments NLP: Out-of-distribution Generalization

16. AFLite۽ ೙ఠ݂ೠ ؘ੉ఠࣇ਷ ݽٚ ݽ؛ীࢲ ࢿמ੉ ௼ѱ ڄয૗ Experiments In-distribution

    Benchmark Re-estimation: SNLI

17. Experiments In-distribution Benchmark Re-estimation: MultiNLI & QNLI

18. • : ImageNet ؘ੉ఠࣇ੄ 20%۽ ೟णೠ EfficientNet-B7 feature • ImageNet-A۽

    ಣоೞפ AFLite-filtered ؘ੉ఠࣇਵ۽ ೟ण೮ਸ ٸ ࢿמ੉ ؊ જ਺ Φ Experiments Vision: Adversarial Image Classification

19. ImageNet੄ dev setਸ ೙ఠ݂ೞҊ ಣо೮ਸ ٸ ࢿמ ೞۅ੉ ؊ ఀ

    Experiments In-distribution Image Classification

20. ӝઓীب ࠁҊػ ౠ੿ ನૉী ؀ೠ Bias, ݽনࠁ׮ ૕х݅ਵ۽ ҳ࠙ೞח ޙઁ

    ١җ Ѿਸ э੉ೣ Experiments Filtered Examples

21. • Adversarial Filtering SWAG: A Large-Scale Adversarial Dataset for Grounded

    Commonsense Inference [EMNLP’18] HellaSwag: Can a Machine Really Finish Your Sentence? [ACL’19] • AFLite WinoGrande: An Adversarial Winograd Schema Challenge at Scale [arXiv’19] Adversarial Filters of Dataset Biases [ICML’20] References References

22. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽

    োۅ ઱ࣁਃ! ੢ࢿࠁ (ML Research Scientist, Pingpong) Email.seongbo@scatterlab.co.kr