Adversarial Filters of Dataset Biases

A42dd3541cd40296dcd8a5e6b4a01bef?s=47 Scatter Lab Inc.
September 04, 2020

Adversarial Filters of Dataset Biases

A42dd3541cd40296dcd8a5e6b4a01bef?s=128

Scatter Lab Inc.

September 04, 2020
Tweet

Transcript

  1. Adversarial Filters of Dataset Biases ੢ࢿࠁ (ML Research Scientist, Pingpong)

  2. ݾର ݾର 1. োҳ੸ ߓ҃ 2. AFLite 1. ৘द: WinoGrande

    ؘ੉ఠࣇ 2. ੌ߈ചػ ঌҊ્ܻ 3. प೷ 1. Synthetic Data 2. NLP 3. Vision
  3. োҳ੸ ߓ҃ োҳ੸ ߓ҃

  4. ‘߮஖݃௼ ؘ੉ఠࣇীࢲ ֫਷ ࢿמਸ ׳ࢿ೮׮Ҋ ೧׼ ޙઁܳ ೧Ѿ೮׮Ҋ ݈ೡ ࣻ

    ੓ਸө?’ • In-distribution పझ౟ࣇীࢲח ੜೞ૑݅ Out-of-distribution adversarial sampleীח ডೠ അ࢚ • Input-Output рী ੄ب஖ ঋ਷ Spurious correlation੉ ࢤ҂ӝ ٸޙ • ੉ܳ ೧Ѿೠ ؘ੉ఠࣇਸ ٜ݅যঠ दझమਸ ઁ؀۽ ಣоೡ ࣻ ੓਺ োҳ੸ ߓ҃ High Performance = Problem Solved?
  5. োҳ੗о ૒੽ domain-specificೠ spurious ಁఢਸ ࠙ܨ ߂ ੿੄ೞҊ ੉ܳ ઁѢೞח

    ߑध • োҳ੗੄ domain-specificೠ ૑धҗ ૒ҙী ੄ઓ • ঌҊ્ܻ ࢸ҅੗о ޷୊ Ҋ۰ೞ૑ ޅೠ biasח ழߡ ࠛо োҳ੸ ߓ҃ Previous Approaches
  6. AFLite AFLite

  7. • ޙ੢ীࢲ ؀ݺࢎо оܻఃח ؀࢚ਸ ݏ൤ח ޙઁ • SOTA ੿ഛب

    ড 90% → ݽ؛੉ Spurious correlationਸ ੉ਊೞח ѱ ইקө? • (3), (4)ח ߃઴ ஘ հ݈੉ ੿׹җ ҙ۲ ੓ਸ ഛܫ੉ ֫ই Word association݅ਵ۽ ޙઁܳ ಽ ࣻ ੓਺ AFLite Winograd Schema Challenge (WSC)
  8. • ࢎۈ੉ ૒੽ ؘ੉ఠࣇਸ ٜ݅ݶ ੉۠ Annotation artifactী ੄ೠ Biasܳ

    ೖೞӝ য۰਑ • AFLite۽ ೙ఠ݂ೠ WinoGrande ؘ੉ఠࣇ਷ ݽ؛ ੿ഛبب ծҊ ׮ܲ ߮஖݃௼۽ Transferب ੜؽ AFLite WinoGrande Dataset
  9. 1. ؘ੉ఠ੄ ੌࠗ݅ਵ۽ RoBERTa fine-tune 2. Splitਸ ׳ܻ ೞݶࢲ RoBERTa

    feature۽ linear classifier ೟ण 3. Split పझ౟ࣇীࢲ ੐߬٬݅ਵ۽ ׹ਸ औѱ ଺ਸ ࣻ ੓ח૑ పझ ౟ೞҊ ੋझఢझ߹۽ ঔ࢚࠶ ࣇী ୶о 4. ৈ۞ linear classifierо ੿׹ਸ ݏ൦ ࠺ਯ੉ Thresholdܳ ֈח Ѫ ઺ Top-kѐܳ ୭ઙ ؘ੉ఠࣇীࢲ ઁ৻ 5. ઁ৻غח ѐࣻо kѐо উ غѢա ਗೞח ௼ӝ੄ ؘ੉ఠࣇ੉ ؼ ٸ ө૑ 2~4 ߈ࠂ AFLite AFLite in WinoGrande
  10. • ױয੄ ӓࢿ݅ਵ۽ ಽ ࣻ ੓ח ޙઁܳ Ѧ۞ն • ੉ח

    ష௾ ۨ߰੄ Biasۄӝࠁ׮ח ҳઑ੸ੋ Ѫ੉޲۽ lexical-level heuristicਵ۽ח Ѧ۞ղӝ ൨ٝ AFLite Filtered Examples
  11. • AFLiteܳ ৈ۞ بݫੋਵ۽ ഛ੢ೞҊ model-agnosticೞѱ ੌ߈ച • Contributions: 1.

    ੉࢚੸੉૑݅ intractableೠ AFOptܳ AFLite۽ Ӕࢎೡ ࣻ ੓਺ਸ ࠁੋ׮. (Skip) 2. Vision, NLP ࠙ঠ੄ ৈ۞ ؘ੉ఠࣇীࢲ प೷೧ AFLite੄ ਬബࢿਸ ّ߉ஜೠ׮. 3. Biasܳ হঙ ؘ੉ఠࣇਵ۽ ೟णೠ ݽ؛੉ ੌ߈ചо ੜؽਸ प೷੸ਵ۽ ࠁੋ׮. 4. AFLite۽ ೙ఠ݂ೞݶ ؊ ب੹੸ੋ ߮஖݃௼ ؘ੉ఠࣇਸ ٜ݅ ࣻ ੓਺ਸ ࠁੋ׮. AFLite Adversarial Filters of Dataset Biases
  12. : any feature extractor : a family of classification models

    Φ M AFLite AFLite (Generalized)
  13. Experiments Experiments

  14. Biasing Dataset • Class-specificೠ ੋҕ featureܳ ؘ੉ఠ੄ 75%ী ઱ੑ, աݠ૑ח

    random feature ઱ੑ • Biased sample ઺ ੌࠗח ۨ੉࠶ ߄Է Results • Linear classifier۽ب ֫਷ ࢿמ ׳ࢿ • AFLiteܳ ੸ਊೞݶ ࢚ध੸ੋ ࢿמਵ۽ جই১ Experiments Synthetic Data
  15. • प೷ ؀࢚: SNLI੄ annotation artifactܳ ೖೠ out-of-distribution ؘ੉ఠࣇ 3ઙ

    • Non-entailment ઺ ޙઁ ਬഋ߹۽ Zero-shot పझ౟ Experiments NLP: Out-of-distribution Generalization
  16. AFLite۽ ೙ఠ݂ೠ ؘ੉ఠࣇ਷ ݽٚ ݽ؛ীࢲ ࢿמ੉ ௼ѱ ڄয૗ Experiments In-distribution

    Benchmark Re-estimation: SNLI
  17. Experiments In-distribution Benchmark Re-estimation: MultiNLI & QNLI

  18. • : ImageNet ؘ੉ఠࣇ੄ 20%۽ ೟णೠ EfficientNet-B7 feature • ImageNet-A۽

    ಣоೞפ AFLite-filtered ؘ੉ఠࣇਵ۽ ೟ण೮ਸ ٸ ࢿמ੉ ؊ જ਺ Φ Experiments Vision: Adversarial Image Classification
  19. ImageNet੄ dev setਸ ೙ఠ݂ೞҊ ಣо೮ਸ ٸ ࢿמ ೞۅ੉ ؊ ఀ

    Experiments In-distribution Image Classification
  20. ӝઓীب ࠁҊػ ౠ੿ ನૉী ؀ೠ Bias, ݽনࠁ׮ ૕х݅ਵ۽ ҳ࠙ೞח ޙઁ

    ١җ Ѿਸ э੉ೣ Experiments Filtered Examples
  21. • Adversarial Filtering SWAG: A Large-Scale Adversarial Dataset for Grounded

    Commonsense Inference [EMNLP’18] HellaSwag: Can a Machine Really Finish Your Sentence? [ACL’19] • AFLite WinoGrande: An Adversarial Winograd Schema Challenge at Scale [arXiv’19] Adversarial Filters of Dataset Biases [ICML’20] References References
  22. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽

    োۅ ઱ࣁਃ! ੢ࢿࠁ (ML Research Scientist, Pingpong) Email.seongbo@scatterlab.co.kr