Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adversarial Filters of Dataset Biases

Scatter Lab Inc.
September 04, 2020

Adversarial Filters of Dataset Biases

Scatter Lab Inc.

September 04, 2020
Tweet

More Decks by Scatter Lab Inc.

Other Decks in Research

Transcript

  1. ݾର ݾର 1. োҳ੸ ߓ҃ 2. AFLite 1. ৘द: WinoGrande

    ؘ੉ఠࣇ 2. ੌ߈ചػ ঌҊ્ܻ 3. प೷ 1. Synthetic Data 2. NLP 3. Vision
  2. ‘߮஖݃௼ ؘ੉ఠࣇীࢲ ֫਷ ࢿמਸ ׳ࢿ೮׮Ҋ ೧׼ ޙઁܳ ೧Ѿ೮׮Ҋ ݈ೡ ࣻ

    ੓ਸө?’ • In-distribution పझ౟ࣇীࢲח ੜೞ૑݅ Out-of-distribution adversarial sampleীח ডೠ അ࢚ • Input-Output рী ੄ب஖ ঋ਷ Spurious correlation੉ ࢤ҂ӝ ٸޙ • ੉ܳ ೧Ѿೠ ؘ੉ఠࣇਸ ٜ݅যঠ दझమਸ ઁ؀۽ ಣоೡ ࣻ ੓਺ োҳ੸ ߓ҃ High Performance = Problem Solved?
  3. োҳ੗о ૒੽ domain-specificೠ spurious ಁఢਸ ࠙ܨ ߂ ੿੄ೞҊ ੉ܳ ઁѢೞח

    ߑध • োҳ੗੄ domain-specificೠ ૑धҗ ૒ҙী ੄ઓ • ঌҊ્ܻ ࢸ҅੗о ޷୊ Ҋ۰ೞ૑ ޅೠ biasח ழߡ ࠛо োҳ੸ ߓ҃ Previous Approaches
  4. • ޙ੢ীࢲ ؀ݺࢎо оܻఃח ؀࢚ਸ ݏ൤ח ޙઁ • SOTA ੿ഛب

    ড 90% → ݽ؛੉ Spurious correlationਸ ੉ਊೞח ѱ ইקө? • (3), (4)ח ߃઴ ஘ հ݈੉ ੿׹җ ҙ۲ ੓ਸ ഛܫ੉ ֫ই Word association݅ਵ۽ ޙઁܳ ಽ ࣻ ੓਺ AFLite Winograd Schema Challenge (WSC)
  5. • ࢎۈ੉ ૒੽ ؘ੉ఠࣇਸ ٜ݅ݶ ੉۠ Annotation artifactী ੄ೠ Biasܳ

    ೖೞӝ য۰਑ • AFLite۽ ೙ఠ݂ೠ WinoGrande ؘ੉ఠࣇ਷ ݽ؛ ੿ഛبب ծҊ ׮ܲ ߮஖݃௼۽ Transferب ੜؽ AFLite WinoGrande Dataset
  6. 1. ؘ੉ఠ੄ ੌࠗ݅ਵ۽ RoBERTa fine-tune 2. Splitਸ ׳ܻ ೞݶࢲ RoBERTa

    feature۽ linear classifier ೟ण 3. Split పझ౟ࣇীࢲ ੐߬٬݅ਵ۽ ׹ਸ औѱ ଺ਸ ࣻ ੓ח૑ పझ ౟ೞҊ ੋझఢझ߹۽ ঔ࢚࠶ ࣇী ୶о 4. ৈ۞ linear classifierо ੿׹ਸ ݏ൦ ࠺ਯ੉ Thresholdܳ ֈח Ѫ ઺ Top-kѐܳ ୭ઙ ؘ੉ఠࣇীࢲ ઁ৻ 5. ઁ৻غח ѐࣻо kѐо উ غѢա ਗೞח ௼ӝ੄ ؘ੉ఠࣇ੉ ؼ ٸ ө૑ 2~4 ߈ࠂ AFLite AFLite in WinoGrande
  7. • ױয੄ ӓࢿ݅ਵ۽ ಽ ࣻ ੓ח ޙઁܳ Ѧ۞ն • ੉ח

    ష௾ ۨ߰੄ Biasۄӝࠁ׮ח ҳઑ੸ੋ Ѫ੉޲۽ lexical-level heuristicਵ۽ח Ѧ۞ղӝ ൨ٝ AFLite Filtered Examples
  8. • AFLiteܳ ৈ۞ بݫੋਵ۽ ഛ੢ೞҊ model-agnosticೞѱ ੌ߈ച • Contributions: 1.

    ੉࢚੸੉૑݅ intractableೠ AFOptܳ AFLite۽ Ӕࢎೡ ࣻ ੓਺ਸ ࠁੋ׮. (Skip) 2. Vision, NLP ࠙ঠ੄ ৈ۞ ؘ੉ఠࣇীࢲ प೷೧ AFLite੄ ਬബࢿਸ ّ߉ஜೠ׮. 3. Biasܳ হঙ ؘ੉ఠࣇਵ۽ ೟णೠ ݽ؛੉ ੌ߈ചо ੜؽਸ प೷੸ਵ۽ ࠁੋ׮. 4. AFLite۽ ೙ఠ݂ೞݶ ؊ ب੹੸ੋ ߮஖݃௼ ؘ੉ఠࣇਸ ٜ݅ ࣻ ੓਺ਸ ࠁੋ׮. AFLite Adversarial Filters of Dataset Biases
  9. Biasing Dataset • Class-specificೠ ੋҕ featureܳ ؘ੉ఠ੄ 75%ী ઱ੑ, աݠ૑ח

    random feature ઱ੑ • Biased sample ઺ ੌࠗח ۨ੉࠶ ߄Է Results • Linear classifier۽ب ֫਷ ࢿמ ׳ࢿ • AFLiteܳ ੸ਊೞݶ ࢚ध੸ੋ ࢿמਵ۽ جই১ Experiments Synthetic Data
  10. • प೷ ؀࢚: SNLI੄ annotation artifactܳ ೖೠ out-of-distribution ؘ੉ఠࣇ 3ઙ

    • Non-entailment ઺ ޙઁ ਬഋ߹۽ Zero-shot పझ౟ Experiments NLP: Out-of-distribution Generalization
  11. • : ImageNet ؘ੉ఠࣇ੄ 20%۽ ೟णೠ EfficientNet-B7 feature • ImageNet-A۽

    ಣоೞפ AFLite-filtered ؘ੉ఠࣇਵ۽ ೟ण೮ਸ ٸ ࢿמ੉ ؊ જ਺ Φ Experiments Vision: Adversarial Image Classification
  12. ImageNet੄ dev setਸ ೙ఠ݂ೞҊ ಣо೮ਸ ٸ ࢿמ ೞۅ੉ ؊ ఀ

    Experiments In-distribution Image Classification
  13. • Adversarial Filtering SWAG: A Large-Scale Adversarial Dataset for Grounded

    Commonsense Inference [EMNLP’18] HellaSwag: Can a Machine Really Finish Your Sentence? [ACL’19] • AFLite WinoGrande: An Adversarial Winograd Schema Challenge at Scale [arXiv’19] Adversarial Filters of Dataset Biases [ICML’20] References References