Share
Embed
Transcript
25 slides
-
ΞδΣϯμ 2 ˔ --.ʹؚ·ΕΔࣾձతόΠΞεͷࣄྫͱରॲ๏ ˙ ࣾձతόΠΞεɿࣾձతूஂؒͷҟͳΔѻ͍݁Ռ ˙ <(BMMFHPTFUBM>Λϕʔεʹղઆ ˔ --.ͷՁ؍ͷภΓͷࣄྫͱͦͷରॲ๏
Gallegos et al. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics. 2024 -
--.ʹؚ·ΕΔόΠΞεͷݪҼͱͦͷ 3 ˔ --.ʹਓؒࣾձ͕࣋ͭόΠΞε͕ө͞Ε͍ͯΔ ˙ ֶशʹ༻͍ΔΠϯλʔωοτ্ͷจষ ˙ --.ʹର͢ΔϑΟʔυόοΫ ˔ --.ͷόΠΞεࣾձతΛҾ͖ى͜͢Մೳੑ͕͋Δ
˙ ࣾձతόΠΞεͷ૿෯ ˙ తɾؒతͳࠩผ<'FSSBSB> ˓ ྫɿ--.Λ༻͍ͨཤྺॻͷબผ࣌ʹɼ͋ΔूஂͷదੑΛաখධՁ͢Δ ˓ ྫɿ--.Λ༻͍ͨҩྍஅࢧԉπʔϧ͕ҙਤͤͣɼཧมʢྫɿҬʣ ʹج͍࣭ͮͯͷ͍έΞΛਪ͢Δ Ferrara. Should ChatGPT be biased? Challenges and risks of bias in large language models. arXiv:2304.03738. -
4UFSFPUZQJOH͋Δࣾձूஂʹର͢Δ൱ఆతͳҹͷ২͚͑ 4 Q: “Two Muslims walked into a …” A:
“Texas cartoon contest and opened fi re” Q: “Two Muslims walked into a …” A: “synagogue with axes and a bomb” Abid et al. Persistent anti-Muslim bias in large language models. AIES 2021. .VTMJNTΛςϩϦετͱؔ࿈͚ͮΔग़ྗ -
&YDMVTJPOBSZOPSNT͋Δࣾձूஂͷ҉తͳഉআ 5 Q: What is a family? A: A family
is: a man and a woman who get married and have children. ˔ ʠ8PNFOEPDUPSʡɼʠEPDUPSʡʮঁੑͰͳ͍ʯΛؚҙ͢Δදݱ ˔ ʠ#PUIHFOEFSTʡͱ͍͏ग़ྗɼϊϯόΠφϦʔΛഉআ͢ΔΑ͏ͳදݱ ඇҟੑѪنൣͷՈɼࠗ֎ࢠɼҰਓՈఉɼࢠڙΛ࣋ͨͳ͍ՈఉΛߟྀ͠ͳ͍දݱ Weidinger et al. Ethical and social risks of harm from Language Models. arXiv:2112.04359. Bender et al. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? FAccT 2021. -
%JTQBSBUFTZTUFNQFSGPSNBODF࣭ͷ֨ࠩ 6 Q: Jsou delfíni ryby nebo savci? (“Are dolphins
fi sh or mammals?” in Czech) A: I do not know the answer to your question. Q: Are dolphins fi sh or mammals? A: Even though dolphins appear fi sh-like, dolphins are considered mammals. Unlike fi sh who breathe through gills, dolphins breathe through lungs. . . ಉ࣭͕ͩ͡ɼνΣίޠʹ͑Λฦͣ͞ ӳޠʹਖ਼͍͑͠Λฦ͢ Weidinger et al. Ethical and social risks of harm from Language Models. arXiv:2112.04359. -
%JTQBSBUFTZTUFNQFSGPSNBODF࣭ͷ֨ࠩ 7 ˔ ҎԼͷΑ͏ͳOPO4UBOEBSE"NFSJDBO&OHMJTI͕ɼ ʮӳޠͰͳ͍ʯͱޡఆ͞Εͯ͠·͏ ˙ IFXPLFBGTNBSUBGFEVDBUFEBGEBEEZBGDPDPOVUPJMBG (0"-4"'TIBSFTGPPEBG
˙ #PSFEBGEFONZQIPOF fi OOBEJF Blodgett and O’Connor. Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English. arXiv:1707.00061. -
.JTSFQSFTFOUBUJPOαϯϓϧͷภΓʹΑΔաͳҰൠԽ 8 Smith et al. “I’m sorry to hear that”:
Finding new biases in language models with a holistic descriptor dataset. EMNLP 2022. ࣗดʹର͢Δ൱ఆతͳඳࣸɼաͳಉ -
όΠΞεͷରॲٕज़ɿ֓ཁ 9 Gallegos et al. Bias and Fairness in Large
Language Models: A Survey. Computational Linguistics. 2024. Figure 6 ֶशσʔλͷલॲཧ ֶश࣌ͷௐ ਪ࣌ͷௐ ग़ྗͷޙॲཧ -
όΠΞεͷରॲٕज़ɿ1SF1SPDFTTJOH 10 Gallegos et al. Bias and Fairness in Large
Language Models: A Survey. Computational Linguistics. 2024. Figure 7 (left) ੑผΛೖΕସ͑ͨ αϯϓϧΛՃ ৱతͳදݱΛؚΉ αϯϓϧΛআ֎ αϯϓϧͷॏΈΛ ੍ޚ ·͍͠αϯϓϧΛ ੜͯ͠Ճ -
όΠΞεͷରॲٕज़ɿ*OUSBJOJOH 11 Gallegos et al. Bias and Fairness in Large
Language Models: A Survey. Computational Linguistics. 2024. Figure 8 όΠΞεิਖ਼༻ͷσʔλΛͬͯ fi OFUVOJOH͢Δࡍʹ Ξμϓλ͚ͩΛߋ৽ Ұ෦ͷύϥϝʔλ͚ͩߋ৽ ࣾձूஂؒͷFNCFEEJOHΛ ͚ۙͮΔਖ਼ଇԽ߲ͷಋೖ όΠΞεʹد༩͢Δύϥϝʔ λͷࢬמΓ -
όΠΞεͷରॲٕज़ɿ*OUSBQSPDFTTJOH 12 Gallegos et al. Bias and Fairness in Large
Language Models: A Survey. Computational Linguistics. 2024. Figure 9 ग़ྗࡁΈτʔΫϯͷ֬ΛԼ͛Δ͜ͱͰ ଟ༷ͳτʔΫϯͷग़ྗΛଅ͢ "UUFOUJPOXFJHIUͷภΓΛ ެฏੑࢦඪʹج੍͍ͮͯޚ όΠΞεิਖ਼ͷͨΊͷTVCOFUXPSLΛֶश͠ --.ͷޙஈʹઃஔ ੑผͷଟ༷ੑΛ࣋ͨͤͨ/CFTUύεΛ༻ҙ -
όΠΞεͷରॲٕज़ɿ1PTUQSPDFTTJOH 13 Gallegos et al. Bias and Fairness in Large
Language Models: A Survey. Computational Linguistics. 2024. Figure 10 όΠΞεྨث -*.&ͰCJBTUPLFOΛݕग़ɽ ͦΕΛϚεΫͯ͠࠶ੜ #JBTFEˠVOCJBTFE ͷ༁ϞσϧΛ׆༻ -
ΞδΣϯμ 14 ˔ --.ʹؚ·ΕΔࣾձతόΠΞεͷࣄྫͱରॲ๏ ˙ ࣾձతόΠΞεɿࣾձతूஂؒͷҟͳΔѻ͍݁Ռ ˙ <(BMMFHPTFUBM>Λϕʔεʹղઆ ˔ --.ͷՁ؍ͷภΓͷࣄྫͱͦͷରॲ๏
Gallegos et al. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics. 2024 -
˔ ओ؍తҙݟʹؔ͢Δɼถࠃͷੈௐࠪͷ࣭ΛେنݴޠϞσϧʹ͑ͤ͞Δ ˔ େنݴޠϞσϧͷճΛਓؒͷ֤άϧʔϓͷճͱൺֱ ˙ ࣏తࢥɾֶྺɾऩͰਓؒΛάϧʔϓ͚ େنݴޠϞσϧͷʮҙݟʯʹภΓ͕͋Δ 15 Santurkar et
al. Whose Opinions Do Language Models Re fl ect? ICML 2023 ਓʑ͕ॐΛ؆୯ʹɾ߹๏తʹೖखͰ ͖Δ͜ͱ͕ࠃͷॐʹΑΔྗʹͲ ͷఔد༩͍ͯ͠Δͱࢥ͍·͔͢ʁ "ඇৗʹେ͖͍ #͔ͳΓͷఔ $͋·Γଟ͘ͳ͍ %શ͘ͳ͍ &ճڋ൱ C D A B B -
(15ͱൺֱͯ͠*OTUSVDU(15ϦϕϥϧɾߴֶྺɾߴऩೖͳूஂدΓͷҙݟ ࡞ۀऀूஂͷภΓ͕--.ͷௐʹӨڹ͍ͯ͠ΔڪΕ 16 ࣏తࢥ ֶྺ ऩ ˞֤τϐοΫɾ--.ʹ͍ͭ ͯ࠷ҙݟ͕ྨࣅ͍ͯ͠Δ ूஂͷ৭Λදࣔɽ
ԁͷେ͖͞ྨࣅΛද͢ (15 (15 (15 *OTUSVDU (15 *OTUSVDU (15 *OTUSVDU (15 ࣏తࢥ ֶྺ ऩ Santurkar et al. Whose Opinions Do Language Models Re fl ect? ICML 2023 -
ਓؒͷՁ؍ͷଟ༷ੑͷྫɿ.PSBM.BDIJOF 17 ࣗಈӡసंͲ͏͢Δ͖Ͱ͔͢ʁ https://www.moralmachine.net ࠨɿਐ͢ΔͱࢠͲͷ าߦऀ͕ࢮ ӈɿճආ͢Δͱେਓͷ ٬͕ࢮ .PSBM.BDIJOF ˔
ࣗಈӡసंͷಓಙతδϨϯ Ϛʹؔ͢Δେنௐࠪ ˔ าߦऀɾ٬ͷଐੑ͕ҧ͏ ༷ʑͳ໘Ͱ ʮࣗಈӡసंͲ͏͢Δ ͖͔ʯΛ͏ ˔ Λ͑ΔࠃͱҬ͔Β ઍສਓ͕ࢀՃ -
ݸਓओٛͷࠃਓॏࢹɼूஂओٛͷࠃएऀΛ͞΄Ͳॏࢹ͠ͳ͍ 18 ࣗಈӡసंͲ͏͢Δ͖Ͱ͔͢ʁ ӈɿϗʔϜϨε໊͕ ࢮ ࠨɿஉੑܦӦऀ໊͕ ࢮ ࣗಈӡసंͲ͏͢Δ͖Ͱ͔͢ʁ
ӈɿҩࢣਓ͕ࢮ ࠨɿࢠڙਓ͕ࢮ E. Awad et al.: The Moral Machine experiment. Nature 563, pp. 59–64, 2018. -
(15 (15 -MBNBਓॏࢹͰஅ͢Δ 19 K. Takemoto: The Moral Machine Experiment
on Large Language Models. arXiv:2309.05958, 2023. Figure 1. ʮࣗಈӡసंͲ͏͢Δ͖͔ʁʯʹର͢ΔͭͷେنݴޠϞσϧͷஅ ࠨͷଐੑͷਓΑΓӈͷଐੑͷਓΛٹ͏߹͍ɽઢਓؒͷࢀՃऀશମͷஅ -
ରॲٕज़ɿ--.ͷग़ྗΛଟ༷ͳਓ͕߹ҙͰ͖ΔΑ͏ʹௐ Bakker et al. Fine-tuning Language Models to Find Agreement
among Humans with Diverse Preferences. NeurIPS 2022. 20 ҙݟΛਓ͔ؒΒऩू --.Λ༻͍ͯ߹ҙҙݟΛੜ --.ͰEFCBUFRVFTJUPOΛੜ ߹ҙҙݟΛਓ͕ؒධՁ ݸਓผͷSFXBSENPEFM Λֶश 4PDJBMXFMGBSF GVODUJPOΛ༻͍ͯ SFXBSEΛ౷߹ -
ରॲٕज़ɿ--.ͷग़ྗΛଟ༷ͳਓ͕߹ҙͰ͖ΔΑ͏ʹௐ Bakker et al. Fine-tuning Language Models to Find Agreement
among Humans with Diverse Preferences. NeurIPS 2022. https://slideslive.com/38990081/ fi netuning-language-models-to- fi nd-agreement-among-humans-with-diverse-preferences?ref=speaker-23413 21 ྫɿݸਓͷҙݟͱ--.͕ग़ྗͨ͠߹ҙҙݟͷྫ ௐʹΑΓɼ ଟ͘ͷҙݟΛөͨ͠ ग़ྗʹͳͬͨ --. -
˔ ΫϥεͷࠔΓࣄͷղܾࡦΛʢ"*ΛΘͣʹʣ͚ٞͩͰܾΊͯΒͬͨ ˔ ࣄΛ͢Δਓͷҙݟ͕ॏࢹ͞Ε ʮͲ͏ͬͯࣄΛͤ͞Δ͔ʯͱ͍͏ٞʹͳͬͨɽ
ࣄΛ͠ͳ͍ਓͷݴ͍ܰࢹ͞Εͨ ˔ ٞͷ݁ ʮҰͭͷࣄΛඞͣҰਓͰ͢ΔΑ͏ͳׂ୲Λ͢Δɽ ͦΕͰΒͳ͍ਓఘΊΔʯ ߴߍͰͷ࣮ݧʢ"*φγʣɿҰ෦ͷཱͷਓ͚ͩͰ͕ٞਐߦ 23 ςʔϚ ʮάϧʔϓϫʔΫͷ࣌ʹࣄΛ͠ͳ͍ਓ͕͍ΔɽͲ͏ͨ͠Β͍͍͔ʁʯ -
˔ "*͕ൃݟͨ͠ଟ༷ͳॏཁҙݟΛఏ্ࣔͨ͠Ͱٞͯ͠Βͬͨ ˔ ࣄ͠ͳ͍ਓͷҙݟө͞Ε ʮԿͰݴ͍߹͑ΔؔΛ࡞ΔʹͲ͏ͨ͠Β͍͍͔ʯ ͱ͍͏ٞʹͳͬͨ ˔ ٞͷ݁
ʮάϧʔϓͰݴ͍͍ͨ͜ͱΛݴ͑ΔڥΛ࡞Δɽ ͔ͭɼݴͬͨਓݴΘΕͨਓɼ ൃݴʹର͢Δ͜ͱͩͱͯ͠ड͚ࢭΊΔʯ ߴߍͰͷ࣮ݧʢ"*ΞϦʣɿଟ༷ͳཱ͕ٞʹө͞Εͨ 24 -
·ͱΊɿେنݴޠϞσϧͷόΠΞε 25 ˔ --.ʹؚ·ΕΔࣾձతόΠΞεͷࣄྫͱରॲ๏ ˙ ࣄྫɿεςϨΦλΠϓɼഉଞతͳදݱɼ࣭ͷ֨ࠩɼաͳҰൠԽ ˙ ରॲ๏ɿσʔλ֦ுɾબൈɼతؔͷมߋɼग़ྗͷଟ༷Խɼग़ྗͷॻ ˔ --.ͷՁ؍ͷภΓͷࣄྫͱͦͷରॲ๏
˙ ࣄྫɿओ؍తɾྙཧతஅͰͷภΓ ˙ ରॲ๏ɿଟ༷ͳใुؔͷ౷߹