JSAI NeurIPS 2024 参加報告会（AI アライメント）

Slide 1

Slide 1 text

ʲ/FVS*14ʳ "*ΞϥΠϝϯτྖҬͷ࠷৽ಈ޲ "LJGVNJ8BDIJ࿨஍ ྎྑ ୈճਓ޻஌ೳηϛφʔ ೥݄೔ 1

Slide 2

Slide 2 text

"LJGVNJ8BDIJʢ࿨஍ ྎྑʣ ܦྺ • r ɿ*#.౦ژجૅݚڀॴ 3FTFBSDI4DJFOUJTU • r ݱࡏɿ-*/&Ϡϑʔ $IJFG3FTFBSDI4DJFOUJTU ݚڀ෼໺ • ڧԽֶश ºʢ"*4BGFUZ ࣗવݴޠॲཧʣ ஶॻʢڞஶʣ • ʰڧԽֶश͔Β৴པͰ͖Δҙࢥܾఆ΁ʱ /FVS*14ͷ࠾୒࿦จ • ओஶɿ4UFQXJTF"MJHONFOUGPS$POTUSBJOFE-BOHVBHF.PEFM1PMJDZ0QUJNJ[BUJPOT • IUUQTBSYJWPSHBCT • ڞஶɿ'MJQQJOHCBTFE1PMJDZGPS$IBODF$POTUSBJOFE.BSLPW%FDJTJPO1SPDFTTFT • IUUQTBSYJWPSHBCT 2

Slide 3

Slide 3 text

ΞδΣϯμ /FVS*14ͷײ૝ɾงғؾ "*4BGFUZɾΞϥΠϝϯτ ݚڀͷைྲྀ 13*4."MJHONFOU%BUBTFU࿦จ঺հʢ#FTU1BQFSʣ • ,JSLFUBM5IF13*4."MJHONFOU%BUBTFU8IBU1BSUJDJQBUPSZ 3FQSFTFOUBUJWFBOE*OEJWJEVBMJTFE)VNBO'FFECBDL3FWFBMT"CPVUUIF 4VCKFDUJWFBOE.VMUJDVMUVSBM"MJHONFOUPG-BSHF-BOHVBHF.PEFMTz • IUUQTBSYJWPSHBCT "MJHOFS࿦จ঺հʢ0SBMʣ • +JFUBM"MJHOFS&GGJDJFOU"MJHONFOUCZ-FBSOJOHUP$PSSFDUz • IUUQTBSYJWPSHBCT 3

Slide 4

Slide 4 text

/FVS*14ͷײ૝ɾงғؾ 4 ݄ c ݄ c $IBU(15 1SP ೥ ೥ ݄ c (15 ݄ c %"--& ݄ c 4PSB ݄ c (15P ݄ c 0QFO"* P • /FVS*14ͱ /FVS*14Ͱ͔ͳΓงғؾ͕ҧ͏ʂ • /FVS*14ͷ։࠵λΠϛϯάΛ 0QFO"*ͷϦϦʔεͱॏͶΔͱɾɾɾ • ͳΜͳΒɺ࿦จʒ੾࣌ʢ݄ʣͱֶձ։࠵࣌ʢ݄ʣͰ΋ҧ͏ /FVS*14 /FVS*14 ˢ /FVS*14 ͷ࿦จʒ੾

Slide 5

Slide 5 text

"*4BGFUZݚڀͷมԽ 5 ײ૝̍ɿʮਅ݋͞ʯ͕૿ͨ͠ • ݚڀͱ࣮༻ͷڑ཭͕͍ۙͮͨ • "*ͷೳྗ͕ɺ࣮ੈքͰٻΊΒΕΔϨϕϧʹୡ͠͸͡Ί͍ͯΔ • --.ͳͲɺߴੑೳͳ "*Λѻ͏ݚڀͷׂ߹͕૿͑ͨ • "*͕ຊ౰ͷҙຯͰʮةݥͳ΋ͷʯʹͳ͔ͬͨΒ Goodfellow et al. "Generative adversarial nets." In NeurIPS (2014). Tian et al. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." In NeurIPS (2024). ೥౰࣌ͷੜ੒ը૾ ʢ(PPEGFMMPXΑΓഈआʣ ೥࣌఺Ͱͷੜ੒ը૾ ʢ5JBOΑΓഈआʣ ٕज़ͷਐาʹΑΓ ϦεΫ΋૿େ

Slide 6

Slide 6 text

"*4BGFUZݚڀͷมԽ 6 :PTIVB#FOHJP 4BGF(FOFSBUJWF"*XPSLTIPQ ײ૝̎ɿ"(*ɾ"4*ͷ஀ੜΛલఏʹݚڀ͍ͯ͠Δਓ͕૿͑ͨ • "*ͷʮࢦ਺ؔ਺తͳʯਐԽ • ࡢ೥·Ͱɿকདྷతͳ՝୊ͱͯٞ͠࿦ɻ"*͸੍ޚͰ͖Δͱ͍͏લఏ • ࠓ೥ɿ٤ۓͷ՝୊ͱͯٞ͠࿦ɻʮͦ΋ͦ΋੍ޚͰ͖ͳ͍ର৅͕஀ੜͨ͠Βʁʯ *MZB4VUTLFWFS 5FTUPGUJNF"XBSE

Slide 7

Slide 7 text

"*4BGFUZݚڀͷ෼෍ 7 • /FVS*14ެࣜͷ ࿦จՄࢹԽπʔϧ • IUUQTOFVSJQTWJ[IVCBJ • Ωʔϫʔυɿ4BGF • --.ʹ࿦จ͕ଟ͍ • ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ ʢ5FYUUP7JEFPͳͲʣ • ڧԽֶशͰ͸ࠜڧ͍ਓؾʢ݁ߏલ͔Βʣ --. ϚϧνϞʔμϧ ڧԽֶश

Slide 8

Slide 8 text

--. ϚϧνϞʔμϧ ڧԽֶश "*4BGFUZݚڀͷ෼෍ 8 ڧԽֶश --. --.Ͱ૿Ճ͕ݦஶ ˠ ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ

Slide 9

Slide 9 text

"*4BGFUZݚڀͷ෼෍ 9 • /FVS*14ެࣜͷ ࿦จՄࢹԽπʔϧ • IUUQTOFVSJQTWJ[IVCBJ • Ωʔϫʔυɿ"MJHONFOU • --.ʹ࿦จ͕ଟ͍ • ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ ʢ5FYUUP*NBHFͳͲʣ • ڧԽֶशք۾Ͱ΋ਓؾ • 3-)'΍ %10ͷྲྀߦ͕ ཧ༝ͩͱࢥΘΕΔ --. ը૾ɾಈըɾϚϧνϞʔμϧ ڧԽֶश

Slide 10

Slide 10 text

--. ը૾ɾಈըɾϚϧνϞʔμϧ ڧԽֶश "*4BGFUZݚڀͷ෼෍ 10 --.ɾը૾ • --.Ͱ੝Μʹݚڀ ˠ ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ • ڧԽֶशͰ૿Ճ͕ݦஶʢ3-)'΍ %10ͷӨڹʣ

Slide 11

Slide 11 text

11 ݴޠϞσϧͷ ΞϥΠϝϯτ "*4BGFUZ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*ΤʔδΣϯτతͳ࿩ʣ "(*΍ "4*Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌

Slide 12

Slide 12 text

12 طଘͷΞϥΠϝϯτख๏ͷܽ఺ͱ͸ʁ ධՁࢦඪ͕ݸͰ͋Δ͜ͱ ճ౴ͷྑ͠ѱ͕͠εΧϥʔ஋ͰධՁ͞ΕΔ͜ͱ Ϧϯΰʢྛޝ<>ɺֶ໊ .BMVTEPNFTUJDB .BMVTQVNJMBʣͱ͸ όϥՊϦϯΰଐͷམ༿ߴ໦ ·ͨ͸ͦͷՌ࣮Ͱ͢ɻ "OTXFS Ռ෺Ͱ͢ɻ "OTXFS ≻ Ϧϯΰʹ͍ͭͯڭ͑ͯԼ͍͞ 1SPNQU

Slide 13

Slide 13 text

13 طଘͷΞϥΠϝϯτख๏ͷܽ఺ͱ͸ʁ ධՁࢦඪ͕ݸͰ͋Δ͜ͱ ճ౴ͷྑ͠ѱ͕͠εΧϥʔ஋ͰධՁ͞ΕΔ͜ͱ Ϧϯΰʹ͍ͭͯڭ͑ͯԼ͍͞ 1SPNQU Ϧϯΰʢྛޝ<>ɺֶ໊ .BMVTEPNFTUJDB .BMVTQVNJMBʣͱ͸ όϥՊϦϯΰଐͷམ༿ߴ໦ ·ͨ͸ͦͷՌ࣮Ͱ͢ɻ ΞϥϯɾνϡʔϦϯά͕ɺ ੨ࢎԽ߹෺ΛϦϯΰʹృΓ ࣗࡴͨ͠ͱ͞Ε·͢ɻ "OTXFS Ռ෺Ͱ͢ɻ "OTXFS ≻ ≻ ༗༻ੑ ҆શੑ

Slide 14

Slide 14 text

14 طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ 4"$10 ༗༻ੑʹؔ͢Δ σʔλ ҆શੑʹؔ͢Δ σʔλ NBYJNVN MJLFMJIPPE FH %10 ,50 SFGFSFODF -.1PMJDZ NBYJNVN MJLFMJIPPE FH %10 ,50 GJOBM -.1PMJDZ SFXBSEBMJHOFE -.1PMJDZ Wachi, et al. “Stepwise Alignment for Constrained Language Model Policy Optimization.” In NeurIPS (2024). Huang et al. "One-Shot Safety Alignment for Large Language Models via Optimal Dualization." In NeurIPS (2024). Yang et al. "Metaaligner: Towards generalizable multi-objective alignment of language models." In NeurIPS (2024). Ruizhe+ "Decoding-time language model alignment with multiple objectives." In NeurIPS (2024). • ҆શ੍໿෇͖ͷ໰୊Λղ͘ˠ 8BDIJ )VBOH • ଟ໨త࠷దԽ໰୊Λղ͘ ˠ ,BJMBJ 3VJ[IF ը૾͸ :BOH ΑΓഈआ ը૾͸ 8BDIJ ΑΓഈआʢզʑͷ࿦จʣ

Slide 15

Slide 15 text

15 1MVSBMJTUJD"MJHONFOU 1MVSBMJTUJDʢଟݩతʣͳΞϥΠϝϯτʹಛԽͨ͠ϫʔΫγϣοϓ΋։࠵ • IUUQTQMVSBMJTUJDBMJHONFOUHJUIVCJP Our workshop will discuss how to integrate diverse perspectives, values, and expertise into pluralistic AI alignment.

Slide 16

Slide 16 text

13*4."MJHONFOU%BUBTFU 16 • %BUBTFUTBOE#FODINBSLT5SBDLͷड৆࿦จ • IUUQTBSYJWPSHBCT

Slide 17

Slide 17 text

ΞϥΠϝϯτͱ͸ʁ 17 LLM alignment refers to the process of ensuring that LLMs generate outputs that are consistent with human values, goals, and ethical standards. --.ͷΞϥΠϝϯτͱ͸ɺ--.͕ ਓؒͷՁ஋؍ɺ໨ඪɺ͓Αͼྙཧج४ʹҰகͨ͠ग़ྗΛ ੜ੒͢Δ͜ͱΛอূ͢ΔϓϩηεΛࢦ͠·͢ɻ IUUQTXXXUVSJOHDPNSFTPVSDFTMMNBMJHONFOUBOETBGFUZHVJEF

Slide 18

Slide 18 text

ʮਓؒʯͬͯͩΕʁ 18 LLM alignment refers to the process of ensuring that LLMs generate outputs that are consistent with human values, goals, and ethical standards. --.ͷΞϥΠϝϯτͱ͸ɺ--.͕ ਓؒͷՁ஋؍ɺ໨ඪɺ͓Αͼྙཧج४ʹҰகͨ͠ग़ྗΛ ੜ੒͢Δ͜ͱΛอূ͢ΔϓϩηεΛࢦ͠·͢ɻ

Slide 19

Slide 19 text

ਖ਼͍͠ճ౴͸ʁ 19 '*'"ϫʔϧυΧοϓͷ ༏উࠃ͸Ͳͩ͜ͱࢥ͍·͔͢ʁ

Slide 20

Slide 20 text

ΞϥΠϝϯτʹ͓͚Δʮਓؒʯͱ͸ 20 σʔλ ใुؔ਺ ΞϥΠϝϯτ ੈքதͷͩΕ Ͱ΋ͳ͍ʮͩΕ͔ʯ

Slide 21

Slide 21 text

࣮ࡍʹ͸ʮਓؒʯͷ෼෍͸ภΔ 21 • ͋Δࠃɾاۀͷ--.͸ɺͦͷࠃɾاۀͷʮਓؒʯͷՁ஋؍Λ൓ө͢Δ • ถࠃͷ --.ʢྫɿ(15 (FNJOJʣͱதࠃͷ --.ʢྫɿ%FFQ4FFLʣͰ͸ ग़ྗ͕େ͖͘ҟͳΔ • σʔλʹؔ͢Δৄࡉ͸௨ৗ։ࣔ͞Εͳ͍ • ͲͷΑ͏ʹσʔλΛऩूͨ͠ͷ͔ʁ • ͩΕ͕ʁ͍ͭʁͲ͜Ͱʁ • ଟ͔Εগͳ͔Εʮภ͍ͬͯΔʯ͜ͱ͸ؒҧ͍ͳ͍

Slide 22

Slide 22 text

13*4.σʔληοτ 22 ,JSL ΑΓը૾Λഈआ

Slide 23

Slide 23 text

ݸਓతʹڵຯਂ͔ͬͨݚڀ݁Ռ 23 ,JSL ΑΓը૾Λഈआ ޷Έ͸ਓͦΕͧΕ • ঁੑ΍ϊϯόΠφϦʔͷਓ͸ɺ உੑΑΓ΋ʮੑ΍-(#52ʯʹ ͍ͭͯ --.ͱର࿩͢Δ • ߴྸऀ͸ɺएऀΑΓ΋੓࣏΍ཱྀߦʹ ͍ͭͯٞ࿦͢Δ܏޲͕͋Δ • നਓ͸ɺࠇਓΑΓ΋ؾީมಈʹ͍ͭͯ ٞ࿦͢Δ܏޲ʹ͋Δ ಛఆͷάϧʔϓͷՁ஋؍͚ͩΛ൓ө͢Δ ͱɺͦͷଞϢʔβʔͷຬ଍౓͕௿Լ ˠ 1MVSBMJTUJDͳΞϥΠϝϯτ͕ॏཁ

Slide 24

Slide 24 text

24 ݴޠϞσϧͷ ΞϥΠϝϯτ "*4BGFUZ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*ΤʔδΣϯτతͳ࿩ʣ "(*΍ "4*Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌

Slide 25

Slide 25 text

25 5FYUUP*NBHFͷ҆શੑ Park et al. "Direct unlearning optimization for robust and safe text-to-image models." In NeurIPS (2024). Pan et al. "Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning." In NeurIPS (2024) • 5FYUUP*NBHFϞσϧ͕ɺ༗֐ͳը૾Λੜ੒͠ͳ͍Α͏ΞϥΠϝϯτ • ༗֐ͳը૾Λੜ੒͢ΔೳྗΛ๨٫ͤ͞Δ ˠ 1BSL 1BO ը૾͸Ӿཡ஫ҙ ը૾͸ 1BSL ΑΓഈआ

Slide 26

Slide 26 text

26 5FYUUP7JEFPͷ҆શੑ Dai et al. "SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset." In NeurIPS (2024). Miao et al. "T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models.” In NeurIPS (2024). • ಈըੜ੒Ϟσϧͷ҆શੑΛධՁ͢ΔϕϯνϚʔΫΛఏҊ • ϙϧϊɾ๫ྗɾࠩผͳͲ ͷΧςΰϦ • ಉ͡໨తͷ࿦จ͕ಉҰֶձʹ࠾୒ ˠ ڝ૪ͷܹ͠͞Λ෺ޠΔ ը૾͸ %BJ ΑΓഈआ ը૾͸ .JBP ΑΓഈआ ը૾͸Ӿཡ஫ҙ

Slide 27

Slide 27 text

27 ݴޠϞσϧͷ ΞϥΠϝϯτ "*4BGFUZ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*ΤʔδΣϯτతͳ࿩ʣ "(*΍ "4*Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌

Slide 28

Slide 28 text

28 "*ΤʔδΣϯτ Wei et al. "On the Effects of Data Scale on Computer Control Agents." In NeurIPS (2024). • "OESPJE$POUSPM ͱ͍͏σʔληοτΛఏڙʢ(PPHMFൃʣ • ༷ʑͳλεΫΛΧόʔʢͷ "OESPJEΞϓϦɾ ͷλεΫʣ 8FJ ΑΓ ը૾Λഈआ

Slide 29

Slide 29 text

29 "*ΤʔδΣϯτͷ҆શੑ Wei et al. "On the Effects of Data Scale on Computer Control Agents." In NeurIPS (2024). • "*"HFOUͷ҆શੑʹಛԽͨ͠ϫʔΫγϣοϓ • IUUQTXXXNMTBGFUZPSHFWFOUTOFVSJQT • "*͕࣮ߦೳྗΛ΋ͭͷͰɺ࣭ͷҟͳΔϦεΫ • དྷ೥Ҏ߱ɺຊձٞͰ΋࿦จ਺͕૿Ճ͢Δ͜ͱ͕༧૝͞ΕΔ 8FJ ΑΓ ը૾Λഈआ

Slide 30

Slide 30 text

30 ݴޠϞσϧͷ ΞϥΠϝϯτ "*4BGFUZ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*ΤʔδΣϯτతͳ࿩ʣ "(*΍ "4*Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌

Slide 31

Slide 31 text

"MJHOFS 31 • Φʔϥϧൃද࿦จ • IUUQTBSYJWPSHBCT • "4*ͱͷؔ܎ੑ΋ٞ࿦͓ͯ͠Γɺࢹ࠲ͷߴ͍ݚڀ

Slide 32

Slide 32 text

"MJHOFS 32 طଘͷΞϥΠϝϯτख๏ʢFH 3-)' %10ʣͷܽ఺ͱ͸ʁ Φʔϓϯͳ --.ʹ͔͠࢖͑ͳ͍ • (15΍ $MBVEFʹ͸ద༻ෆՄ • χϡʔϥϧωοτϫʔΫͷॏΈΛ࣮ࡍʹߋ৽͢Δඞཁ͕͋Δ ܭࢉෛՙ͕ߴ͍ • (16IPVSͨ͘͞Μඞཁ 3-)'΍ %10ͷৄࡉ͸ɺࡢ೥ͷ +4"*/FVS*14ࢀՃใࠂձͷಈըΛޚཡ͍ͩ͘͞ • :PV5VCFɿIUUQTXXXZPVUVCFDPNXBUDI WZP)C10WZ&

Slide 33

Slide 33 text

"MJHOFS 33 ෆద੾ͳग़ྗ Λ ޷·͍͠ग़ྗ ʹڲਖ਼͢ΔͨΊͷ࢒ࠩΛখ͍͞ϞσϧͰֶश "MJHOFS#d#ͷϞσϧ ˠ ܭࢉ͕༰қ 6QTUSFBN--.ͷग़ྗͷΈඞཁ ˠ Ϋϩʔζ ͳ --.ʹ΋ద༻Մೳ ݩ࿦จΑΓ ը૾Λഈआ

Slide 34

Slide 34 text

"MJHOFS 34 ݩ࿦จΑΓ ը૾Λഈआ • #ͷ "MJHOFSͰ (15΍ $MBVEFͷ༗༻ੑ΍҆શੑΛ޲্

Slide 35

Slide 35 text

8FBLUP4USPOH$PSSFDUJPO 35 Burns et al. "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.” arXiv preprint arXiv:2312.09390 (2023). #VSOT Ͱొ৔ͨ֓͠೦ "MJHOFS͕ఏএ͍ͯ͠Δ 8FBLUP4USPOH$PSSFDUJPO ݩ࿦จΑΓ ը૾Λഈआ • 8FBL4VQFSWJTPS "MJHOFS ͕ɺ4USPOH4UVEFOU ྫɿ(15 Λ ੍ޚɾగਖ਼Ͱ͖Δɺͱ͍͏ϙδςΟϒͳ݁Ռ

Slide 36

Slide 36 text

·ͱΊ /FVS*14ͷײ૝ɾงғؾ "*4BGFUZɾΞϥΠϝϯτ ݚڀͷைྲྀ 13*4."MJHONFOU%BUBTFU࿦จ঺հʢ#FTU1BQFSʣ "MJHOFS࿦จ঺հʢ0SBMʣ ͜ͷࢿྉʹؔͯ͠ɺ࣭໰΍ؒҧ͍ͷࢦఠͳͲ͍͟͝·ͨ͠Β ϝʔϧʹͯ͝࿈བྷ͓ئ͍͠·͢ XBDIJBLJGVNJ HNBJMDPN 36