JSAI NeurIPS 2024 参加報告会（AI アライメント）

第96回人工知能セミナー(2025.2.7)「AIトレンド・トップカンファレンス報告会（NeurIPS2024）：世界最先端のAI研究開発動向が1日でわかる！」
- https://www.ai-gakkai.or.jp/event/ai-seminar/no96_jsai_seminar/

Transcript

1. ʲ/FVS*14
   ʳ "*
   ΞϥΠϝϯτྖҬͷ࠷৽ಈ޲ "LJGVNJ
   8BDIJ
   
   ࿨஍ ྎྑ ୈճਓ޻஌ೳηϛφʔ ೥݄೔ 1
2. "LJGVNJ
   8BDIJʢ࿨஍ ྎྑʣ ܦྺ • 
   r ɿ*#.
   ౦ژجૅݚڀॴ 3FTFBSDI
   4DJFOUJTU • 
   r ݱࡏɿ-*/&Ϡϑʔ $IJFG
   3FTFBSDI
   4DJFOUJTU ݚڀ෼໺ • ڧԽֶश ºʢ"*
   4BGFUZ
   ࣗવݴޠॲཧʣ ஶॻʢڞஶʣ • ʰڧԽֶश͔Β৴པͰ͖Δҙࢥܾఆ΁ʱ /FVS*14
   ͷ࠾୒࿦จ • ओஶɿ4UFQXJTF
   "MJHONFOU
   GPS
   $POTUSBJOFE
   -BOHVBHF
   .PEFM
   1PMJDZ
   0QUJNJ[BUJPOT • IUUQTBSYJWPSHBCT • ڞஶɿ'MJQQJOHCBTFE
   1PMJDZ
   GPS
   $IBODF$POTUSBJOFE
   .BSLPW
   %FDJTJPO
   1SPDFTTFT • IUUQTBSYJWPSHBCT 2
3. ΞδΣϯμ  /FVS*14
   
   ͷײ૝ɾงғؾ  "*
   4BGFUZɾΞϥΠϝϯτ ݚڀͷைྲྀ  13*4.
   "MJHONFOU
   %BUBTFU
   ࿦จ঺հʢ#FTU
   1BQFSʣ • ,JSL
   FU
   BM
   5IF
   13*4.
   "MJHONFOU
   %BUBTFU
   8IBU
   1BSUJDJQBUPSZ
   3FQSFTFOUBUJWF
   BOE
   *OEJWJEVBMJTFE
   )VNBO
   'FFECBDL
   3FWFBMT
   "CPVU
   UIF
   4VCKFDUJWF
   BOE
   .VMUJDVMUVSBM
   "MJHONFOU
   PG
   -BSHF
   -BOHVBHF
   .PEFMTz • IUUQTBSYJWPSHBCT  "MJHOFS
   ࿦จ঺հʢ0SBMʣ • +J
   FU
   BM
   "MJHOFS
   &GGJDJFOU
   "MJHONFOU
   CZ
   -FBSOJOH
   UP
   $PSSFDUz • IUUQTBSYJWPSHBCT 3
4. /FVS*14
   
   ͷײ૝ɾงғؾ 4 ݄ c ݄ c $IBU(15 1SP ೥ ೥ ݄ c (15 ݄ c %"--& ݄ c 4PSB ݄ c (15P ݄ c 0QFO"* P • /FVS*14
   
   ͱ /FVS*14
   
   Ͱ͔ͳΓงғؾ͕ҧ͏ʂ • /FVS*14
   ͷ։࠵λΠϛϯάΛ 0QFO"*
   ͷϦϦʔεͱॏͶΔͱɾɾɾ • ͳΜͳΒɺ࿦จʒ੾࣌ʢ݄ʣͱֶձ։࠵࣌ʢ݄ʣͰ΋ҧ͏ /FVS*14 /FVS*14 ˢ /FVS*14 ͷ࿦จʒ੾
5. "*
   4BGFUZ
   ݚڀͷมԽ 5 ײ૝̍ɿʮਅ݋͞ʯ͕૿ͨ͠ • ݚڀͱ࣮༻ͷڑ཭͕͍ۙͮͨ • "*
   ͷೳྗ͕ɺ࣮ੈքͰٻΊΒΕΔϨϕϧʹୡ͠͸͡Ί͍ͯΔ • --.
   ͳͲɺߴੑೳͳ "*
   Λѻ͏ݚڀͷׂ߹͕૿͑ͨ • "*
   ͕ຊ౰ͷҙຯͰʮةݥͳ΋ͷʯʹͳ͔ͬͨΒ Goodfellow et al. "Generative adversarial nets." In NeurIPS (2014). Tian et al. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." In NeurIPS (2024). ೥౰࣌ͷੜ੒ը૾ ʢ(PPEGFMMPX
   ΑΓഈआʣ ೥࣌఺Ͱͷੜ੒ը૾ ʢ5JBO
   ΑΓഈआʣ ٕज़ͷਐาʹΑΓ ϦεΫ΋૿େ
6. "*
   4BGFUZ
   ݚڀͷมԽ 6 :PTIVB
   #FOHJP 4BGF
   (FOFSBUJWF
   "*
   XPSLTIPQ ײ૝̎ɿ"(*ɾ"4*
   ͷ஀ੜΛલఏʹݚڀ͍ͯ͠Δਓ͕૿͑ͨ • "*
   ͷʮࢦ਺ؔ਺తͳʯਐԽ • ࡢ೥·Ͱɿকདྷతͳ՝୊ͱͯٞ͠࿦ɻ"*
   ͸੍ޚͰ͖Δͱ͍͏લఏ • ࠓ೥ɿ٤ۓͷ՝୊ͱͯٞ͠࿦ɻʮͦ΋ͦ΋੍ޚͰ͖ͳ͍ର৅͕஀ੜͨ͠Βʁʯ *MZB
   4VUTLFWFS 5FTUPGUJNF
   "XBSE
7. "*
   4BGFUZ
   ݚڀͷ෼෍ 7 • /FVS*14
   ެࣜͷ ࿦จՄࢹԽπʔϧ • IUUQTOFVSJQTWJ[IVCBJ • Ωʔϫʔυɿ4BGF • --.
   ʹ࿦จ͕ଟ͍ • ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ ʢ5FYUUP7JEFP
   ͳͲʣ • ڧԽֶशͰ͸ࠜڧ͍ਓؾʢ݁ߏલ͔Βʣ --. ϚϧνϞʔμϧ ڧԽֶश
8. --. ϚϧνϞʔμϧ ڧԽֶश "*
   4BGFUZ
   ݚڀͷ෼෍ 8   ڧԽֶश --. --.Ͱ૿Ճ͕ݦஶ ˠ ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ
9. "*
   4BGFUZ
   ݚڀͷ෼෍ 9 • /FVS*14
   ެࣜͷ ࿦จՄࢹԽπʔϧ • IUUQTOFVSJQTWJ[IVCBJ • Ωʔϫʔυɿ"MJHONFOU • --.
   ʹ࿦จ͕ଟ͍ • ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ ʢ5FYUUP*NBHF
   ͳͲʣ • ڧԽֶशք۾Ͱ΋ਓؾ • 3-)'
   ΍ %10
   ͷྲྀߦ͕ ཧ༝ͩͱࢥΘΕΔ --. ը૾ɾಈըɾϚϧνϞʔμϧ ڧԽֶश
10. --. ը૾ɾಈըɾϚϧνϞʔμϧ ڧԽֶश "*
    4BGFUZ
    ݚڀͷ෼෍ 10   --.ɾը૾ • --.Ͱ੝Μʹݚڀ ˠ ผϞʔμϧɾϚϧνϞʔμϧʹ೿ੜ • ڧԽֶशͰ૿Ճ͕ݦஶʢ3-)'
    ΍ %10
    ͷӨڹʣ
11. 11 ݴޠϞσϧͷ ΞϥΠϝϯτ "*
    4BGFUZ
    ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*
    ΤʔδΣϯτతͳ࿩ʣ "(*
    ΍ "4*
    Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌
12. 12 طଘͷΞϥΠϝϯτख๏ͷܽ఺ͱ͸ʁ ධՁࢦඪ͕ݸͰ͋Δ͜ͱ 
    ճ౴ͷྑ͠ѱ͕͠εΧϥʔ஋ͰධՁ͞ΕΔ͜ͱ Ϧϯΰʢྛޝ<>ɺֶ໊ .BMVT
    EPNFTUJDB .BMVT
    QVNJMBʣͱ͸ όϥՊϦϯΰଐͷམ༿ߴ໦ ·ͨ͸ͦͷՌ࣮Ͱ͢ɻ "OTXFS Ռ෺Ͱ͢ɻ "OTXFS
    ≻ Ϧϯΰʹ͍ͭͯڭ͑ͯԼ͍͞ 1SPNQU
    
13. 13 طଘͷΞϥΠϝϯτख๏ͷܽ఺ͱ͸ʁ ධՁࢦඪ͕ݸͰ͋Δ͜ͱ 
    ճ౴ͷྑ͠ѱ͕͠εΧϥʔ஋ͰධՁ͞ΕΔ͜ͱ Ϧϯΰʹ͍ͭͯڭ͑ͯԼ͍͞ 1SPNQU
    Ϧϯΰʢྛޝ<>ɺֶ໊ .BMVT
    EPNFTUJDB .BMVT
    QVNJMBʣͱ͸ όϥՊϦϯΰଐͷམ༿ߴ໦ ·ͨ͸ͦͷՌ࣮Ͱ͢ɻ ΞϥϯɾνϡʔϦϯά͕ɺ ੨ࢎԽ߹෺ΛϦϯΰʹృΓ ࣗࡴͨ͠ͱ͞Ε·͢ɻ "OTXFS Ռ෺Ͱ͢ɻ "OTXFS
    ≻ ≻ ༗༻ੑ ҆શੑ
14. 14 طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ 4"$10 ༗༻ੑʹؔ͢Δ σʔλ ҆શੑʹؔ͢Δ σʔλ NBYJNVN MJLFMJIPPE FH
    %10
    ,50 SFGFSFODF -.
    1PMJDZ NBYJNVN MJLFMJIPPE FH
    %10
    ,50 GJOBM -.
    1PMJDZ SFXBSEBMJHOFE -.
    1PMJDZ Wachi, et al. “Stepwise Alignment for Constrained Language Model Policy Optimization.” In NeurIPS (2024). Huang et al. "One-Shot Safety Alignment for Large Language Models via Optimal Dualization." In NeurIPS (2024). Yang et al. "Metaaligner: Towards generalizable multi-objective alignment of language models." In NeurIPS (2024). Ruizhe+ "Decoding-time language model alignment with multiple objectives." In NeurIPS (2024). • ҆શ੍໿෇͖ͷ໰୊Λղ͘ˠ 8BDIJ
    
    )VBOH
     • ଟ໨త࠷దԽ໰୊Λղ͘ ˠ ,BJMBJ
    
    3VJ[IF
     ը૾͸ :BOH
     ΑΓഈआ ը૾͸ 8BDIJ
    
    ΑΓഈआʢզʑͷ࿦จʣ
15. 15 1MVSBMJTUJD
    "MJHONFOU 1MVSBMJTUJDʢଟݩతʣͳΞϥΠϝϯτʹಛԽͨ͠ϫʔΫγϣοϓ΋։࠵ • IUUQTQMVSBMJTUJDBMJHONFOUHJUIVCJP Our workshop will discuss how to integrate diverse perspectives, values, and expertise into pluralistic AI alignment.
16. 13*4.
    "MJHONFOU
    %BUBTFU 16 • %BUBTFUT
    BOE
    #FODINBSLT
    5SBDL
    ͷड৆࿦จ • IUUQTBSYJWPSHBCT
17. ΞϥΠϝϯτͱ͸ʁ 17 LLM alignment refers to the process of ensuring that LLMs generate outputs that are consistent with human values, goals, and ethical standards. --.ͷΞϥΠϝϯτͱ͸ɺ--.͕ ਓؒͷՁ஋؍ɺ໨ඪɺ͓Αͼྙཧج४ʹҰகͨ͠ग़ྗΛ ੜ੒͢Δ͜ͱΛอূ͢ΔϓϩηεΛࢦ͠·͢ɻ IUUQTXXXUVSJOHDPNSFTPVSDFTMMNBMJHONFOUBOETBGFUZHVJEF
18. ʮਓؒʯͬͯͩΕʁ 18 LLM alignment refers to the process of ensuring that LLMs generate outputs that are consistent with human values, goals, and ethical standards. --.ͷΞϥΠϝϯτͱ͸ɺ--.͕ ਓؒͷՁ஋؍ɺ໨ඪɺ͓Αͼྙཧج४ʹҰகͨ͠ग़ྗΛ ੜ੒͢Δ͜ͱΛอূ͢ΔϓϩηεΛࢦ͠·͢ɻ
19. ਖ਼͍͠ճ౴͸ʁ 19 
    '*'"ϫʔϧυΧοϓͷ ༏উࠃ͸Ͳͩ͜ͱࢥ͍·͔͢ʁ
20. ΞϥΠϝϯτʹ͓͚Δʮਓؒʯͱ͸ 20 σʔλ ใुؔ਺ ΞϥΠϝϯτ ੈքதͷͩΕ Ͱ΋ͳ͍ʮͩΕ͔ʯ
21. ࣮ࡍʹ͸ʮਓؒʯͷ෼෍͸ภΔ 21 • ͋Δࠃɾاۀͷ--.͸ɺͦͷࠃɾاۀͷʮਓؒʯͷՁ஋؍Λ൓ө͢Δ • ถࠃͷ --.ʢྫɿ(15
    (FNJOJʣͱதࠃͷ --.ʢྫɿ%FFQ4FFLʣͰ͸ ग़ྗ͕େ͖͘ҟͳΔ • σʔλʹؔ͢Δৄࡉ͸௨ৗ։ࣔ͞Εͳ͍ • ͲͷΑ͏ʹσʔλΛऩूͨ͠ͷ͔ʁ • ͩΕ͕ʁ͍ͭʁͲ͜Ͱʁ • ଟ͔Εগͳ͔Εʮภ͍ͬͯΔʯ͜ͱ͸ؒҧ͍ͳ͍
22. 13*4.
    σʔληοτ 22 ,JSL
     ΑΓը૾Λഈआ
23. ݸਓతʹڵຯਂ͔ͬͨݚڀ݁Ռ 23 ,JSL
     ΑΓը૾Λഈआ ޷Έ͸ਓͦΕͧΕ • ঁੑ΍ϊϯόΠφϦʔͷਓ͸ɺ உੑΑΓ΋ʮੑ΍-(#52 ʯʹ ͍ͭͯ --.
    ͱର࿩͢Δ • ߴྸऀ͸ɺएऀΑΓ΋੓࣏΍ཱྀߦʹ ͍ͭͯٞ࿦͢Δ܏޲͕͋Δ • നਓ͸ɺࠇਓΑΓ΋ؾީมಈʹ͍ͭͯ ٞ࿦͢Δ܏޲ʹ͋Δ ಛఆͷάϧʔϓͷՁ஋؍͚ͩΛ൓ө͢Δ ͱɺͦͷଞϢʔβʔͷຬ଍౓͕௿Լ ˠ 1MVSBMJTUJD
    ͳΞϥΠϝϯτ͕ॏཁ
24. 24 ݴޠϞσϧͷ ΞϥΠϝϯτ "*
    4BGFUZ
    ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*
    ΤʔδΣϯτతͳ࿩ʣ "(*
    ΍ "4*
    Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌
25. 25 5FYUUP*NBHF
    ͷ҆શੑ Park et al. "Direct unlearning optimization for robust and safe text-to-image models." In NeurIPS (2024). Pan et al. "Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning." In NeurIPS (2024) • 5FYUUP*NBHF
    Ϟσϧ͕ɺ༗֐ͳը૾Λੜ੒͠ͳ͍Α͏ΞϥΠϝϯτ • ༗֐ͳը૾Λੜ੒͢ΔೳྗΛ๨٫ͤ͞Δ ˠ 1BSL
    
    1BO
     ը૾͸Ӿཡ஫ҙ ը૾͸ 1BSL
    
    ΑΓഈआ
26. 26 5FYUUP7JEFP
    ͷ҆શੑ Dai et al. "SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset." In NeurIPS (2024). Miao et al. "T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models.” In NeurIPS (2024). • ಈըੜ੒Ϟσϧͷ҆શੑΛධՁ͢ΔϕϯνϚʔΫΛఏҊ • ϙϧϊɾ๫ྗɾࠩผͳͲ 
    ͷΧςΰϦ • ಉ͡໨తͷ࿦จ͕ಉҰֶձʹ࠾୒ ˠ ڝ૪ͷܹ͠͞Λ෺ޠΔ ը૾͸ %BJ
    
    ΑΓഈआ ը૾͸ .JBP
     ΑΓഈआ ը૾͸Ӿཡ஫ҙ
27. 27 ݴޠϞσϧͷ ΞϥΠϝϯτ "*
    4BGFUZ
    ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*
    ΤʔδΣϯτతͳ࿩ʣ "(*
    ΍ "4*
    Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌
28. 28 "*
    ΤʔδΣϯτ Wei et al. "On the Effects of Data Scale on Computer Control Agents." In NeurIPS (2024). • "OESPJE$POUSPM ͱ͍͏σʔληοτΛఏڙʢ(PPHMF
    ൃʣ • ༷ʑͳλεΫΛΧόʔʢ
    ͷ "OESPJE
    ΞϓϦɾ 
    ͷλεΫʣ 8FJ
     ΑΓ ը૾Λഈआ
29. 29 "*
    ΤʔδΣϯτͷ҆શੑ Wei et al. "On the Effects of Data Scale on Computer Control Agents." In NeurIPS (2024). • "*
    "HFOU
    ͷ҆શੑʹಛԽͨ͠ϫʔΫγϣοϓ • IUUQTXXXNMTBGFUZPSHFWFOUTOFVSJQT • "*
    ͕࣮ߦೳྗΛ΋ͭͷͰɺ࣭ͷҟͳΔϦεΫ • དྷ೥Ҏ߱ɺຊձٞͰ΋࿦จ਺͕૿Ճ͢Δ͜ͱ͕༧૝͞ΕΔ 8FJ
     ΑΓ ը૾Λഈआ
30. 30 ݴޠϞσϧͷ ΞϥΠϝϯτ "*
    4BGFUZ
    ੝Γ্͕͍ͬͯΔʂ ผͷϞʔμϧɾϚϧνϞʔμϧ΁ ࣮ߦೳྗΛ࣋ͨͤͨͱ͖ͷ҆શੑ ʢ"*
    ΤʔδΣϯτతͳ࿩ʣ "(*
    ΍ "4*
    Λݟਾ͑ͨ҆શੑ طଘͷΞϥΠϝϯτख๏ͷ֦ுɾվྑ ཧ࿦ղੳɾݪཧղ໌
31. "MJHOFS 31 • Φʔϥϧൃද࿦จ • IUUQTBSYJWPSHBCT • "4*
    ͱͷؔ܎ੑ΋ٞ࿦͓ͯ͠Γɺࢹ࠲ͷߴ͍ݚڀ
32. "MJHOFS 32 طଘͷΞϥΠϝϯτख๏ʢFH
    3-)'
    %10ʣͷܽ఺ͱ͸ʁ  Φʔϓϯͳ --.
    ʹ͔͠࢖͑ͳ͍ • (15
    ΍ $MBVEF
    
    ʹ͸ద༻ෆՄ • χϡʔϥϧωοτϫʔΫͷॏΈΛ࣮ࡍʹߋ৽͢Δඞཁ͕͋Δ  ܭࢉෛՙ͕ߴ͍ • (16
    IPVS
    ͨ͘͞Μඞཁ 3-)'
    ΍ %10
    ͷৄࡉ͸ɺࡢ೥ͷ +4"*
    /FVS*14
    ࢀՃใࠂձͷಈըΛޚཡ͍ͩ͘͞ • :PV5VCFɿIUUQTXXXZPVUVCFDPNXBUDI WZP)C10WZ&
33. "MJHOFS 33 ෆద੾ͳग़ྗ Λ ޷·͍͠ग़ྗ ʹڲਖ਼͢ΔͨΊͷ࢒ࠩΛখ͍͞ϞσϧͰֶश "MJHOFS
    
    #d#
    ͷϞσϧ ˠ ܭࢉ͕༰қ 6QTUSFBN
    --.
    ͷग़ྗͷΈඞཁ ˠ Ϋϩʔζ ͳ --.
    ʹ΋ద༻Մೳ ݩ࿦จΑΓ ը૾Λഈआ
34. "MJHOFS 34 ݩ࿦จΑΓ ը૾Λഈआ • #
    ͷ "MJHOFS
    Ͱ (15
    ΍ $MBVEF
    
    ͷ༗༻ੑ΍҆શੑΛ޲্
35. 8FBLUP4USPOH
    $PSSFDUJPO 35 Burns et al. "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.” arXiv preprint arXiv:2312.09390 (2023). #VSOT
     Ͱొ৔ͨ֓͠೦ "MJHOFS
    ͕ఏএ͍ͯ͠Δ 8FBLUP4USPOH
    $PSSFDUJPO ݩ࿦จΑΓ ը૾Λഈआ • 8FBL
    4VQFSWJTPS
    "MJHOFS
    ͕ɺ4USPOH
    4UVEFOU
    ྫɿ(15
    Λ ੍ޚɾగਖ਼Ͱ͖Δɺͱ͍͏ϙδςΟϒͳ݁Ռ
36. ·ͱΊ  /FVS*14
    
    ͷײ૝ɾงғؾ  "*
    4BGFUZɾΞϥΠϝϯτ ݚڀͷைྲྀ  13*4.
    "MJHONFOU
    %BUBTFU
    ࿦จ঺հʢ#FTU
    1BQFSʣ  "MJHOFS
    ࿦จ঺հʢ0SBMʣ ͜ͷࢿྉʹؔͯ͠ɺ࣭໰΍ؒҧ͍ͷࢦఠͳͲ͍͟͝·ͨ͠Β ϝʔϧʹͯ͝࿈བྷ͓ئ͍͠·͢ XBDIJBLJGVNJ <BU>
    HNBJMDPN 36