Share
Embed
NLPコロキウム:Unlearning Traces the Influential Training Data of Language Models
Masaru Isonuma
July 03, 2024
Transcript
18 slides
-
Unlearning Traces the Influential Training Data of Language Models үপ
େ1,2 Ivan Titov1, 3 1ΤσΟϯόϥେֶ 2౦ژେֶ 3ΞϜεςϧμϜେֶ -
• ௨ৗͷֶश – ֶशσʔλ𝑧͕ੜ͞ΕΔ֬𝑝𝜽 (𝑧)Λ࠷େԽ͢ΔΑ͏ʹɺϞσϧ𝜽Λߋ৽ 𝜽 = arg max 𝜽
log 𝑝𝜽 (𝑧)ɿޯ߱Լ๏Λར༻ • ٯֶश – ֶशσʔλ𝑧͕ੜ͞ΕΔ֬𝑝𝜽 (𝑧)Λ࠷খԽ͢ΔΑ͏ʹɺϞσϧ𝜽Λߋ৽ 𝜽 = arg min 𝜽 log 𝑝𝜽 (𝑧)ɿޯ্ঢ๏Λར༻ Unlearning ʢٯֶश/ֶशʣͱ 2 𝒛: Masaru lives in the UK UK Japan US 𝒛: Masaru lives in the UK UK Japan US -
• ༗ͳֶशࡁσʔλΛϞσϧ͔Β٫ – ϓϥΠόγʔ৵ࢭ – ஶ࡞ݖ৵ࢭ – όΠΞεআڈ – ॏෳσʔλআڈ
ٯֶशݚڀͷྲྀΕͱҐஔ͚ 3 2015ɿCVΛத৺ʹٯֶश͕ΘΕΔ Cao et al., Towards making systems forget with machine unlearning. IEEE symposium on security and privacy, 2015. 2023ɿLLMʹٯֶशΛར༻ Jang et al., Knowledge Unlearning for Mitigating Privacy Risks in Language Models. ACL, 2023. 2024ɿຊݚڀ • ֶशσʔλͷӨڹਪఆ ٯֶशݚڀͷྲྀΕ త -
• ͋ΔϞσϧͷग़ྗʹର͠ɺֶ֤शσʔλͲͷఔӨڹΛ༩͔͑ͨʁ • ֶशσʔλΛൈֶ͍ͯशΛ܁Γฦ͢͜ͱͰɺֶ֤शσʔλͷӨڹΛଌΕΔ͕ɺܭࢉίετ͕େ ֶशσʔλͷӨڹਪఆͱ 4 χϡʔε ॻ੶ SNS ֶशσʔλ
Ϟσϧͷग़ྗ + + → χϡʔε ॻ੶ SNS + + → χϡʔε ॻ੶ SNS + + → ແ ༗ ༗ -
• ֶशσʔλΛൈ͘ΘΓʹɺֶशࡁϞσϧ͔Βٯֶश • ٯֶशલޙͷϞσϧΛൺֱ͢Δ͜ͱͰɺֶशσʔλͷӨڹΛਪఆ Ξϓϩʔν 5 χϡʔε ॻ੶ SNS ֶशࡁϞσϧ
Ϟσϧͷग़ྗ + + → χϡʔε ॻ੶ SNS + + → χϡʔε ॻ੶ SNS + + → ແ ༗ ༗ χϡʔε − ॻ੶ − SNS − ٯֶश -
• ֶशσʔληοτ𝑍! ͕ධՁσʔληοτ𝑍"ʹ༩͑ΔӨڹ (ਅͷ) ΛɺଛࣦؔͷࠩͰఆٛ (leave-one-out) 𝐼#$%#& 𝑍! , 𝑍"
= 𝐿 𝑍", 𝜽'(𝒊 − 𝐿(𝑍", 𝜽) ) ઃఆͱΞϓϩʔν 6 ධՁσʔλ𝑍! ֶशσʔλ𝑍" ֶश ධՁ ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍$ + + + + → → ධՁσʔλ𝑍! ֶशσʔλ𝑍" + + → ֶशσʔλ𝑍$ ֶशσʔλ𝑍# 𝑍" Λൈֶ͍ͯशͨ͠Ϟσϧ શσʔλΛֶशͨ͠Ϟσϧ -
• ֶशσʔληοτ𝑍! ͕ධՁσʔληοτ𝑍"ʹ༩͑ΔӨڹ (ਅͷ) ΛɺଛࣦؔͷࠩͰఆٛ (leave-one-out) 𝐼#$%#& 𝑍! , 𝑍"
= 𝐿 𝑍", 𝜽'(𝒊 − 𝐿(𝑍", 𝜽) ) • 𝜽'(" ͷܭࢉେม ⇒ 𝑍! Λൈ͘ΘΓʹɺશσʔλΛֶशͨ͠Ϟσϧ͔Βɺ𝑍! Λٯֶशͯ͠𝐼#$%#& Λਪఆ 𝜽'(" = argmin 𝜽 ∑(#+(" 𝐿 𝑍, , 𝜽 = argmin 𝜽 ∑(# 𝐿 𝑍, , 𝜽 − 𝐿 𝑍! , 𝜽 ઃఆͱΞϓϩʔν 7 ධՁσʔλ𝑍! ֶशσʔλ𝑍" ֶश ධՁ ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍$ + + + + → → ධՁσʔλ𝑍! ֶशσʔλ𝑍" + + → ֶशσʔλ𝑍$ ֶशσʔλ𝑍# 𝑍" Λൈֶ͍ͯशͨ͠Ϟσϧ શσʔλΛֶशͨ͠Ϟσϧ -
• UnTracɿֶशσʔληοτ𝑍! Λٯֶशͯ͠ධՁσʔληοτ𝑍"ͰධՁ UnTrac 𝑍! , 𝑍" = 𝐿 𝑍",
𝜽- − 𝐿(𝑍", 𝜽) ) • ֶशσʔλ͕ଟ͍߹ɺܭࢉίετߴ ఏҊख๏1ɿUnTrac 8 ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ શσʔλΛֶशͨ͠Ϟσϧ ٯֶश ධՁ ֶशσʔλ𝑍# ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ ֶशσʔλ𝑍" + + + + − − → → ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ ֶशσʔλ𝑍$ + + − → ֶशσʔλ𝑍" Λٯֶशͨ͠Ϟσϧ ʢޯ্ঢ๏ʣ શσʔλΛֶशͨ͠Ϟσϧ -
• UnTrac-InvɿධՁσʔληοτ𝑍"Λٯֶशֶͯ͠शσʔληοτ𝑍! ͰධՁ UnTrac − Inv 𝑍! , 𝑍" =
𝐿 𝑍! , 𝜽- − 𝐿(𝑍! , 𝜽) ) • ͍͔ͭ͘ͷԾఆɾ݅ͷͱɺUnTrac-InvUnTracΛۙࣅ ఏҊख๏2ɿUnTrac-Inv 9 ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ શσʔλΛֶशͨ͠Ϟσϧ ٯֶश ධՁ ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ + + + + − − → → ֶशσʔλ𝑍# ֶशσʔλ𝑍" ධՁσʔλ𝑍! ֶशσʔλ𝑍# ֶशσʔλ𝑍" ֶशσʔλ𝑍$ + + − → ֶशσʔλ𝑍$ ධՁσʔλ𝑍#Λٯֶशͨ͠Ϟσϧ ʢޯ্ঢ๏ʣ શσʔλΛֶशͨ͠Ϟσϧ -
• 𝜽. = 𝜽.'/ + 𝜂∇𝜽 𝐿 𝑧. , 𝜽.'/
; 𝐿 𝑍", 𝜽. − 𝐿(𝑍", 𝜽.'/ ) ≈ ∇𝜽 𝐿 𝑍", 𝜽.'/ (𝜽. − 𝜽.'/ )ͱ1࣍ۙࣅ͢Δͱɺ UnTrac 𝑍, 𝑍" = ∑.0/ - 𝐿 𝑍", 𝜽. − 𝐿(𝑍", 𝜽.'/ ) ≈ ∑.0/ - 𝜂∇𝜽 𝐿 𝑧. , 𝜽.'/ 1∇𝜽 𝐿 𝑍", 𝜽.'/ • ಉ༷ʹɺ UnTrac − Inv 𝑍, 𝑍" = ∑.0/ -$ 𝐿 𝑍, 𝜽. − 𝐿(𝑍, 𝜽.'/ ) ≈ ∑.0/ -$ 𝜂∇𝜽 𝐿 𝑍, 𝜽.'/ 1∇𝜽 𝐿 𝑧. ", 𝜽.'/ • ٯֶश͕1εςοϓͰ (𝑇 = 𝑇" = 1)ɺ1όονʹશαϯϓϧؚ͕·ΕΔͱ͖ (𝑧. = 𝑍, 𝑧. " = 𝑍")ɺ྆ऀҰக → όοναΠζ͕େ͖͘ɺ͔ͭٯֶशͷεςοϓ͕গͳ͍߹ʹUnTrac-Inv༗ޮ UnTrac-Inv͕UnTracΛۙࣅ͢Δ݅ 10 -
• طଘख๏ͷଟ͘ɺUnTrac/UnTrac-InvͷಛघέʔεͱΈͳͤΔ • UnTracͰશͯͷֶशαϯϓϧΛ1εςοϓٯֶशͨ͠߹ΛҰ࣍ۙࣅ͢Δͱɺ UnTrac 𝑍, 𝑍" = 𝐿 𝑍",
𝜽/ − 𝐿 𝑍", 𝜽) ≈ ∇𝜽 𝐿 𝑍", 𝜽) 1 𝜽/ − 𝜽) طଘख๏ͱͷؔ 11 GradDot GradCos HIF ∇𝜽 𝐿 𝑍#, 𝜽$ %∇𝜽 𝐿 𝑍, 𝜽$ ∇𝜽 𝐿 𝑍#, 𝜽$ %∇𝜽 𝐿 𝑍, 𝜽$ ∇𝜽 𝐿 𝑍#, 𝜽$ ∇𝜽 𝐿 𝑍, 𝜽$ ∇𝜽𝐿 𝑍#, 𝜽$ %𝐻𝜽 &'∇𝜽𝐿 𝑍, 𝜽$ ਪఆ طଘख๏ 𝜽' − 𝜽$ = 𝜂∇𝜽 𝐿 𝑍, 𝜽$ (SGD) 𝜽' − 𝜽$ = 𝜂 ∇𝜽) *, 𝜽& ∇𝜽) *, 𝜽& (RMSProp, Adam) 𝜽' − 𝜽$ = 𝐻𝜽 &'∇𝜽𝐿 𝑍, 𝜽$ (χϡʔτϯ๏) Ұக͢Δ݅ Fisher kernel ∇𝜽 𝐿 𝑍#, 𝜽$ %𝐹𝜽 &'∇𝜽 𝐿 𝑍, 𝜽$ 𝜽' − 𝜽$ = 𝐹𝜽 &'∇𝜽 𝐿 𝑍, 𝜽$ (ࣗવޯ๏) -
• ࣄલֶशࡁT5 (3B)ͷϑΝΠϯνϡʔχϯάʹ༻͍ͨλεΫͷӨڹΛదʹਪఆͰ͖Δ͔ݕূ • ຊख๏ͷݒ೦ɿ ͋Δग़ྗܗࣜͷλεΫΛٯֶश͢ΔͱɺٯֶशޙͷϞσϧͲͷλεΫͰͦͷܗࣜͰग़ྗ͠ͳ͘ͳΔʁ → ධՁλεΫͱग़ྗܗ͕ࣜಉ͡λεΫͷӨڹ͕աେධՁ͞ΕΔڪΕ • ධՁλεΫͱྨࣅ/ඇྨࣅºಉ͡/ҟͳΔग़ྗܗࣜͷ4λεΫΛֶश
࣮ݧ1ɿT5ͷϑΝΠϯνϡʔχϯά 12 σʔληοτͷαϯϓϧྫɻ{} ෦αϯϓϧ͝ͱʹҟͳΔɻ -
• ྆ख๏ͱɺྨࣅλεΫʢ1, 2ʣͷӨڹΛඇྨࣅλεΫʢ3, 4ʣΑΓߴ͘ਪఆ • ࣮ࡍʹleave-one-outΛߦ͏ͱɺྨࣅλεΫʢ1, 2ʣͷӨڹඇྨࣅλεΫʢ3, 4ʣΑΓߴ͍ Þ ग़ྗܗࣜʹ͞΄Ͳࠨӈ͞ΕͣɺֶशλεΫͷӨڹΛదʹਪఆ
࣮ݧ1ɿ݁Ռ 13 UnTrac UnTrac-Inv 1: ྨࣅλεΫ/ಉ͡ग़ྗܗࣜ 2: ྨࣅλεΫ/ҟͳΔग़ྗܗࣜ 3: ඇྨࣅλεΫ/ಉ͡ग़ྗܗࣜ 4: ඇྨࣅλεΫ/ҟͳΔग़ྗܗࣜ ٯֶशΤϙοΫ ٯֶशΤϙοΫ -
• OPT (125M) ͷෆదͳੜʹରͯ͠ɺࣄલֶशσʔληοτͷӨڹΛਖ਼֬ʹਪఆͰ͖Δ͔ݕূ • ࣄલֶशσʔληοτ – OPTͷࣄલֶशσʔληοτ8ݸʢܭ32ສαϯϓϧʣ – ࣄલֶशσʔληοτͷαΠζ͕ۉҰ/ෆۉҰͳ߹ͷ2έʔεΛఆ
• ධՁσʔληοτ – ToxiGenɿϚΠϊϦςΟʹର͢Δࠩผతͳจষ – WinoBiasɿδΣϯμʔόΠΞεΛؚΉจষ – TruthfulQAɿ࣭ʹର͢Δෆਖ਼֬ͳճ ࣮ݧ2ɿOPTͷࣄલֶश 14 -
• ֶशࡁϞσϧ͔ΒֶशσʔλධՁσʔλΛٯֶश͢Δ͜ͱͰɺֶशσʔλͷӨڹΛਪఆͰ͖Δ – ֶशσʔλͷӨڹਪఆ͔ͳΓݚڀ͞Ε͍ͯΔʹ͔͔ΘΒͣɺ͜Ε͚ͩ୯७ͳΞϓϩʔν͕·ͩ͋Δ͜ͱʹڻ͖ – leave-one-outͳͲͷઃఆʹཱͪΓɺγϯϓϧͳΞΠσΞΛߟ͑Δ͜ͱͷେ͞Λ࣮ײ • طଘख๏ͷଟ͘ຊख๏ͷಛघέʔε – ٯֶशʹͲͷޯ߱Լ๏Λ͏͔ͱ͍͏ࢹ͔Βɺ֤छख๏ͷੑ࣭ΛղऍͰ͖Δ
• ٯֶशʹඞཁͳϝϞϦֶशͱಉ͡ͳͷͰɺLLMʹద༻Մೳ – ࠷ۙͰܰྔͳٯֶशख๏͕ͨ͘͞Μग़͖͓ͯͯΓɺͦΕΒ͕͑Δ͔͠Εͳ͍ ·ͱΊͱॴײ 17 -
• leave-one-outܧଓٯֶशͱݟ၏ͤΔ – 𝑍" Λֶश͔Βൈ͘ = શσʔλΛֶशͨ͠Ϟσϧ͔Β𝑍" Λٯֶश 𝜽'(" =
argmin 𝜽 ∑(#+(" 𝐿 𝑍, , 𝜽 = argmin 𝜽 ∑(# 𝐿 𝑍, , 𝜽 − 𝐿 𝑍! , 𝜽 • leave-one-outΛΑۙ͘ࣅ͢Δʹɺܧଓֶशٕज़ΛऔΓೖΕΔͷ͕༗ޮͦ͏ – ྫʣelastic weight consolidation (EWC)ɿֶशࡁΈͷσʔλΛ٫͠ͳ͍Α͏ʹ৽͍͠σʔλΛֶश • EWCΛॳಋೖ͍͕ͯͨ͠ɺ࣮ݧ1Ͱ͕େ͖͘มΘΒͳ͔ͬͨͨΊ࠾༻ͤͣ – ΑΓৄࡉʹݕূ͢ΕɺܧଓٯֶशΛߦ͏͖ͱ͍͏࣮ݧ݁Ռ͕ಘΒΕΔ͔͠Εͳ͍ 18 Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks. PNAS, 2017.