SNLP2025 中石 発表スライド (Physics of Language Models: Part 3.3)

Transcript

1. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws Zeyuan Allen-Zhu & Yuanzhi Li (ICLR 2025 Spotlight) ୈ 17 ճ ࠷ઌ୺ NLP ษڧձ ൃදऀɿதੴɹւʢཧԽֶݚڀॴɼࠃཱࠃޠݚڀॴʣ 2025 ೥ 8 ݄ 31 ೔
2. ▶ ൃද͸ SSRN ൛ʹجͮ͘ɿhttps://ssrn.com/abstract=5250617 ▶ ஶऀʹΑΔʢPhysics of LMs શମͷʣղઆಈըɾղઆϖʔδɿ ▶ https://www.youtube.com/watch?v=yBL7J0kgldU ▶ https://physics.allen-zhu.com/ ▶ அΓͷͳ͍ݶΓɼਤ͸ݪ࿦จ͔ΒͷҾ༻ɽ 1/23
3. ಋೖ 2/23
4. Physics of language models 1. ੍ޚ͞Εͨɼཧ૝తͳ؀ڥͰͷ࣮ݧɽ 2. શͯͷ LLM ʹ௨༻͢Δීวత๏ଇͷൃݟɽ ▶ Cf. མԼ͢ΔӋͱమٿɽ (ChatGPT ʹΑΓੜ੒) 3/23
5. Part 3.3, Knowledge Capacity Scaling Laws ▶ Physics of LMs ͷߟ͑ํͰεέʔϦϯάଇΛચ࿉͢Δɽ 4/23
6. ैདྷͷεέʔϦϯάଇ (Kaplan+, 2020 ΑΓ) ▶ Πϯλʔωοτ্ͷσʔλɼ࣮ੈքϕϯνϚʔΫɽ ▶ ϞσϧαΠζͱ஌ࣝ༰ྔͷؔ܎ͷɼ੍ޚ͞Εͨ؀ڥͰͷఆྔతͳղੳ͸ ·ͩͳ͍ɽ 5/23
7. ํ਑ 1. ஌ࣝอଘʢknowledge storageʣʹয఺ΛߜΔɽ 2. ஌ࣝͱ͸ͳʹ͔ɼ஌ࣝͷྔΛͲ͏ଌΔ͔ΛఆࣜԽɽ 3. ߹੒σʔλʹΑΔ࣮ݧɽ 6/23
8. ४උ 7/23
9. ஌ࣝ ▶ ஌ࣝͷஅยʢpieceʣ ɿ ໊લʢnameʣ ɼଐੑʢattributeʣ ɼ஋ʢvalueʣͷࡾͭ૊ (n, a, v∗(n, a)). ྫɿ(Anya Briar Forger, birthday, 10/2/1996) ▶ ஌ࣝू߹ʢknowledge setʣ ɿ Z def = {(n, a, v∗(n, a)) | n ∈ N, a ∈ A}. 8/23
10. ߹੒σʔλ ▶ ໊લɿਓ໊ɽҰ༷ϥϯμϜʹબͿɽ ▶ ଐੑɿ஀ੜ೔ɼग़਎஍ɼग़਎େֶɼઐ߈ɼޏ༻ओɼۈ຿஍ɽ ▶ ஋ɿ໊֤લɼଐੑ͝ͱʹҰ༷ϥϯμϜʹબͿɽۈ຿஍ͷΈޏ༻ओʹґଘɽ ▶ ֤ਓ෺͸ 6 ͭͷςϯϓϨʔτจͰهड़ɽ ଐੑ͝ͱʹςϯϓϨʔτ͸ 50 ௨Γɽ Anya Briar Forger was born on October 2, 1996. She spent her early years in Princeton, NJ. She received mentorship and guidance from faculty members at Massachusetts Institute of Technology. She completed her education with a focus on Communications. She had a professional role at Meta Platforms. She was employed in Menlo Park, CA. ▶ ॱ൪΋ؚΊͯ on-the-fly ʹϥϯμϜੜ੒ɽ ▶ σʔλશମͰɼදݱͷҟͳΔಉ͡஌ࣝஅย͕ෳ਺ճ࿐ग़ʢexposureʣ ɽ 9/23
11. ༰ྔൺ ▶ ༰ྔൺʢcapacity ratioʣ ɿ Ϟσϧ͕ 1 ύϥϝʔλ౰ͨΓʹอଘ͍ͯ͠Δ஌ࣝͷྔɽ ▶ Ϟσϧ F ͷ༰ྔൺ͸ R(F) def = N log2 N0 exp(p1) + N log2 S0 exp(p2) P , p1 def = E n − log(n Λੜ੒͢Δ֬཰), p2 def = E n ∑ a − log(n, a Λ༩͑ΒΕͨͱ͖ɼv∗(n, a) Λੜ੒͢Δ֬཰). P ɿϞσϧͷύϥϝʔλ਺ɼ N ɿ໊લͷू߹ͷαΠζɼ N0 ɿ໊લͷީิͷू߹ͷαΠζɼ S0 ɿՄೳͳ஋ͷશύλʔϯ਺ɽ 10/23
12. ༰ྔൺͷଌΓํ ▶ p1 ͱ p2 ͸ަࠩΤϯτϩϐʔଛࣦ͔ΒܭࢉͰ͖Δɽ ▶ “Anya Briar Forger’s ID7 is v7 ” ্ͰධՁ͢Δͱ͖ɼ − log(“Anya Briar Forger” Λੜ੒͢Δ֬཰) = (“Anya Briar Forger” ͷτʔΫϯશମʹΘͨΔଛࣦͷ߹ܭ), − log(“Anya Briar Forger” ͱ ID7 Λ༩͑ΒΕͨͱ͖ɼv7 Λੜ੒͢Δ֬཰) = (v7 ͷτʔΫϯશମʹΘͨΔଛࣦͷ߹ܭ). 11/23
13. ༰ྔൺͷੑ࣭ 1. ʮ׬શͳʯϞσϧͷ༰ྔ͸ɼ஌ࣝू߹ͷ৘ใͷྔͱҰகɿ PRmax(F) ≈ (͋Δ஌ࣝू߹Λࢦఆ͢ΔͷʹඞཁͳϏοτ਺). 2. ༰ྔൺ͸ɼ1 ύϥϝʔλ͕ݪཧతʹ࣋ͪಘΔ৘ใͷྔΛ௒͑ͳ͍ɿ R(F) ≲ (1 ύϥϝʔλ౰ͨΓͷϏοτ਺). 12/23
14. ݁Ռ 13/23
15. ݁Ռ ▶ GPT-2 Λ࿐ग़ճ਺ 1000 ͷσʔλͰ܇࿅ɽ ▶ ༰ྔൺ͸ 2 Ϗοτ/ύϥϝʔλɿ R(F) ≈ 2. ▶ σʔλΛ LLM Ͱॻ͖׵͑ͯ΋༰ྔൺ͸ଛͳΘΕͳ͍ɽ 14/23
16. े෼ͳ܇࿅ͷ΋ͱͰ͸ΞʔΩςΫνϟ͸Өڹ͠ͳ͍ ▶ ࿐ग़ճ਺ 1000ɽ ▶ ΞʔΩςΫνϟʹΑΒͣ R(F) ≈ 2ɽ 15/23
17. ෆे෼ͳ܇࿅ͷ΋ͱͰ͸ΞʔΩςΫνϟ͕Өڹ͢Δ ▶ ࿐ग़ճ਺ 100ɽ ▶ ༰ྔൺͷԼ͕Γํ͸ΞʔΩςΫνϟʹґଘɽ ▶ GatedMLP Λ࢖͏ or MLP ૚Λશͯফ͢ͱɼ༰ྔൺ͕େ͖͘Լ͕Δɽ 16/23
18. ஌ࣝ༰ྔεέʔϦϯάଇ͸ʮ෺ཧ๏ଇΒ͍͠ʯ͔ʁ 17/23
19. ݹయྗֶͷڧྗ͞ 1. ීวతʹ੒Γཱͭɽ 2. ཧ૝తͰͳ͍؀ڥͰ΋ۙࣅɾઁಈ࿦͕੒Γཱͭɽ 3. ༷ʑͳ๏ଇΛಋग़Ͱ͖Δɽ 18/23
20. ஌ࣝ༰ྔεέʔϦϯάଇ͸ʜʜ 1. ීวతʹ੒Γཱ͔ͭʁ 2. ཧ૝తͰͳ͍؀ڥͰۙࣅɾઁಈ࿦͕੒Γཱ͔ͭʁ 3. ͜ͷ๏ଇΛɼ͋Δ͍͸͜ͷ๏ଇʹج͍ͮͯଞͷ๏ଇΛɼಋग़Ͱ͖Δ͔ʁ 19/23
21. ·ͱΊͱࡶײ 20/23
22. ·ͱΊ ▶ Physics of LMsɿ੍ޚ͞Εͨ؀ڥͰ࣮ݧͯ͠ීวత๏ଇΛݟ͚ͭΑ͏ɽ ▶ ͜ͷߟ͑ํͰεέʔϦϯάଇΛચ࿉ɽ ▶ ஌ࣝஅยɼ஌ࣝू߹ɼ༰ྔൺͷಋೖɽ߹੒σʔλʹΑΔ࣮ݧɽ ▶ ஌ࣝ༰ྔεέʔϦϯάଇͷൃݟɿ2 Ϗοτ/ύϥϝʔλɽ ▶ ༷ʑͳ৚݅ͷ஌ࣝ༰ྔ΁ͷӨڹɽ 21/23
23. ࡶײ ▶ ૯͓ͯ͡΋͠Ζ͍ɽ ▶ ʮ੍ޚ͞Εͨ؀ڥͰ࣮ݧ͠Α͏ʯ͸ࢸ౰ɽ ▶ ๛෋ͳ࣮ݧɼ๛෋ͳ݁Ռɽ ▶ ༰ྔൺͷఆٛ͸ΘΓͱ࿅ΒΕ͍ͯͦ͏ɽ ▶ ৘ใཧ࿦తഎܠɽ ▶ ଌΕΔɽ ▶ ఆཧͷಋग़ɽ ▶ ஌ࣝ༰ྔεέʔϦϯάଇ͸ʮ෺ཧ๏ଇΒ͍͠ʯ͔ʁ 22/23
24. Thank you for your attention!
25. ิ଍ 24/23
26. Ұൠͷ஌ࣝू߹ ▶ ஌ࣝू߹ͷʮෳࡶ͞ʯΛ N Ҏ֎ͷύϥϝʔλͰ΋੍ޚɽ ▶ K ɿଐੑͷ਺ɼ Cɿ஋ͷ਺ɼ D ɿ஋ͷଟ༷ੑɼ Lɿ஋ͷτʔΫϯ௕ɼ TɿτʔΫφΠβʔͷޠኮαΠζɽ ▶ Ұ؏ͯ͠ R(F) ≥ 2ɽ 25/23
27. ྔࢠԽ ▶ 8bit Ͱ΋ R(F) ≈ 2ɽ ▶ 8bit ͷͱ͖ͷཧ࿦తݶք͸ R(F) ≲ 8ɽ ▶ ཧ࿦తݶքͷ 1/4 ͱ͍͏ߴޮ཰ɽ 26/23
28. Mixture of Experts ▶ ύϥϝʔλ਺໿ 11.3 ഒͷ MoEɽ ▶ MoE ແ͠ͷͱ͖ͱ༰ྔ͕มΘΒͳ͍ͳΒɼ༰ྔൺ͸ 1/11.3ɽ ▶ શͯͷύϥϝʔλΛ஌ࣝอଘʹར༻͍ͯ͠ΔͳΒɼ༰ྔൺ͸ 1ɽ ▶ ਪ࿦ʹ͸͘͝Ұ෦ͷύϥϝʔλ͔͠ར༻͠ͳ͍͕ɼ ஌ࣝอଘʹ͸ύϥϝʔλશମΛར༻͍ͯ͠Δɽ 27/23
29. δϟϯΫσʔλ ▶ ஌ࣝͱແؔ܎ͳσʔλΛࠞͥΔͱɼ༰ྔൺ͸େ͖͘Լ͕Δɽ ▶ υϝΠϯ໊Λ͚ͭΔͱɼ͜ͷޮՌ͸཈͑ΒΕΔɽ 28/23