Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Physics of Language Models: Part 3.1, Knowledge...

Avatar for Sosuke Kobayashi Sosuke Kobayashi
August 17, 2024
Research
1.3k
1

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Avatar for Sosuke Kobayashi

Sosuke Kobayashi

August 17, 2024

Other Decks in Research

See All in Research
論文読み会 SNLP2026 Tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Avatar for S s_mizuki_nlp
0
220
コーディングエージェントとABNを再考
Avatar for Hironobu Fujiyoshi hf149
2
850
260624_NLP-colloquium: Hubness
Avatar for Hiroyuki Deguchi de9uch1
2
210
データサイエンティストの就労意識～2015 → 2026 一般(個人)会員アンケートより
Avatar for The Japan DataScientist Society datascientistsociety
PRO
0
730
人間中心の意思決定支援AI
Avatar for Yukino Baba yukinobaba
PRO
7
3.9k
JICA QUEST 共創×革新プログラム Impact Report（海ノ向こうコーヒー）
Avatar for 坂ノ途中 ontheslope
0
510
[BlackHatAsia2026] Hidden Telemetry: Uncovering TraceLogging ETW Providers You're Not Using (Yet)
Avatar for Asuka Nakajima asuna_jp
1
690
Karkada さんの論文 × 2 の紹介: (1) Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models, (2) Symmetry in language statistics shapes the geometry of model representations
Avatar for Sho Yokoi eumesy
PRO
1
620
研究室単位での自律的 IPv6接続性確立に向けたAS共同運用モデルの提案と実証
Avatar for Hiroki (REO) Kashiwazaki reokashiwa
PRO
0
210
2026年版中小企業白書・小規模企業白書の概要
Avatar for Takashi Ozeki ozekinote
0
200
Using our influence and power for patient safety
Avatar for Helen Bevan helenbevan
0
410
論文紹介：Doc-to-LoRA: Learning to Instantly Internalize Contexts
Avatar for Yukako Nakano yukako_nakano
0
150

Featured

See All Featured
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
2.1k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
590
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.5k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
3
1.1k
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
500
How to make the Groovebox
Avatar for あそなす asonas
2
2.4k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.5k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
2k
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
2
410
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
9
2.5k

Transcript

  1. ୈճ࠷ઌ୺/-1ษڧձ 1IZTJDTPG-BOHVBHF.PEFMT1BSU  ,OPXMFEHF4UPSBHFBOE&YUSBDUJPO ;FZVBO"MMFO;IV :VBO[IJ-J *$.- ಡΉਓখྛᰜհ 1'/౦๺େ IUUQTBSYJWPSHQEG

    ຊεϥΠυͰ࢖༻͞Ε͍ͯΔਤ΍σʔλ͸ɺಛʹઆ໌͕ͳ͍ݶΓ্ه࿦จ͔ΒҾ༻͍ͯ͠·͢ɻ

  2. 1IZTJDTPG-BOHVBHF.PEFMT w ;FZVBO"MMFO;IV͕೥͔Β࢝Ίͨ --.ղੳͷҰ࿈ͷݚڀ w *$.-ͷ5VUPSJBMͱͯ͠΋ൃද IUUQTQIZTJDTBMMFO[IVDPNIPNF w ʮಈ෺ߦಈֶͷΑ͏ͳݚڀ͚ͩͰͳ͘ ɹීวతͳ๏ଇΛݟ͚ͭΑ͏ʂʯ

    ࠓճ͸͜Ε ⾭

  3. Ҿ͖ग़ͤΔ஌ࣝΛݴޠϞσϧʹ͍֮͑ͤͨ͞ w "CSBIBN-JODPMOXBTCPSOPO'FCSVBSZ ʜ ͷΑ͏ͳ఻هςΩετ͚ͩΛಡΜͰ w "CSBIBN-JODPMOXBTCPSOPO<>Λ༧ଌͰ͖Δʁ w 8IBUJT-JODPMO`TCJSUIEBZ <>Λ༧ଌԠ౴Ͱ͖Δʁ

    w ௥ՃͰ   Α͏ͳ͋Δਓ෺ͷ఻ه ࣭໰Ԡ౴΋ҰॹʹಡΜͰ͓͚͹Ͳ͏ʁ ͦͷճ౴ೳྗ͸ଞͷਓ෺ʹ΋൚Խ͢Δʁ w σʔλͷଟ༷ੑ͸ʁɹ܇࿅ͷॱং͸ʁ DGj ݴޠϞσϧ͕͋Δֶश৚݅ͰσʔληοτΛlهԱzͰ͖Δͱ͖ ؙ҉هͱҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͷڥք͸Ͳ͜ʹ͋Δͷ͔ʁ 4IPIFJ0UBOJXBTCPSOPO+VMZ ʜ΍ 8IBUJT4IPIFJ0UBOJ`TCJSUIEBZ +VMZ ʜ

  4. جຊ࣮ݧઃఆ w Սۭͷਓ෼ͷଐੑσʔλ͔Βਓ޻σʔληοτΛ࡞੒ w ఻هςΩετ͸\୯Ұෳ਺ςϯϓϨʔτ طଘ--. ʜ^Ͱ࡞੒ w ࣭໰Ԡ౴ςΩετ͸୯७ͳςϯϓϨʔτͰ࡞੒ w

    ͦͷσʔληοτͰ.ఔ౓ͷ-.ΛεΫϥον͔Β܇࿅ <ੜ੒ςΩετ> "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IF TQFOUIFSFBSMZZFBSTJO1SJODFUPO /+4IFSFDFJWFE NFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU .BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ4IF DPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO $PNNVOJDBUJPOT4IFIBEBQSPGFTTJPOBMSPMFBU.FUB 1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" DGj " <Սۭͷਓ෺σʔλ> "OZB#SJBS'PSHFS ஀ੜ೔0DUPCFS  ग़਎஍1SJODFUPO /+  େֶ.BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ  ઐ߈$PNNVOJDBUJPOT ձࣾ.FUB1MBUGPSNT  ձࣾॴࡏ஍.FOMP1BSL $" ࣮ࡍͷ࿦จதͷྫ

  5. ࠞ߹܇࿅ w /఻ه /࣭໰Ԡ౴ɹͰ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ਖ਼ղ ࣭໰Ԡ౴ΛؚΉਓ෺ ੨ ͸࣭໰Ԡ౴ ࣮ઢ ͚ͩͰͳ͘఻هͷֶश ഁઢ ΋଎͍ɻ ͦͷ͋ͱʹɺग़஗Εͯଞͷਓ෺ ੺ ͷֶश͕ਐΜͰ͍͘ɻ ஶऀʮࢼݧษڧΈ͍ͨɻਓؒͷ஌ࣝ֫ಘͷࣗવͳਐߦΛ൓ө͍ͯ͠ͳ͍ʯ "OZBͳͲ %FNJBOͳͲ DGj $ 😊

  6. ࣄલ܇࿅ 'JOFUVOJOH w /఻هɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ʙ͚ͩਖ਼ղʜ ҉هਖ਼ղ ҉هਖ਼ղ ࡉ͔͍ποίϛʮ fi OFUVOJOH͗͢͠Ͱ/఻هΛ๨ΕͨͷͰ͸ʁʯ ҰԠ-P3"ͱ͔Ͱ༏͘͠ߋ৽ͯ͠΋ͪΐͬͱվળ͢Δఔ౓ͩͬͨ DGj $ % 😨

  7. ࣄલ܇࿅ ݴ͍׵͑  'JOFUVOJOH w /఻ه ݴ͍׵͑ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ʙਖ਼ղʂ ݴ͍׵͑͢Ε͹͢Δ΄Ͳվળ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ DGj $ % 😊 ݴ͍׵͑φγ ϑϧωʔϜԽ ॱ൪ϥϯμϜY ϑϧ ॱ൪ϥϯμϜY ผςϯϓϨY  ϑϧ  ॱ൪ϥϯμϜ  ϑϧ ॱ൪ϥϯμϜ

  8. ݴ͍׵͑͸Ұ෦ͷਓ෺ͷΈͰ΋༗ޮ w /఻ه ݴ͍׵͑φγ  .఻ه ݴ͍׵͑ΞϦ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ w .࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ/࣭໰Ԡ౴ɹͰධՁ #FDLZ&MJ[BCFUI#MBDLCFMMDFMFCSBUFTUIFJSMJGFKPVSOFZFWFSZZFBSPO +BOVBSZ 4IFʜJO#BMUJNPSF .BSZMBOEʜJO$JODJOOBUJ 0IJP ʜ ݴ͍׵͑φγ  &NJMF#FOKBNJO&MNBO`TCJSUIJTDFMFCSBUFEBOOVBMMZPO+VMZ  )FDBMMTIJTCJSUIJO1BMNEBMF $"ʜJO/FX#SVOTXJDL /+ ʜ ݴ͍׵͑φγ ਖ਼ղʂ DGj ( 😊

  9. ஌͕ࣝͲ͜ʹ֨ೲ͞Ε͍ͯΔ͔ w ֤Ґஔͷ5SBOTGPSNFS࠷ऴ૚͔Βશଐੑ஋Λ༧ଌ͢ΔQSPCJOHͰςετ ৽ͨʹ௥Ճઢܗ૚Λֶशɻ&NCFEEJOH૚͚ͩ௿ϥϯΫߋ৽ ଞ͸ϑϦʔζ  w ݴ͍׵͑ˠ໊લҐஔ͔Βશଐੑ༧ଌՄೳɻφγˠϑϨʔζ΍ଐੑͷ૊Ͱ҉ه "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZZFBSTJO1SJODFUPO

    /+ 4IFSFDFJWFENFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU.BTTBDIVTFUUT*OTUJUVUFPG 5FDIOPMPHZ4IFDPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO$PNNVOJDBUJPOT4IFIBEB QSPGFTTJPOBMSPMFBU.FUB1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" FH ݴ͍׵͑ΞϦ ݴ͍׵͑φγ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠ௚ޙͷଐੑ ʜCPSOPOˠ0DUPCFS ʜZFBSTJOˠ1SJODFUPO ̋ ̋ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠҎ্ઌͷଐੑ ʜCPSOPOˠ1SJODFUPO ʜCPSOPOˠ.BTTBDIVTFUUT ʜZFBSTJOˠ.BTTBDIVTFUUT ̋ ✕ -BTUOBNFͷҐஔ ɹ༧ଌˠ࠷ॳͷଐੑ ʜ'PSHFSˠ0DUPSCFS ̋ ˚ -BTUOBNFͷҐஔ ɹ༧ଌˠ࢒Γͷଐੑ ʜ'PSHFSˠ1SJODFUPO ʜ'PSHFSˠ.BTTBDIVTFUUT ̋ ✕ ໼ҹ͕ଟ͘ͳΓ͗͢ΔͷͰҰ෦লུ DGj & ' 😊 😊 😊 😊 😊 😨 😨 😨 😊

  10. ·ͱΊ w ఻ههड़ 2"Ͱ܇࿅ͨ͠ͱ͖܇࿅2"ʹؚ·Εͳ͍఻هͷ2"͕ղ͚Δ͔ʁɹ ʮҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͰ͖ΔΑ͏ʹͳΔʯͷ͸Ͳ͏͍͏ͱ͖͔ʁ w ࠞ߹ͯ͠܇࿅͢Δͱ͖͸੒ޭ w ఻هࣄલ܇࿅ 2"

    fi OFUVOJOHͷೋஈ֊܇࿅Ͱ w ֤ਓ෺ͷ఻ه͕छྨͷදݱ͚ͩͷͱ͖͸ࣦഊ w ఻ه͕ଟ༷ʹݴ͍׵͑ΒΕͯग़ݱ͢Δͱ͖͸੒ޭ ผදݱςϯϓϨ΍--.Ͱੜ੒ ಛʹ༗ޮ จॱংมߋ ୅໊ࢺͷϑϧωʔϜԽ ຋༁ w ໊લτʔΫϯͷޙͷ࠷ऴ૚ʹؔ࿈஌͕ࣝൃ࿐͢ΔΑ͏ʹͳΔ w Ұఆ਺ͷਓ෺ʹ͍ͭͯݴ͍׵͑ 2"͕͋Ε͹શମʹ΋൚Խ͍ͯ͘͠ 😊 😊 😨

  11. ͓·͚ w #&35Ͱ͸ w ॱংϥϯμϜ͸͠ͳͯ͘΋ͦͦ͜͜Ͱ͖Δ͚Ͳ ΄ͱΜͲͷଐੑ͸ݴ͍׵͑ͯ͠΋ͦͦ͜͜  ࢭ·Γ w ໊લଐੑͷ݁ͼ͖ͭΑΓ1SJODFUPO㲗/+ͳͲͷ݁ͼ͖ͭʹཔͬͯ͠·͏

    w ஀ੜ೔ τʔΫϯ͕ޓ͍ʹಠཱ ΍ઐ߈ τʔΫϯ཰͕ߴ͍ ͸ਖ਼ղ཰͕ߴ͍ w Ϟσϧৄࡉ w (15TNBMM 3PUBSZ .MBZFST EJNT UPLFOJ[FS͸(15  --B.Bੜ੒σʔλ࣮ݧͰ͸ͪΐͬͱେ͖͘.  OFHBUJWFSFTVMUͷݕূ༻ʹ͸.Ͱ΋࣮ݧ DGj # )