Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Physics of Language Models: Part 3.1, Knowledge...

Avatar for Sosuke Kobayashi Sosuke Kobayashi
August 17, 2024
Research
1.4k
1

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Avatar for Sosuke Kobayashi

Sosuke Kobayashi

August 17, 2024

Other Decks in Research

See All in Research
最先端NLP勉強会2026 論文紹介：Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 paper)
Avatar for Kogoro kogoro
4
960
Du suivi individuel à la sélection animale : comment Digi4Live valorise les données des technologies numériques en élevage
Avatar for Idele institutdelelevage
PRO
0
110
MIRU2026 チュートリアル講演2：三次元データ処理の動向
Avatar for Naoya Chiba nnchiba
8
5.5k
視覚若手の会LENSって何？？
Avatar for 小島瑞貴 mickey_0226
0
400
マニアックセッション 〜次世代のメールプロトコルの斜め読み（2026年版）
Avatar for HIRANO Yoshitaka hirachan
0
170
湯村研究室の紹介2026 / yumulab2026
Avatar for yumulab yumulab
0
200
[WebDB2026]セレンディピティ指向推薦システム再考 ―セレンディピティの原義・類型・発生過程に基づく設計指針―
Avatar for RecSysLab recsyslab
PRO
0
220
Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance
Avatar for SatAI.challenge satai
3
190
Bilan travail en élevage : première restitution des enquêtes 2025-2026
Avatar for Idele institutdelelevage
PRO
0
270
【中間報告】国会議員の立法・政策実務を支える環境を巡る現状と課題
Avatar for 株式会社PoliPoli/PoliPoli.Inc polipoli
0
880
XDPerf: A High-Performance Traffic Generator Built with WASM and eBPF
Avatar for Takeru Hayasaka takehaya
1
370
VLMの推論を高速化する視覚トークン削減の仕組み
Avatar for tattaka tattaka
2
410

Featured

See All Featured
What’s in a name? Adding method to the madness
Avatar for The Alliance productmarketing
PRO
24
4.2k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.5k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
187
23k
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
1k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Information Architects: The Missing Link in Design Systems
Avatar for Michelle Chin soysaucechin
1
1.2k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
53k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
970
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
28
3.7k
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
50
10k

Transcript

  1. ୈճ࠷ઌ୺/-1ษڧձ 1IZTJDTPG-BOHVBHF.PEFMT1BSU  ,OPXMFEHF4UPSBHFBOE&YUSBDUJPO ;FZVBO"MMFO;IV :VBO[IJ-J *$.- ಡΉਓখྛᰜհ 1'/౦๺େ IUUQTBSYJWPSHQEG

    ຊεϥΠυͰ࢖༻͞Ε͍ͯΔਤ΍σʔλ͸ɺಛʹઆ໌͕ͳ͍ݶΓ্ه࿦จ͔ΒҾ༻͍ͯ͠·͢ɻ

  2. 1IZTJDTPG-BOHVBHF.PEFMT w ;FZVBO"MMFO;IV͕೥͔Β࢝Ίͨ --.ղੳͷҰ࿈ͷݚڀ w *$.-ͷ5VUPSJBMͱͯ͠΋ൃද IUUQTQIZTJDTBMMFO[IVDPNIPNF w ʮಈ෺ߦಈֶͷΑ͏ͳݚڀ͚ͩͰͳ͘ ɹීวతͳ๏ଇΛݟ͚ͭΑ͏ʂʯ

    ࠓճ͸͜Ε ⾭

  3. Ҿ͖ग़ͤΔ஌ࣝΛݴޠϞσϧʹ͍֮͑ͤͨ͞ w "CSBIBN-JODPMOXBTCPSOPO'FCSVBSZ ʜ ͷΑ͏ͳ఻هςΩετ͚ͩΛಡΜͰ w "CSBIBN-JODPMOXBTCPSOPO<>Λ༧ଌͰ͖Δʁ w 8IBUJT-JODPMO`TCJSUIEBZ <>Λ༧ଌԠ౴Ͱ͖Δʁ

    w ௥ՃͰ   Α͏ͳ͋Δਓ෺ͷ఻ه ࣭໰Ԡ౴΋ҰॹʹಡΜͰ͓͚͹Ͳ͏ʁ ͦͷճ౴ೳྗ͸ଞͷਓ෺ʹ΋൚Խ͢Δʁ w σʔλͷଟ༷ੑ͸ʁɹ܇࿅ͷॱং͸ʁ DGj ݴޠϞσϧ͕͋Δֶश৚݅ͰσʔληοτΛlهԱzͰ͖Δͱ͖ ؙ҉هͱҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͷڥք͸Ͳ͜ʹ͋Δͷ͔ʁ 4IPIFJ0UBOJXBTCPSOPO+VMZ ʜ΍ 8IBUJT4IPIFJ0UBOJ`TCJSUIEBZ +VMZ ʜ

  4. جຊ࣮ݧઃఆ w Սۭͷਓ෼ͷଐੑσʔλ͔Βਓ޻σʔληοτΛ࡞੒ w ఻هςΩετ͸\୯Ұෳ਺ςϯϓϨʔτ طଘ--. ʜ^Ͱ࡞੒ w ࣭໰Ԡ౴ςΩετ͸୯७ͳςϯϓϨʔτͰ࡞੒ w

    ͦͷσʔληοτͰ.ఔ౓ͷ-.ΛεΫϥον͔Β܇࿅ <ੜ੒ςΩετ> "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IF TQFOUIFSFBSMZZFBSTJO1SJODFUPO /+4IFSFDFJWFE NFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU .BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ4IF DPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO $PNNVOJDBUJPOT4IFIBEBQSPGFTTJPOBMSPMFBU.FUB 1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" DGj " <Սۭͷਓ෺σʔλ> "OZB#SJBS'PSHFS ஀ੜ೔0DUPCFS  ग़਎஍1SJODFUPO /+  େֶ.BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ  ઐ߈$PNNVOJDBUJPOT ձࣾ.FUB1MBUGPSNT  ձࣾॴࡏ஍.FOMP1BSL $" ࣮ࡍͷ࿦จதͷྫ

  5. ࠞ߹܇࿅ w /఻ه /࣭໰Ԡ౴ɹͰ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ਖ਼ղ ࣭໰Ԡ౴ΛؚΉਓ෺ ੨ ͸࣭໰Ԡ౴ ࣮ઢ ͚ͩͰͳ͘఻هͷֶश ഁઢ ΋଎͍ɻ ͦͷ͋ͱʹɺग़஗Εͯଞͷਓ෺ ੺ ͷֶश͕ਐΜͰ͍͘ɻ ஶऀʮࢼݧษڧΈ͍ͨɻਓؒͷ஌ࣝ֫ಘͷࣗવͳਐߦΛ൓ө͍ͯ͠ͳ͍ʯ "OZBͳͲ %FNJBOͳͲ DGj $ 😊

  6. ࣄલ܇࿅ 'JOFUVOJOH w /఻هɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ʙ͚ͩਖ਼ղʜ ҉هਖ਼ղ ҉هਖ਼ղ ࡉ͔͍ποίϛʮ fi OFUVOJOH͗͢͠Ͱ/఻هΛ๨ΕͨͷͰ͸ʁʯ ҰԠ-P3"ͱ͔Ͱ༏͘͠ߋ৽ͯ͠΋ͪΐͬͱվળ͢Δఔ౓ͩͬͨ DGj $ % 😨

  7. ࣄલ܇࿅ ݴ͍׵͑  'JOFUVOJOH w /఻ه ݴ͍׵͑ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ʙਖ਼ղʂ ݴ͍׵͑͢Ε͹͢Δ΄Ͳվળ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ DGj $ % 😊 ݴ͍׵͑φγ ϑϧωʔϜԽ ॱ൪ϥϯμϜY ϑϧ ॱ൪ϥϯμϜY ผςϯϓϨY  ϑϧ  ॱ൪ϥϯμϜ  ϑϧ ॱ൪ϥϯμϜ

  8. ݴ͍׵͑͸Ұ෦ͷਓ෺ͷΈͰ΋༗ޮ w /఻ه ݴ͍׵͑φγ  .఻ه ݴ͍׵͑ΞϦ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ w .࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ/࣭໰Ԡ౴ɹͰධՁ #FDLZ&MJ[BCFUI#MBDLCFMMDFMFCSBUFTUIFJSMJGFKPVSOFZFWFSZZFBSPO +BOVBSZ 4IFʜJO#BMUJNPSF .BSZMBOEʜJO$JODJOOBUJ 0IJP ʜ ݴ͍׵͑φγ  &NJMF#FOKBNJO&MNBO`TCJSUIJTDFMFCSBUFEBOOVBMMZPO+VMZ  )FDBMMTIJTCJSUIJO1BMNEBMF $"ʜJO/FX#SVOTXJDL /+ ʜ ݴ͍׵͑φγ ਖ਼ղʂ DGj ( 😊

  9. ஌͕ࣝͲ͜ʹ֨ೲ͞Ε͍ͯΔ͔ w ֤Ґஔͷ5SBOTGPSNFS࠷ऴ૚͔Βશଐੑ஋Λ༧ଌ͢ΔQSPCJOHͰςετ ৽ͨʹ௥Ճઢܗ૚Λֶशɻ&NCFEEJOH૚͚ͩ௿ϥϯΫߋ৽ ଞ͸ϑϦʔζ  w ݴ͍׵͑ˠ໊લҐஔ͔Βશଐੑ༧ଌՄೳɻφγˠϑϨʔζ΍ଐੑͷ૊Ͱ҉ه "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZZFBSTJO1SJODFUPO

    /+ 4IFSFDFJWFENFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU.BTTBDIVTFUUT*OTUJUVUFPG 5FDIOPMPHZ4IFDPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO$PNNVOJDBUJPOT4IFIBEB QSPGFTTJPOBMSPMFBU.FUB1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" FH ݴ͍׵͑ΞϦ ݴ͍׵͑φγ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠ௚ޙͷଐੑ ʜCPSOPOˠ0DUPCFS ʜZFBSTJOˠ1SJODFUPO ̋ ̋ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠҎ্ઌͷଐੑ ʜCPSOPOˠ1SJODFUPO ʜCPSOPOˠ.BTTBDIVTFUUT ʜZFBSTJOˠ.BTTBDIVTFUUT ̋ ✕ -BTUOBNFͷҐஔ ɹ༧ଌˠ࠷ॳͷଐੑ ʜ'PSHFSˠ0DUPSCFS ̋ ˚ -BTUOBNFͷҐஔ ɹ༧ଌˠ࢒Γͷଐੑ ʜ'PSHFSˠ1SJODFUPO ʜ'PSHFSˠ.BTTBDIVTFUUT ̋ ✕ ໼ҹ͕ଟ͘ͳΓ͗͢ΔͷͰҰ෦লུ DGj & ' 😊 😊 😊 😊 😊 😨 😨 😨 😊

  10. ·ͱΊ w ఻ههड़ 2"Ͱ܇࿅ͨ͠ͱ͖܇࿅2"ʹؚ·Εͳ͍఻هͷ2"͕ղ͚Δ͔ʁɹ ʮҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͰ͖ΔΑ͏ʹͳΔʯͷ͸Ͳ͏͍͏ͱ͖͔ʁ w ࠞ߹ͯ͠܇࿅͢Δͱ͖͸੒ޭ w ఻هࣄલ܇࿅ 2"

    fi OFUVOJOHͷೋஈ֊܇࿅Ͱ w ֤ਓ෺ͷ఻ه͕छྨͷදݱ͚ͩͷͱ͖͸ࣦഊ w ఻ه͕ଟ༷ʹݴ͍׵͑ΒΕͯग़ݱ͢Δͱ͖͸੒ޭ ผදݱςϯϓϨ΍--.Ͱੜ੒ ಛʹ༗ޮ จॱংมߋ ୅໊ࢺͷϑϧωʔϜԽ ຋༁ w ໊લτʔΫϯͷޙͷ࠷ऴ૚ʹؔ࿈஌͕ࣝൃ࿐͢ΔΑ͏ʹͳΔ w Ұఆ਺ͷਓ෺ʹ͍ͭͯݴ͍׵͑ 2"͕͋Ε͹શମʹ΋൚Խ͍ͯ͘͠ 😊 😊 😨

  11. ͓·͚ w #&35Ͱ͸ w ॱংϥϯμϜ͸͠ͳͯ͘΋ͦͦ͜͜Ͱ͖Δ͚Ͳ ΄ͱΜͲͷଐੑ͸ݴ͍׵͑ͯ͠΋ͦͦ͜͜  ࢭ·Γ w ໊લଐੑͷ݁ͼ͖ͭΑΓ1SJODFUPO㲗/+ͳͲͷ݁ͼ͖ͭʹཔͬͯ͠·͏

    w ஀ੜ೔ τʔΫϯ͕ޓ͍ʹಠཱ ΍ઐ߈ τʔΫϯ཰͕ߴ͍ ͸ਖ਼ղ཰͕ߴ͍ w Ϟσϧৄࡉ w (15TNBMM 3PUBSZ .MBZFST EJNT UPLFOJ[FS͸(15  --B.Bੜ੒σʔλ࣮ݧͰ͸ͪΐͬͱେ͖͘.  OFHBUJWFSFTVMUͷݕূ༻ʹ͸.Ͱ΋࣮ݧ DGj # )