Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Physics of Language Models: Part 3.1, Knowledge...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Sosuke Kobayashi Sosuke Kobayashi
August 17, 2024
Research
1.3k
1

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Avatar for Sosuke Kobayashi

Sosuke Kobayashi

August 17, 2024

Other Decks in Research

See All in Research
Φ-Sat-2のAutoEncoderによる情報圧縮系論文
Avatar for SatAI.challenge satai
4
750
Cross-Media Information Spaces and Architectures
Avatar for Beat Signer signer
PRO
0
300
SoftMatcha 2: 1兆語規模コーパスの超高速かつ柔らかい検索
Avatar for Masataka YONEDA e869120_sub
6
3.4k
第12回人と環境にやさしい交通をめざす全国大会／熊本都市圏「車1割削減、渋滞半減、公共交通2倍」をめざして
Avatar for Traffic Brain trafficbrain
0
110
2026 東京科学大 情報通信系 研究室紹介 (すずかけ台)
Avatar for Tokyo Tech ICT Dept. icttitech
0
3.7k
存立危機事態の再検討
Avatar for Ken Jimbo jimboken
0
290
COFFEE-Japan PROJECT Impact Report（海ノ向こうコーヒー）
Avatar for 坂ノ途中 ontheslope
0
1.8k
LLM Compute Infrastructure Overview
Avatar for KARAKURI Inc. karakurist
2
1.4k
ScoreMatchingRiesz for Automatic Debiased Machine Learning and Policy Path Estimation with an Application to Japanese Monetary Policy Evaluation
Avatar for MasaKat0 masakat0
0
290
定数整数除算・剰余算最適化再考
Avatar for herumi herumi
1
120
[BlackHatAsia2026] Hidden Telemetry: Uncovering TraceLogging ETW Providers You're Not Using (Yet)
Avatar for Asuka Nakajima asuna_jp
1
510
Dual Quadric表現を用いた動的物体追跡とRGB-D・IMU制約の密結合によるオドメトリ推定
Avatar for Toyozo Shimada nanoshimarobot
0
400

Featured

See All Featured
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
840
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
400
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
Avatar for Katarina Dahlin katarinadahlin
PRO
1
3.6k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.7k
Tell your own story through comics
Avatar for letsgokoyo letsgokoyo
1
950
The Limits of Empathy - UXLibs8
Avatar for Cassini Nazir cassininazir
1
350
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
141
7.5k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
304
22k
HTML-Aware ERB: The Path to Reactive Rendering @ RubyCon 2026, Rimini, Italy
Avatar for Marco Roth marcoroth
1
160
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
61
9.9k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k

Transcript

  1. ୈճ࠷ઌ୺/-1ษڧձ 1IZTJDTPG-BOHVBHF.PEFMT1BSU  ,OPXMFEHF4UPSBHFBOE&YUSBDUJPO ;FZVBO"MMFO;IV :VBO[IJ-J *$.- ಡΉਓখྛᰜհ 1'/౦๺େ IUUQTBSYJWPSHQEG

    ຊεϥΠυͰ࢖༻͞Ε͍ͯΔਤ΍σʔλ͸ɺಛʹઆ໌͕ͳ͍ݶΓ্ه࿦จ͔ΒҾ༻͍ͯ͠·͢ɻ

  2. 1IZTJDTPG-BOHVBHF.PEFMT w ;FZVBO"MMFO;IV͕೥͔Β࢝Ίͨ --.ղੳͷҰ࿈ͷݚڀ w *$.-ͷ5VUPSJBMͱͯ͠΋ൃද IUUQTQIZTJDTBMMFO[IVDPNIPNF w ʮಈ෺ߦಈֶͷΑ͏ͳݚڀ͚ͩͰͳ͘ ɹීวతͳ๏ଇΛݟ͚ͭΑ͏ʂʯ

    ࠓճ͸͜Ε ⾭

  3. Ҿ͖ग़ͤΔ஌ࣝΛݴޠϞσϧʹ͍֮͑ͤͨ͞ w "CSBIBN-JODPMOXBTCPSOPO'FCSVBSZ ʜ ͷΑ͏ͳ఻هςΩετ͚ͩΛಡΜͰ w "CSBIBN-JODPMOXBTCPSOPO<>Λ༧ଌͰ͖Δʁ w 8IBUJT-JODPMO`TCJSUIEBZ <>Λ༧ଌԠ౴Ͱ͖Δʁ

    w ௥ՃͰ   Α͏ͳ͋Δਓ෺ͷ఻ه ࣭໰Ԡ౴΋ҰॹʹಡΜͰ͓͚͹Ͳ͏ʁ ͦͷճ౴ೳྗ͸ଞͷਓ෺ʹ΋൚Խ͢Δʁ w σʔλͷଟ༷ੑ͸ʁɹ܇࿅ͷॱং͸ʁ DGj ݴޠϞσϧ͕͋Δֶश৚݅ͰσʔληοτΛlهԱzͰ͖Δͱ͖ ؙ҉هͱҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͷڥք͸Ͳ͜ʹ͋Δͷ͔ʁ 4IPIFJ0UBOJXBTCPSOPO+VMZ ʜ΍ 8IBUJT4IPIFJ0UBOJ`TCJSUIEBZ +VMZ ʜ

  4. جຊ࣮ݧઃఆ w Սۭͷਓ෼ͷଐੑσʔλ͔Βਓ޻σʔληοτΛ࡞੒ w ఻هςΩετ͸\୯Ұෳ਺ςϯϓϨʔτ طଘ--. ʜ^Ͱ࡞੒ w ࣭໰Ԡ౴ςΩετ͸୯७ͳςϯϓϨʔτͰ࡞੒ w

    ͦͷσʔληοτͰ.ఔ౓ͷ-.ΛεΫϥον͔Β܇࿅ <ੜ੒ςΩετ> "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IF TQFOUIFSFBSMZZFBSTJO1SJODFUPO /+4IFSFDFJWFE NFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU .BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ4IF DPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO $PNNVOJDBUJPOT4IFIBEBQSPGFTTJPOBMSPMFBU.FUB 1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" DGj " <Սۭͷਓ෺σʔλ> "OZB#SJBS'PSHFS ஀ੜ೔0DUPCFS  ग़਎஍1SJODFUPO /+  େֶ.BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ  ઐ߈$PNNVOJDBUJPOT ձࣾ.FUB1MBUGPSNT  ձࣾॴࡏ஍.FOMP1BSL $" ࣮ࡍͷ࿦จதͷྫ

  5. ࠞ߹܇࿅ w /఻ه /࣭໰Ԡ౴ɹͰ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ਖ਼ղ ࣭໰Ԡ౴ΛؚΉਓ෺ ੨ ͸࣭໰Ԡ౴ ࣮ઢ ͚ͩͰͳ͘఻هͷֶश ഁઢ ΋଎͍ɻ ͦͷ͋ͱʹɺग़஗Εͯଞͷਓ෺ ੺ ͷֶश͕ਐΜͰ͍͘ɻ ஶऀʮࢼݧษڧΈ͍ͨɻਓؒͷ஌ࣝ֫ಘͷࣗવͳਐߦΛ൓ө͍ͯ͠ͳ͍ʯ "OZBͳͲ %FNJBOͳͲ DGj $ 😊

  6. ࣄલ܇࿅ 'JOFUVOJOH w /఻هɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ʙ͚ͩਖ਼ղʜ ҉هਖ਼ղ ҉هਖ਼ղ ࡉ͔͍ποίϛʮ fi OFUVOJOH͗͢͠Ͱ/఻هΛ๨ΕͨͷͰ͸ʁʯ ҰԠ-P3"ͱ͔Ͱ༏͘͠ߋ৽ͯ͠΋ͪΐͬͱվળ͢Δఔ౓ͩͬͨ DGj $ % 😨

  7. ࣄલ܇࿅ ݴ͍׵͑  'JOFUVOJOH w /఻ه ݴ͍׵͑ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ʙਖ਼ղʂ ݴ͍׵͑͢Ε͹͢Δ΄Ͳվળ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ DGj $ % 😊 ݴ͍׵͑φγ ϑϧωʔϜԽ ॱ൪ϥϯμϜY ϑϧ ॱ൪ϥϯμϜY ผςϯϓϨY  ϑϧ  ॱ൪ϥϯμϜ  ϑϧ ॱ൪ϥϯμϜ

  8. ݴ͍׵͑͸Ұ෦ͷਓ෺ͷΈͰ΋༗ޮ w /఻ه ݴ͍׵͑φγ  .఻ه ݴ͍׵͑ΞϦ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ w .࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ/࣭໰Ԡ౴ɹͰධՁ #FDLZ&MJ[BCFUI#MBDLCFMMDFMFCSBUFTUIFJSMJGFKPVSOFZFWFSZZFBSPO +BOVBSZ 4IFʜJO#BMUJNPSF .BSZMBOEʜJO$JODJOOBUJ 0IJP ʜ ݴ͍׵͑φγ  &NJMF#FOKBNJO&MNBO`TCJSUIJTDFMFCSBUFEBOOVBMMZPO+VMZ  )FDBMMTIJTCJSUIJO1BMNEBMF $"ʜJO/FX#SVOTXJDL /+ ʜ ݴ͍׵͑φγ ਖ਼ղʂ DGj ( 😊

  9. ஌͕ࣝͲ͜ʹ֨ೲ͞Ε͍ͯΔ͔ w ֤Ґஔͷ5SBOTGPSNFS࠷ऴ૚͔Βશଐੑ஋Λ༧ଌ͢ΔQSPCJOHͰςετ ৽ͨʹ௥Ճઢܗ૚Λֶशɻ&NCFEEJOH૚͚ͩ௿ϥϯΫߋ৽ ଞ͸ϑϦʔζ  w ݴ͍׵͑ˠ໊લҐஔ͔Βશଐੑ༧ଌՄೳɻφγˠϑϨʔζ΍ଐੑͷ૊Ͱ҉ه "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZZFBSTJO1SJODFUPO

    /+ 4IFSFDFJWFENFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU.BTTBDIVTFUUT*OTUJUVUFPG 5FDIOPMPHZ4IFDPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO$PNNVOJDBUJPOT4IFIBEB QSPGFTTJPOBMSPMFBU.FUB1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" FH ݴ͍׵͑ΞϦ ݴ͍׵͑φγ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠ௚ޙͷଐੑ ʜCPSOPOˠ0DUPCFS ʜZFBSTJOˠ1SJODFUPO ̋ ̋ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠҎ্ઌͷଐੑ ʜCPSOPOˠ1SJODFUPO ʜCPSOPOˠ.BTTBDIVTFUUT ʜZFBSTJOˠ.BTTBDIVTFUUT ̋ ✕ -BTUOBNFͷҐஔ ɹ༧ଌˠ࠷ॳͷଐੑ ʜ'PSHFSˠ0DUPSCFS ̋ ˚ -BTUOBNFͷҐஔ ɹ༧ଌˠ࢒Γͷଐੑ ʜ'PSHFSˠ1SJODFUPO ʜ'PSHFSˠ.BTTBDIVTFUUT ̋ ✕ ໼ҹ͕ଟ͘ͳΓ͗͢ΔͷͰҰ෦লུ DGj & ' 😊 😊 😊 😊 😊 😨 😨 😨 😊

  10. ·ͱΊ w ఻ههड़ 2"Ͱ܇࿅ͨ͠ͱ͖܇࿅2"ʹؚ·Εͳ͍఻هͷ2"͕ղ͚Δ͔ʁɹ ʮҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͰ͖ΔΑ͏ʹͳΔʯͷ͸Ͳ͏͍͏ͱ͖͔ʁ w ࠞ߹ͯ͠܇࿅͢Δͱ͖͸੒ޭ w ఻هࣄલ܇࿅ 2"

    fi OFUVOJOHͷೋஈ֊܇࿅Ͱ w ֤ਓ෺ͷ఻ه͕छྨͷදݱ͚ͩͷͱ͖͸ࣦഊ w ఻ه͕ଟ༷ʹݴ͍׵͑ΒΕͯग़ݱ͢Δͱ͖͸੒ޭ ผදݱςϯϓϨ΍--.Ͱੜ੒ ಛʹ༗ޮ จॱংมߋ ୅໊ࢺͷϑϧωʔϜԽ ຋༁ w ໊લτʔΫϯͷޙͷ࠷ऴ૚ʹؔ࿈஌͕ࣝൃ࿐͢ΔΑ͏ʹͳΔ w Ұఆ਺ͷਓ෺ʹ͍ͭͯݴ͍׵͑ 2"͕͋Ε͹શମʹ΋൚Խ͍ͯ͘͠ 😊 😊 😨

  11. ͓·͚ w #&35Ͱ͸ w ॱংϥϯμϜ͸͠ͳͯ͘΋ͦͦ͜͜Ͱ͖Δ͚Ͳ ΄ͱΜͲͷଐੑ͸ݴ͍׵͑ͯ͠΋ͦͦ͜͜  ࢭ·Γ w ໊લଐੑͷ݁ͼ͖ͭΑΓ1SJODFUPO㲗/+ͳͲͷ݁ͼ͖ͭʹཔͬͯ͠·͏

    w ஀ੜ೔ τʔΫϯ͕ޓ͍ʹಠཱ ΍ઐ߈ τʔΫϯ཰͕ߴ͍ ͸ਖ਼ղ཰͕ߴ͍ w Ϟσϧৄࡉ w (15TNBMM 3PUBSZ .MBZFST EJNT UPLFOJ[FS͸(15  --B.Bੜ੒σʔλ࣮ݧͰ͸ͪΐͬͱେ͖͘.  OFHBUJWFSFTVMUͷݕূ༻ʹ͸.Ͱ΋࣮ݧ DGj # )