Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Physics of Language Models: Part 3.1, Knowledge...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Sosuke Kobayashi Sosuke Kobayashi
August 17, 2024
Research
1.3k
1

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Avatar for Sosuke Kobayashi

Sosuke Kobayashi

August 17, 2024

Other Decks in Research

See All in Research
2026 東京科学大 情報通信系 研究室紹介 (大岡山)
Avatar for Tokyo Tech ICT Dept. icttitech
0
3.9k
YOLO26_ Key Architectural Enhancements and Performance Benchmarking for Real-Time Object Detection
Avatar for SatAI.challenge satai
3
830
Ghost in the 7‑Zip: The Shadow of Residential Proxies Creeping into Your Life
Avatar for NTT docomo Business nttcom
0
1.3k
Fukui Shibiten 39 - AI Art
Avatar for IWABUCHI Yu(u)ki butchi butchi
0
140
Dual Quadric表現を用いた動的物体追跡とRGB-D・IMU制約の密結合によるオドメトリ推定
Avatar for Toyozo Shimada nanoshimarobot
0
420
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
Avatar for SatAI.challenge satai
2
330
SoftMatcha 2: 1兆語規模コーパスの超高速かつ柔らかい検索
Avatar for Masataka YONEDA e869120_sub
7
3.5k
通時的な類似度行列に基づく単語の意味変化の分析
Avatar for hajime kiyama rudorudo11
0
320
研究室単位での自律的 IPv6接続性確立に向けたAS共同運用モデルの提案と実証
Avatar for Hiroki (REO) Kashiwazaki reokashiwa
0
110
Claude Code × autoresearch 実践
Avatar for 数理の弾丸 mathbullet
0
170
PGDM: Physically Guided Diffusion Model for L Downscaling
Avatar for SatAI.challenge satai
2
300
Data Visualization Tools in the Age of AI
Avatar for Fritz Lekschas flekschas
0
170

Featured

See All Featured
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
1
270
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
247
13k
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
57
14k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
430
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.3k
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
370
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
180
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
230
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.8k

Transcript

  1. ୈճ࠷ઌ୺/-1ษڧձ 1IZTJDTPG-BOHVBHF.PEFMT1BSU  ,OPXMFEHF4UPSBHFBOE&YUSBDUJPO ;FZVBO"MMFO;IV :VBO[IJ-J *$.- ಡΉਓখྛᰜհ 1'/౦๺େ IUUQTBSYJWPSHQEG

    ຊεϥΠυͰ࢖༻͞Ε͍ͯΔਤ΍σʔλ͸ɺಛʹઆ໌͕ͳ͍ݶΓ্ه࿦จ͔ΒҾ༻͍ͯ͠·͢ɻ

  2. 1IZTJDTPG-BOHVBHF.PEFMT w ;FZVBO"MMFO;IV͕೥͔Β࢝Ίͨ --.ղੳͷҰ࿈ͷݚڀ w *$.-ͷ5VUPSJBMͱͯ͠΋ൃද IUUQTQIZTJDTBMMFO[IVDPNIPNF w ʮಈ෺ߦಈֶͷΑ͏ͳݚڀ͚ͩͰͳ͘ ɹීวతͳ๏ଇΛݟ͚ͭΑ͏ʂʯ

    ࠓճ͸͜Ε ⾭

  3. Ҿ͖ग़ͤΔ஌ࣝΛݴޠϞσϧʹ͍֮͑ͤͨ͞ w "CSBIBN-JODPMOXBTCPSOPO'FCSVBSZ ʜ ͷΑ͏ͳ఻هςΩετ͚ͩΛಡΜͰ w "CSBIBN-JODPMOXBTCPSOPO<>Λ༧ଌͰ͖Δʁ w 8IBUJT-JODPMO`TCJSUIEBZ <>Λ༧ଌԠ౴Ͱ͖Δʁ

    w ௥ՃͰ   Α͏ͳ͋Δਓ෺ͷ఻ه ࣭໰Ԡ౴΋ҰॹʹಡΜͰ͓͚͹Ͳ͏ʁ ͦͷճ౴ೳྗ͸ଞͷਓ෺ʹ΋൚Խ͢Δʁ w σʔλͷଟ༷ੑ͸ʁɹ܇࿅ͷॱং͸ʁ DGj ݴޠϞσϧ͕͋Δֶश৚݅ͰσʔληοτΛlهԱzͰ͖Δͱ͖ ؙ҉هͱҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͷڥք͸Ͳ͜ʹ͋Δͷ͔ʁ 4IPIFJ0UBOJXBTCPSOPO+VMZ ʜ΍ 8IBUJT4IPIFJ0UBOJ`TCJSUIEBZ +VMZ ʜ

  4. جຊ࣮ݧઃఆ w Սۭͷਓ෼ͷଐੑσʔλ͔Βਓ޻σʔληοτΛ࡞੒ w ఻هςΩετ͸\୯Ұෳ਺ςϯϓϨʔτ طଘ--. ʜ^Ͱ࡞੒ w ࣭໰Ԡ౴ςΩετ͸୯७ͳςϯϓϨʔτͰ࡞੒ w

    ͦͷσʔληοτͰ.ఔ౓ͷ-.ΛεΫϥον͔Β܇࿅ <ੜ੒ςΩετ> "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IF TQFOUIFSFBSMZZFBSTJO1SJODFUPO /+4IFSFDFJWFE NFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU .BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ4IF DPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO $PNNVOJDBUJPOT4IFIBEBQSPGFTTJPOBMSPMFBU.FUB 1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" DGj " <Սۭͷਓ෺σʔλ> "OZB#SJBS'PSHFS ஀ੜ೔0DUPCFS  ग़਎஍1SJODFUPO /+  େֶ.BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ  ઐ߈$PNNVOJDBUJPOT ձࣾ.FUB1MBUGPSNT  ձࣾॴࡏ஍.FOMP1BSL $" ࣮ࡍͷ࿦จதͷྫ

  5. ࠞ߹܇࿅ w /఻ه /࣭໰Ԡ౴ɹͰ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ਖ਼ղ ࣭໰Ԡ౴ΛؚΉਓ෺ ੨ ͸࣭໰Ԡ౴ ࣮ઢ ͚ͩͰͳ͘఻هͷֶश ഁઢ ΋଎͍ɻ ͦͷ͋ͱʹɺग़஗Εͯଞͷਓ෺ ੺ ͷֶश͕ਐΜͰ͍͘ɻ ஶऀʮࢼݧษڧΈ͍ͨɻਓؒͷ஌ࣝ֫ಘͷࣗવͳਐߦΛ൓ө͍ͯ͠ͳ͍ʯ "OZBͳͲ %FNJBOͳͲ DGj $ 😊

  6. ࣄલ܇࿅ 'JOFUVOJOH w /఻هɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ʙ͚ͩਖ਼ղʜ ҉هਖ਼ղ ҉هਖ਼ղ ࡉ͔͍ποίϛʮ fi OFUVOJOH͗͢͠Ͱ/఻هΛ๨ΕͨͷͰ͸ʁʯ ҰԠ-P3"ͱ͔Ͱ༏͘͠ߋ৽ͯ͠΋ͪΐͬͱվળ͢Δఔ౓ͩͬͨ DGj $ % 😨

  7. ࣄલ܇࿅ ݴ͍׵͑  'JOFUVOJOH w /఻ه ݴ͍׵͑ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ʙਖ਼ղʂ ݴ͍׵͑͢Ε͹͢Δ΄Ͳվળ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ DGj $ % 😊 ݴ͍׵͑φγ ϑϧωʔϜԽ ॱ൪ϥϯμϜY ϑϧ ॱ൪ϥϯμϜY ผςϯϓϨY  ϑϧ  ॱ൪ϥϯμϜ  ϑϧ ॱ൪ϥϯμϜ

  8. ݴ͍׵͑͸Ұ෦ͷਓ෺ͷΈͰ΋༗ޮ w /఻ه ݴ͍׵͑φγ  .఻ه ݴ͍׵͑ΞϦ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ w .࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ/࣭໰Ԡ౴ɹͰධՁ #FDLZ&MJ[BCFUI#MBDLCFMMDFMFCSBUFTUIFJSMJGFKPVSOFZFWFSZZFBSPO +BOVBSZ 4IFʜJO#BMUJNPSF .BSZMBOEʜJO$JODJOOBUJ 0IJP ʜ ݴ͍׵͑φγ  &NJMF#FOKBNJO&MNBO`TCJSUIJTDFMFCSBUFEBOOVBMMZPO+VMZ  )FDBMMTIJTCJSUIJO1BMNEBMF $"ʜJO/FX#SVOTXJDL /+ ʜ ݴ͍׵͑φγ ਖ਼ղʂ DGj ( 😊

  9. ஌͕ࣝͲ͜ʹ֨ೲ͞Ε͍ͯΔ͔ w ֤Ґஔͷ5SBOTGPSNFS࠷ऴ૚͔Βશଐੑ஋Λ༧ଌ͢ΔQSPCJOHͰςετ ৽ͨʹ௥Ճઢܗ૚Λֶशɻ&NCFEEJOH૚͚ͩ௿ϥϯΫߋ৽ ଞ͸ϑϦʔζ  w ݴ͍׵͑ˠ໊લҐஔ͔Βશଐੑ༧ଌՄೳɻφγˠϑϨʔζ΍ଐੑͷ૊Ͱ҉ه "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZZFBSTJO1SJODFUPO

    /+ 4IFSFDFJWFENFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU.BTTBDIVTFUUT*OTUJUVUFPG 5FDIOPMPHZ4IFDPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO$PNNVOJDBUJPOT4IFIBEB QSPGFTTJPOBMSPMFBU.FUB1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" FH ݴ͍׵͑ΞϦ ݴ͍׵͑φγ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠ௚ޙͷଐੑ ʜCPSOPOˠ0DUPCFS ʜZFBSTJOˠ1SJODFUPO ̋ ̋ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠҎ্ઌͷଐੑ ʜCPSOPOˠ1SJODFUPO ʜCPSOPOˠ.BTTBDIVTFUUT ʜZFBSTJOˠ.BTTBDIVTFUUT ̋ ✕ -BTUOBNFͷҐஔ ɹ༧ଌˠ࠷ॳͷଐੑ ʜ'PSHFSˠ0DUPSCFS ̋ ˚ -BTUOBNFͷҐஔ ɹ༧ଌˠ࢒Γͷଐੑ ʜ'PSHFSˠ1SJODFUPO ʜ'PSHFSˠ.BTTBDIVTFUUT ̋ ✕ ໼ҹ͕ଟ͘ͳΓ͗͢ΔͷͰҰ෦লུ DGj & ' 😊 😊 😊 😊 😊 😨 😨 😨 😊

  10. ·ͱΊ w ఻ههड़ 2"Ͱ܇࿅ͨ͠ͱ͖܇࿅2"ʹؚ·Εͳ͍఻هͷ2"͕ղ͚Δ͔ʁɹ ʮҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͰ͖ΔΑ͏ʹͳΔʯͷ͸Ͳ͏͍͏ͱ͖͔ʁ w ࠞ߹ͯ͠܇࿅͢Δͱ͖͸੒ޭ w ఻هࣄલ܇࿅ 2"

    fi OFUVOJOHͷೋஈ֊܇࿅Ͱ w ֤ਓ෺ͷ఻ه͕छྨͷදݱ͚ͩͷͱ͖͸ࣦഊ w ఻ه͕ଟ༷ʹݴ͍׵͑ΒΕͯग़ݱ͢Δͱ͖͸੒ޭ ผදݱςϯϓϨ΍--.Ͱੜ੒ ಛʹ༗ޮ จॱংมߋ ୅໊ࢺͷϑϧωʔϜԽ ຋༁ w ໊લτʔΫϯͷޙͷ࠷ऴ૚ʹؔ࿈஌͕ࣝൃ࿐͢ΔΑ͏ʹͳΔ w Ұఆ਺ͷਓ෺ʹ͍ͭͯݴ͍׵͑ 2"͕͋Ε͹શମʹ΋൚Խ͍ͯ͘͠ 😊 😊 😨

  11. ͓·͚ w #&35Ͱ͸ w ॱংϥϯμϜ͸͠ͳͯ͘΋ͦͦ͜͜Ͱ͖Δ͚Ͳ ΄ͱΜͲͷଐੑ͸ݴ͍׵͑ͯ͠΋ͦͦ͜͜  ࢭ·Γ w ໊લଐੑͷ݁ͼ͖ͭΑΓ1SJODFUPO㲗/+ͳͲͷ݁ͼ͖ͭʹཔͬͯ͠·͏

    w ஀ੜ೔ τʔΫϯ͕ޓ͍ʹಠཱ ΍ઐ߈ τʔΫϯ཰͕ߴ͍ ͸ਖ਼ղ཰͕ߴ͍ w Ϟσϧৄࡉ w (15TNBMM 3PUBSZ .MBZFST EJNT UPLFOJ[FS͸(15  --B.Bੜ੒σʔλ࣮ݧͰ͸ͪΐͬͱେ͖͘.  OFHBUJWFSFTVMUͷݕূ༻ʹ͸.Ͱ΋࣮ݧ DGj # )