Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PAKDD2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT (Parts 3 and 4)

Kyosuke Nishida
May 25, 2023
Research
7
1.7k

PAKDD2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT (Parts 3 and 4)

Jun Suzuki, Kyosuke Nishida, Naoaki Okazaki: "PAKDD 2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT," The 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2023) , May 25, 2023.

Parts 1 and 2: https://www.fai.cds.tohoku.ac.jp/research/activities/
Parts 3 and 4: this deck.
Part 5: https://speakerdeck.com/chokkan/efforts-for-responsible-llms-pakdd-2023-tutorial-2

Kyosuke Nishida

May 25, 2023
Tweet

More Decks by Kyosuke Nishida

See All by Kyosuke Nishida
大規模言語モデル入門 / LLM introduction (SES2023)
kyoun
47
12k
論文紹介 / Llama 2: Open Foundation and Fine-Tuned Chat Models
kyoun
5
6.2k
Collaborative AI: 視覚・言語・行動の融合
kyoun
21
7.4k
NLPとVision-and-Languageの基礎・最新動向 (1) / DEIM Tutorial Part 1: NLP
kyoun
24
9.1k
NLPとVision-and-Languageの基礎・最新動向 (2) / DEIM Tutorial Part 2 Vision-and-Language
kyoun
20
10k
論文紹介 / Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
kyoun
1
650
自然言語処理とVision-and-Language / A Tutorial on NLP & Vision-and-Language
kyoun
22
12k
論文紹介 / PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
kyoun
1
680
言語と視覚に基づく質問応答の最新動向 / Recent Trends in Vision-and-Language Studies for QA
kyoun
13
3.9k

Other Decks in Research

See All in Research
ゼロからわかるリザバーコンピューティング
kurotaky
1
290
音声処理ツールキットESPnetの現在と未来
kanbayashi1125
2
540
Generative Spoken Dialogue Language Modeling [対話論文読み会@電通大]
yuta0306
1
130
「EBPMエコシステム」の可能性
daimoriwaki
0
200
眠眠ガチャ：ガチャを活用した睡眠意欲向上アプリの開発 / EC71inui
yumulab
0
150
自己教師あり学習による事前学習（CVIMチュートリアル）
naok615
2
1.4k
2024-01-23-az
sofievl
1
750
F0に基づいて伸縮された画像文字からの音声合成 [ASJ2024春]
nehi0615
0
120
リサーチに組織を巻き込むための「準備8割」の話
terasho
0
470
MLtraq: Track your AI experiments at hyperspeed
micheda
1
110
サウナでのプロジェクションマッピングの可能性の検討 / EC71koizumi
yumulab
0
170
Source Code Diff Revolution (JetBrains Open Reading Club)
tsantalis
0
260

Featured

See All Featured
Designing Experiences People Love
moore
136
23k
Visualization
eitanlees
136
14k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
121
39k
Build The Right Thing And Hit Your Dates
maggiecrowley
24
2k
How to name files
jennybc
65
93k
Fantastic passwords and where to find them - at NoRuKo
philnash
37
2.5k
Learning to Love Humans: Emotional Interface Design
aarron
267
39k
Building an army of robots
kneath
300
41k
Git: the NoSQL Database
bkeepers
PRO
422
63k
Happy Clients
brianwarren
92
6.4k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.6k
Into the Great Unknown - MozCon
thekraken
10
990

Transcript

  1. 1BSUTBOE 5FDIOPMPHJFTBOE3FDFOU"DIJFWFNFOUT JO$IBU(15MJLF--.T /55)VNBO*OGPSNBUJDT-BCPSBUPSJFT ,ZPTVLF /JTIJEB .BZ  1",%%5VUPSJBM "(FOUMF*OUSPEVDUJPOUP5FDIOPMPHJFT#FIJOE

    -BOHVBHF.PEFMTBOE3FDFOU"DIJFWFNFOUJO$IBU(15

  2.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU<JOUSPEVDUPSZ>-BOHVBHFNPEFMT -.T l 1BSU-BSHFMBOHVBHFNPEFMT --.T

    u 4QFBLFS 1SPGFTTPS+VO4V[VLJ 5PIPLV6OJWFSTJUZ u 4MJEFT IUUQTXXXGBJDETUPIPLVBDKQSFTFBSDIBDUJWJUJFT l 1BSU5FDIOPMPHJFT6OEFSMZJOH$IBU(15MJLF--.T l 1BSU3FDFOUBDIJFWFNFOUTJO$IBU(15MJLF--.T l 1BSU&GGPSUTGPS3FTQPOTJCMF--.T u 4QFBLFS1SPGFTTPS/BPBLJ 0LB[BLJ 5PLZP*OTUJUVUFPG5FDIOPMPHZ u 4MJEFT IUUQTTQFBLFSEFDLDPNDIPLLBOFGGPSUTGPSSFTQPOTJCMFMMNT QBLEEUVUPSJBM

  3.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU5FDIOPMPHJFT6OEFSMZJOH$IBU(15MJLF--.T

  4.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 0QFO"* NPEFMTSFMFBTFEGPMMPXJOH(15 u $PEFY5SBJOJOHGSPNDPEFBOEUFYU

    u *OTUSVDU(15 BOE$IBU(15*OTUSVDUJPOUVOJOHBOE3-)' u (15BOE$IBU(15QMVHJOT.VMUJNPEBMBOEFYUFSOBMLOPXMFEHF JOUFHSBUJPO l 0QFOTPVSDFNPEFMT u --B.""OPQFO--.USBJOFEPOPWFSPOFUSJMMJPOUPLFOT u "MQBDBBOE7JDVOB'JOFUVOFENPEFMTCBTFEPO--B." u .15"OFXDPNNFSDJBMMZVTBCMFNPEFMDBQBCMFPGIBOEMJOHMPOH DPOUFYU u --B7" BOE.JOJ(15NVMUJNPEBMNPEFMT

  5.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV &WPMVUJPOGSPN(15UP(15 l #FUXFFO(15BOE(15 UIFSFBSFUISFFNPEFMTSFGFSSFEUPBT UIF(15GBNJMZ l

    (15 .BZ QBQFS ˣ 5SBJOJOH-.TXJUITPVSDFDPEF l $PEFY +VMZ QBQFS ˣ 5SBJOJOH-.TUPGPMMPXJOTUSVDUJPOTXJUIIVNBOGFFECBDL l *OTUSVDU(15 +BO  QBQFS ˣ 0QUJNJ[BUJPO GPSDIBU l $IBU(15 /PW  CMPH ˣ .VMUJNPEBMJOUFHSBUJPO l (15 .BS UFDIOJDBMSFQPSUCMPH

  6.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $PEFY<$IFO 0QFO"* +VMZ> l #VJMUPOUIFTBNFGPVOEBUJPOBT(15 l

    5PFOBCMFUIF--.UPHFOFSBUFDPEFJOSFTQPOTFUPVTFSJOTUSVDUJPOT  JUTUSBJOJOHEBUBTFUJODMVEFTCJMMJPOTPGMJOFTPGDPEF l *UTDPOUFYUJTFYQBOEFEUP UPLFOT l 5IFMBUFTU"1*WFSTJPO DPEFEBWJODJ XPSLTJOPWFSUFOMBOHVBHFT UIFQSPNQU NFUIPEEFGJOJUJPOTBOE *OQVUPVUQVUFYBNQMFT QSPWJEFEUPUIFNPEFM NPEFMHFOFSBUFEDPNQMFUJPO IUUQTBSYJWPSHBCT

  7.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (JU)VC$PQJMPU<+VOF> l $PEFYQPXFSTBOl"*QBJSQSPHSBNNFSzPO(JUIVC l 4JODFJUTSFMFBTF JUIBTUSBOTGPSNFEEFWFMPQFSQSPEVDUJWJUZGPS

    NPSFUIBOPOFNJMMJPOQFPQMF IFMQJOHEFWFMPQFSTDPEFVQUP GBTUFS<> <>IUUQTHJUIVCCMPHHJUIVCDPQJMPUGPSCVTJOFTTJTOPXBWBJMBCMF

  8.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 5PLFOJ[FSPG$PEFY l .PEJGJFEGSPN(15 XJUIUIFBEEJUJPOPGUPDPOTFDVUJWF XIJUFTQBDFTUPUIFWPDBCVMBSZ 5IFTFFYBNQMFTXFSFHFOFSBUFEVTJOHIUUQTQMBUGPSNPQFOBJDPNUPLFOJ[FS

    $PEFY`TUPLFOJ[BUJPO SL@CBTF (15 QL@CBTF DPOTFDVUJWFXIJUFTQBDFT BSFDPOWFSUFEJOUPBTJOHMFUPLFO

  9.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV *OTUSVDU(15 <0VZBOH 0QFO"* .BSDI> l 4JHOJGJDBOUMZPVUQFSGPSNT(15JOGPMMPXJOHVTFSJOTUSVDUJPOT

     SFTVMUJOHJOPVUQVUTUIBUBSFNPSFUSVUIGVMBOEMFTTUPYJD IUUQTPQFOBJDPNSFTFBSDIJOTUSVDUJPOGPMMPXJOHTBNQMF VTFSJOTUSVDUJPO 5IJTBOTXFSlBMJHOTzXJUIUIFJOTUSVDUJPO (15GBJMTUPSFTQPOEUPUIFVTFSJOTUSVDUJPOT

  10.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV IUUQTBSYJWPSHBCT *OTUSVDUJPOUVOJOH 3FJOGPSDFNFOU-FBSOJOHGSPN)VNBO'FFECBDL 3-)' -FBSOJOHTUFQTPG*OTUSVDU(15

  11.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV *OTUSVDUJPOUVOJOH<8FJ (PPHMF 4FQ><$IVOH (PPHMF 0DU> l

    'JOFUVOJOHMBOHVBHFNPEFMTPOBDPMMFDUJPOPGUBTLTEFTDSJCFEWJB JOTUSVDUJPOT JNQSPWFT[FSPTIPUQFSGPSNBODFPOVOTFFOUBTLT l 0QFO"* VTFEBCPVULUSBJOJOHQSPNQUT GSPNUIF0QFO"* "1* BOEMBCFMFSXSJUUFO GPSJOTUSVDUJPOUVOJOH u 5IJTTVQFSWJTFEGJOFUVOFENPEFMJTDBMMFEUIF4'5 NPEFM IUUQTBSYJWPSHBCT

  12.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 3FXBSENPEFMJOH l -BCFMFSTXFSFQSFTFOUFEXJUICFUXFFOBOESFTQPOTFTUPFBDI QSPNQUBOEBTLFEUPSBOLUIFN l 5IFSFXBSENPEFM

    TJ[FPG# XBTUSBJOFEXJUIBCPVULUSBJOJOH QSPNQUT GSPNUIF0QFO"* "1*BOEMBCFMFSXSJUUFO  ,SFTQPOTFT ! " DPNQBSJTPOT SBOLJOH

  13.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 3FJOGPSDFNFOU-FBSOJOHGSPN)VNBO'FFECBDL 3-)' l 0SJHJOBMMZEFWFMPQFEGPSUSBJOJOHTJNQMFSPCPUTBOE"UBSJHBNFT <$ISJTUJBOP *CBS[

    > UIFOBQQMJFEUPUIFGJOFUVOJOH PG-.TUPTVNNBSJ[FUFYU<4UJFOOPO > l 0QFO"* IBTBMTPBQQMJFE3-)'UPGJOFUVOFUIF4'5NPEFM VTJOH BCPVULUSBJOJOHQSPNQUT GSPNUIF0QFO"* "1*  l 3-0CKFDUJWF110QUY BWBSJBOUPG110<4DIVMNBO > ",VMMCBDL-JCMFS QFOBMUZGSPNUIF4'5NPEFM UP NJUJHBUFPWFSPQUJNJ[BUJPOPGUIFSFXBSENPEFM 1SFUSBJOJOHNJYUPGJYUIFQFSGPSNBODF SFHSFTTJPOTPOQVCMJD/-1EBUBTFUT

  14.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15<0QFO"* /PW  > l 0QUJNJ[FEGPSDIBUVTJOHUIFTBNFNFUIPETBT*OTUSVDU(15<>

    u /PGVSUIFSEFUBJMTBCPVUUIFBSDIJUFDUVSF NPEFMTJ[F EBUBTFU DPOTUSVDUJPO PSUSBJOJOHNFUIPETIBWFCFFOSFQPSUFE <>IUUQTPQFOBJDPNCMPHDIBUHQU 'PSJOTUSVDUJPOUVOJOH  )VNBO"*USBJOFSTQSPWJEFEDPOWFSTBUJPOT JOXIJDIUIFZQMBZFECPUITJEFT<>

  15.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 5PLFOJ[FSPG$IBU(15 l /FXMZDSFBUFE DBMMFElDML@CBTFz BOEUIFWPDBCVMBSZTJ[FIBT CFFOFYQBOEFEVQUPL

    *OTUSVDU(15 SL@CBTF `T UPLFOJ[BUJPOPGl͜Μʹͪ͸z lIFMMPzJO+BQBOFTF 5PLFOTIBWFCFFOJODSFBTFEGPS QSPHSBNJOHMBOHVBHFTBOEOPO&OHMJTIMBOHVBHFT "QBSUPG lDML@CBTFz WPDBCVMBSZ

  16.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15<0QFO"* .BSDI > l $BOBDDFQUJNBHFBOEUFYUJOQVUTBOEQSPEVDFUFYUPVUQVUT (15`TSFTQPOTF

    IUUQTBSYJWPSHBCT

  17.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15<0QFO"* .BSDI > l "5SBOTGPSNFSTUZMFNPEFMQSFUSBJOFEUPQSFEJDUUIFOFYUUPLFOJO BEPDVNFOU

    XJUIBDPOUFYUPGLUPLFOT BOEGJOFUVOFEVTJOH3-)' u /PGVSUIFSEFUBJMTBCPVUUIFBSDIJUFDUVSF NPEFMTJ[F IBSEXBSF USBJOJOH DPNQVUBUJPO EBUBTFUDPOTUSVDUJPO PSUSBJOJOHNFUIPETIBWFCFFO SFQPSUFE *UJTVODMFBSIPXUIFJNBHFTBOEUFYUGPSN UIFJOQVUTFRVFODFBOEBSFGFEUP(15 *UJTVODMFBSIPXJUQSPDFTTFTMPOHTFRVFODFT PGVQUPLUPLFOTBUIJHITQFFE

  18.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 1MVHJOTBSFFYUFOTJPOT

    JODMVEJOHUIJSEQBSUZBQQMJDBUJPOT UIBU BMMPXNPEFMTUPJOUFSBDUXJUIFYUFSOBM"1*T IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT

  19.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IFl8FC#SPXTFSzQMVHJOBMMPXTBDDFTTUPVQUPEBUFJOGPSNBUJPO

    IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 18 IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 5IFNPEFMJTBCMFUPQSPWJEFVQUPEBUF JOGPSNBUJPO FWFOUIPVHIJUJTPOMZUSBJOFE POEBUBVQUP0DUPCFS

  20.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IFl8FC#SPXTFSz

    QMVHJOBMMPXTBDDFTTUPVQUPEBUFJOGPSNBUJPO IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 19 IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT

  21.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IF8PMGSBNQMVHJOBMMPXTBDDFTTUPQPXFSGVMDPNQVUBUJPO

     BDDVSBUFNBUI DVSBUFELOPXMFEHF BOEWJTVBMJ[BUJPO 20

  22.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUTPGUIJTQBSU l 0QFO"*bT NPEFMT u $PEFYUSBJOJOHGSPNDPEFBOEUFYU

    u *OTUSVDU(15 BOE$IBU(15JOTUSVDUJPOUVOJOHBOE3-)' u (15NVMUJNPEBMJOUFHSBUJPO l 0QFOTPVSDFNPEFMT u --B."BOPQFO--.USBJOFEPOPWFSPOFUSJMMJPOUPLFOT u "MQBDBBOE7JDVOB'JOFUVOFENPEFMTCBTFEPO--B." u .15BDPNNFSDJBMMZVTBCMFNPEFMDBQBCMFPGIBOEMJOHMPOH DPOUFYU u --B7" BOE.JOJ(15NVMUJNPEBMNPEFMT

  23.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV --B." <5PVWSPO .FUB 'FC  >

    l "OPQFOTPVSDF--.USBJOFEXJUIQVCMJDMZBWBJMBCMFEBUBTFUT l 5IFQSFUSBJOJOHEBUBIBTCFFOSFDFOUMZSFQSPEVDFEBT l3FE1BKBNB%BUBz<5PHFUIFS$PNQVUFS>BOEJTBWBJMBCMFGPSBOZPOF IUUQTBSYJWPSHBCT USJMMJPO UPLFOT

  24.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B."#JTDPNQBSBCMFUP(15 # PONPTUCFODINBSLT l 5IFOVNCFSPGUSBJOJOHUPLFOTJO--B."

    5 XBTTJHOJGJDBOUMZ HSFBUFSUIBOUIBUPG(15 #  u DG$IJODIJMMB<)PGGNBO %FFQNJOE > ;FSPTIPUQFSGPSNBODFPODPNNPOTFOTFSFBTPOJOH IUUQTBSYJWPSHBCT

  25.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "SDIJUFDUVSFBOEUSBJOJOHNFUIPE l --B." MFWFSBHFTWBSJPVTJNQSPWFNFOUTUIBUXFSFQSPQPTFE BGUFS UIFPSJHJOBM5SBOTGPSNFSQVCMJDBUJPO

    u (15`TQSFOPSNBMJ[BUJPO u 1B-.`T 4XJ(-6 BDUJWBUJPOGVODUJPO<4IB[FFS> u (15/FP`T3PUBSZ&NCFEEJOHT<4V > l 5IFUSBJOJOHNFUIPEJTUIFTBNFBT(15 JF OFYUUPLFO QSFEJDUJPO  l --B."JUTFMGJTOPUUSBJOFEXJUIJOTUSVDUJPOUVOJOHBOE3-)'

  26.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "MQBDB<4UBOGPSE6 .BSDI > l "NPEFMGJOFUVOFEGSPNUIF--B." #NPEFMPO,JOTUSVDUJPO

    GPMMPXJOHEFNPOTUSBUJPOT l "TFMGJOTUSVDUUFDIOJRVF<8BOH >XBTVTFEUPBVUPNBUJDBMMZ HFOFSBUFUIFEFNPOTUSBUJPOGSPNPOMZIVNBOXSJUUFOTFFEUBTLT u 5IFUFYUEBWJODJ *OTUSVDU(15 NPEFMXBTVTFEJOUIJTTUVEZ IUUQTDSGNTUBOGPSEFEV BMQBDBIUNM

  27.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "MQBDB-P3" <.BSDI > l -P3" <)V

    >GSFF[FTUIFQSFUSBJOFEXFJHIUTBOEJOKFDUT USBJOBCMFBEBQUFST SBOLEFDPNQPTJUJPONBUSJDFT JOUPFBDIMBZFS l 6TJOH--B." -P3" BOEUIF"MQBDBEBUBTFU BOZPOFDBODIFBQMZ QSPEVDF$IBU(15MJLFNPEFMTXJUIJOIPVSTPOBTJOHMF(16 IUUQTBSYJWPSHBCT IUUQTHJUIVCDPNUMPFOBMQBDBMPSB 7BSJPVTBEBQUFSTIBWFCFFO SFMFBTFEUPNBLF"MQBDBDPNQBUJCMF XJUIOPO&OHMJTIMBOHVBHFT

  28.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JDVOB<6$#FSLFMFZ $.6 4UBOGPSE BOE6$4BO%JFHP.BSDI > l

    "NPEFMGJOFUVOFEGSPN--B." PO,VTFSTIBSFE DPOWFSTBUJPOTDPMMFDUFEGSPNUIF4IBSF(15 XFCTJUF l &WBMVBUJPOVTJOH(15BTBKVEHFTIPXTUIBU7JDVOB# BDIJFWFTNPSFUIBOUIFRVBMJUZPG$IBU(15 5XPXFFLTBQBSU 5XPXFFLTBQBSU IUUQTMNTZTPSHCMPHWJDVOB

  29.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JDVOB`T"VUPNBUJDFWBMVBUJPOVTJOH(15 l (15DBOQSPEVDFIJHIMZDPOTJTUFOUSBOLTBOEBEFUBJMFE BTTFTTNFOUXIFODPNQBSJOHDIBUCPUT`BOTXFST IUUQTMNTZTPSHCMPHWJDVOB

  30.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4UBCMF7JDVOB <4UBCJMJUZ"* "QSJM > l "MQBDBBOE7JDVOBVTFEPOMZJOTUSVDUJPOUVOJOH

    l 4UBCMF7JDVOB JTUIFGJSTUMBSHFTDBMFPQFOTPVSDFDIBUCPUUSBJOFE WJB3-)'VTJOHUIFGPMMPXJOHPQFOEBUBTFUT u 0QFO"TTJTUBOU $POWFSTBUJPOT 0"445 <"QSJM > u 4UBOGPSE)VNBO1SFGFSFODFT 4)1 <'FCSVBSZ > u "OUISPQJD))3-)'<#BJ "QSJM >

  31.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .15<MosaicML, May 5, 2023] l "OPQFOTPVSDF

    DPNNFSDJBMMZVTBCMF--. USBJOFEGSPNTDSBUDI PO5UPLFOTPGUFYUBOEDPEF l .15#NBUDIFTUIFRVBMJUZPG--B."#POTUBOEBSEUBTLT IUUQTXXXNPTBJDNM DPNCMPHNQUC

  32.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .15#4UPSZ8SJUFSL <MosaicML, May 5, 2023] l

    #VJMUCZGJOFUVOJOH.15#XJUIBDPOUFYUMFOHUIPGLUPLFOTUP SFBEBOEXSJUFTUPSJFTXJUITVQFSMPOHDPOUFYUMFOHUIT u $BOSFBEUIFFOUJSFUFYUPGl3PNFPBOE+VMJFUz<> XIJDIIBTBCPVUL UPLFOTVTJOH(15`TUPLFOJ[FS DML@CBTF  l 6TFT"-J#J <1SFTT "VHVTU>JOTUFBEPGQPTJUJPOBMFNCFEEJOHT UPIBOEMFTVDIMPOHDPOUFYUT <>IUUQTXXXHVUFOCFSHPSHDBDIFFQVCQHUYU

  33.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "-J#J "UUFOUJPOXJUI-JOFBS#JBTFT l CJBTFTRVFSZLFZBUUFOUJPOTDPSFTXJUIBIFBETQFDJGJDQFOBMUZUIBU JTQSPQPSUJPOBMUPUIFJSEJTUBODF https://arxiv.org/abs/2108.12409

    XIFSFTDBMBSNJTBIFBETQFDJGJDTMPQFGJYFECFGPSFUSBJOJOH <1SFTT 68BTIJOHUPO'"*3 "VHVTU > "MJ#JFOBCMFTUIF53"*/ 4)035 5&45-0/(DPODFQU  XIFSFUIFNPEFMUSBJOFEPOUIF DPOUFYUMFOHUIPGQFSGPSNT XFMMGPSLJOQVUUPLFOTBU JOGFSFODFUJNF UIBOLTUP"MJCJ

  34.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l 5IFSFBSFTFWFSBMPQFOTPVSDFTUVEJFTGPSNVMUJNPEBMNPEFMT l 'PSFYBNQMF --B7" BMMPXTGPSWJTVBMMZHSPVOEFEDIBUTMJLF(15

    IUUQTBSYJWPSHBCT --B7" <-JV 68JTDPOTJOr.BEJTPO "QSJM >

  35.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B7" HFOFSBUFTWJTVBMJOTUSVDUJPOUVOJOHEBUBCZQBTTJOH$0$0 DBQUJPOTBOECPVOEJOHCPYFTUPUIFMBOHVBHFPOMZ(15 l 5IFSFTVMUJOHEBUBTFUJTVTFEUPGJOFUVOFNVMUJNPEBMNPEFMT

    6TJOH-BOHVBHFPOMZ(15 7JTVBM *OTUSVDUJPO5VOJOH EBUBTFU IUUQTBSYJWPSHBCT --B7"`T 7JTVBM*OTUSVDUJPO5VOJOH

  36.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B7" DPOTJTUTPG--B." BOEUIFWJTVBMFODPEFSPG$-*1 l 'PSBOJOQVUJNBHF

    UIFQSFUSBJOFE$-*1WJTVBMFODPEFS 7J5-  DBOPVUQVUHSJEGFBUVSFTBTWJTVBMUPLFOT $-*1`THSJEGFBUVSFT <3BEGPSE +BOVBSZ> 8IBUJTVOVTVBMBCPVU UIJTJNBHF  5IFVOVTVBM BTQFDUPG UIJTJNBHF JTBNBO JSPOJOHʜ --B." IUUQTBSYJWPSHBCT .PEFM"SDIJUFDUVSF

  37.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .JOJ(15<;IV ,JOH"CEVMMBI6 "QSJM > IUUQTXXXZPVUVCFDPNXBUDI W@@UGUPYQ#"X

  38.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .JOJ(15<;IV ,JOH"CEVMMBI6 "QSJM > l $POTJTUTPG7JDVOBBOE#-*1`TWJTVBMFODPEFS<-J

    4BMFTGPSDF b> l 3FRVJSFTPOMZUSBJOJOHUIFMJOFBSQSPKFDUJPOMBZFSUPBMJHOUIFWJTVBM GFBUVSFTXJUIUIF7JDVOB BLIP-2 IUUQTBSYJWPSHBCT

  39.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4VNNBSZ l .PEFSO--.T TUBSUJOHGSPN$PEFY BSFUSBJOFEPONBTTJWF BNPVOUTPGUFYUBOEDPEF

    l *NQPSUBOUUFDIOJRVFTGPSBMJHOJOHNPEFMTUPGPMMPXJOTUSVDUJPOTBSF lJOTUSVDUJPOUVSOJOHzBOElSFJOGPSDFNFOUMFBSOJOHGSPNIVNBO GFFECBDL 3-)' z l 5IFUFDIOJDBMEFUBJMTPG(15IBWFOPUZFUCFFOSFQPSUFE l 4JODFUIFMBVODIPG--B." UIFSFIBTCFFOSFNBSLBCMFQSPHSFTTJO UIFEFWFMPQNFOUPGPQFOTPVSDFNPEFMTBOEEBUBTFUT

  40.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU3FDFOUBDIJFWFNFOUTJO$IBU(15MJLF--.T

  41.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 JTUIFGBTUFTUHSPXJOHBQQFWFS l $IBU(15 SFBDIFTNJMMJPOVTFSTUXPNPOUITBGUFSMBVODI l

    *UIBTBUUSBDUFEBHSFBUEFBMPGBUUFOUJPOEVFUPJUTIJHIQFSGPSNBODF IUUQTUXJUUFSDPN&DPOPNZ"QQTUBUVT

  42.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4QBSLTPG"(*<#VCCFDL .JDSPTPGU .BZ > l 0OFXFFLBGUFSUIFSFMFBTFPG(15

    .JDSPTPGUQVCMJTIFEBQBQFS FOUJUMFEl4QBSLTPG"SUJGJDJBM(FOFSBM*OUFMMJHFODF&BSMZ&YQFSJNFOUT XJUI(15z l 5IFBVUIPSTTUBUFEUIBU(15DPVMECFDPOTJEFSFEBOFBSMZ WFSTJPOPGBSUJGJDJBMJOUFMMJHFODF "(* CFDBVTFPGUIFCSFBEUIBOE EFQUIPG(15TDBQBCJMJUJFTJONBUIFNBUJDT DPEJOH WJTJPO  NFEJDJOF MBX QTZDIPMPHZ BOENPSF l 5IFBVUIPSTVTFEUIFUFSNl"(*zUPSFGFSUPTZTUFNTUIBU EFNPOTUSBUFCSPBEJOUFMMJHFODFDBQBCJMJUJFTBUPSBCPWFIVNBO MFWFM

  43.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  44.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15QBTTFTUIFCBSFYBN<,BU[ *MMJOPJT5FDI .BSDI > l (15

    [FSPTIPUFWBMVBUJPOT IBTQBTTFEUIFFOUJSF6OJGPSN#BS &YBNJOBUJPO 6#& JODMVEJOHOPUPOMZNVMUJQMFDIPJDF .#& CVU BMTP PQFOFOEFERVFTUJPOT .&&BOE.15  l (15BDIJFWFEBOPWFSBMMTDPSFPGQPJOUT TJHOJGJDBOUMZ TVSQBTTJOHUIFQBTTJOHTDPSFSBOHFPGUPQPJOUT IUUQTQBQFSTTTSODPNTPMQBQFSTDGN BCTUSBDU@JE

  45.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15 [FSPTIPUFWBM FYDFFETUIFQBTTJOHTDPSFPO64.-& 6OJUFE4UBUFT.FEJDBM-JDFOTJOH&YBNJOBUJPO CZPWFSQPJOUT

    IUUQTBSYJWPSHBCT QBTTUISFTIPMETBQQSPYJNBUFMZPG NVMUJQMFDIPJDFRVFTUJPOT (15QBTTFTNFEJDBMMJDFOTJOHFYBN</PSJ .JDSPTPGU .BZ>

  46.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15JTDPNQBSBCMFUPIVNBOQFSGPSNBODFPOBCFODINBSLPG VOTFFO QSPHSBNNJOHQSPCMFNTPO-FFU$PEF l (15BMTPQBTTFTNPDLUFDIOJDBMJOUFSWJFXTBUNBKPSUFDI

    DPNQBOJFTPO-FFU$PEF (15DPVMECFBTPGUXBSFFOHJOFFS<#VCCFL .JDSPTPGU .BSDI > QBTT!L NFBTVSFTXIFUIFSUIFNPEFM QSPEVDFTBDPSSFDUTPMVUJPOXJUIJOLUIBUUFNQUT (15TPMWFEBMMRVFTUJPOTJOPOMZ NJOVUFTPGUIFBMMPUUFE IPVST https://arxiv.org/abs/2303.12712

  47.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNT(15PONPTUBDBEFNJDBOEQSPGFTTJPOBM FYBNT JODMVEJOHNVMUJQMFDIPJDFBOEGSFFSFTQPOTFRVFTUJPOT l (15`TQFSGPSNBODFJTCFUUFSXIFOJUUBLFTWJTJPOBTJOQVUT

    IUUQTBSYJWPSHBCT 1FSGPSNBODFPOFYBNT<0QFO"* "QSJM >

  48.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNFE TUBUFPGUIFBSUNPEFMTXJUICFODINBSL TQFDJGJDUSBJOJOH PONBOZBDBEFNJDCFODINBSLT (15DBOCFBUGJOFUVOFE405"NPEFMT<0QFO"*

    "QSJM > https://openai.com/ research/gpt-4

  49.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNTUIF&OHMJTIMBOHVBHFQFSGPSNBODFPGFYJTUJOH --.T GPSUIFWBTUNBKPSJUZPGMBOHVBHFTUFTUFE JODMVEJOHMPX SFTPVSDFMBOHVBHFTTVDIBT-BUWJBO

    8FMTI BOE4XBIJMJ 1FSGPSNBODFPO.VMUJMJOHVBM..-6<0QFO"* "QSJM > .VMUJMJOHVBM ..-6CFODINBSL USBOTMBUFEXJUI "[VSF

  50.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15PVUQFSGPSNTDSPXEXPSLFST<1BO 6$# "QSJM > l (15BVUPNBUJPOPGTDFOBSJPMBCFMJOHJOBTPDJBMEFDJTJPONBLJOH

    UBTLBDIJFWFECFUUFSSFTVMUTUIBOIVNBOBOOPUBUPST IUUQTBSYJWPSHBCT 8IFOBOFOTFNCMFPGUISFF FYQFSUT JTDPSSFDU (15 PVUQFSGPSNTBOFOTFNCMFPG UISFFDSPXEXPSLFSTJO DBUFHPSJFT

  51.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  52.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 1FSGPSNBODFPOBDBEFNJDCFODINBSLT<0QFO"* > l (15BDIJFWFTWFSZIJHITDPSFTPOCFODINBSLTTVDIBTDIBSUT  EJBHSBNT

    BOEJOGPHSBQIJDT BMUIPVHIOPUZFUBUUIFIVNBOMFWFM 51 Human 80.78% 85.01% N/A N/A 98.1% 98.0% 89.61% N/A * Updated 70.5%* (DePlot) 90.1%* (BAIDU-DI) A: 4-shot, CoT prompting https://openai.com/research/gpt-4

  53.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOESBXJNBHFTXJUIQSPHSBNNJOHMBOHVBHFT 1SPNQUl%SBXBVOJDPSOJO5JL;z 5IFSJHIUNPTUJNBHFJTBMBUFSWFSTJPOPG(15JOUSBJOJOH

  54.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOESBXJNBHFTXJUIQSPHSBNNJOHMBOHVBHFT 1SPNQUl1SPEVDF+BWBDSJQU DPEF UIBUDSFBUFTBSBOEPNHSBQIJDBM JNBHFUIBUMPPLTMJLFBQBJOUJOHPG ,BOEJOTLZz $PNQPTJUJPO  BXPSLPGBSUCZ 8BTTJMZ,BOEJOTLZ

  55.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TJOUFSBDUJWFESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOBMTPJOUFSBDUJWFMZESBXJNBHFT

  56.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JTVBMEPDVNFOUVOEFSTUBOEJOH[OpenAI, March 14, 2023] l 5IJTFYBNQMFSFRVJSFTVOEFSTUBOEJOHBQIZTJDTQSPCMFNXJUIB

    EJBHSBNXSJUUFOJO'SFODI 55 Prompt: Answer question I.1.a. Think step-by-step. GPT-4’s response: IUUQTBSYJWPSHBCT

  57.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JTVBMEPDVNFOUVOEFSTUBOEJOH[OpenAI, March 14, 2023] l (15DBOSFBEBOETVNNBSJ[FEPDVNFOUJNBHFTBOEBMTPBOTXFS

    UIFRVFTUJPOBCPVUGJHVSFT l 5IJTBDIJFWFNFOUJTSFNBSLBCMFBTJUTVSQBTTFTUIFDBQBCJMJUJFTPG QSFWJPVTNPEFMTCZBTJHOJGJDBOUNBSHJO 56 User: Below is part of the InstuctGPT paper. Could you read and summarize it to me? GPT-4: The InstructGPT paper focuses on training large language models to follow instructions with human feedback. … User: Could you explain the process described in Figure 2? GPT-4: Figure 2 in the InstructGPT paper illustrates the three steps of their method: 1. Supervised fine-tuning (SFT): … 2. Reward model (RM) training: … 3. Reinforcement learning using Proximal Policy Optimization (PPO): … These three steps are iteratively performed to train the InstructGPT model, which aims to generate outputs that align better with human preferences and follow instructions more effectively. IUUQTBSYJWPSHBCT

  58.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  59.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV MMBNBDQQ <(FPSHJ(FSHBOPW BOEDPOUSJCVUPST .BSDI ʙ> l

    *UFOBCMFT MBOHVBHFNPEFMTUPCFSVOPOB.BD#PPLVTJOHCJU JOUFHFSRVBOUJ[BUJPOXJUIPVUTJHOJGJDBOUQFSGPSNBODFEFHSBEBUJPO https://github.com/AUGMXNT/llm-experiments “q8” and “q4” means 8-bit and 4-bit quantization, respectively.

  60.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "VUP(15<5PSBO#SVDF3JDIBSETBOEDPOUSJCVUPST ʙ> l "OPQFOTPVSDFBQQMJDBUJPOUIBUDIBJOTUPHFUIFS(15`T lUIPVHIUTzUPBDIJFWFVTFSTFUHPBMTBVUPOPNPVTMZ IUUQTUXJUUFSDPNTBJSBIVMTUBUVT

  61.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "VUP(15<5PSBO#SVDF3JDIBSETBOEDPOUSJCVUPST ʙ> l l4FMGQSPNQUJOHz UIFBCJMJUZUPEFWFMPQBOEFYFDVUFOFXQSPNQUT CBTFEPOJOJUJBMJOQVU

    XJMMCFBOFXUSFOEGPS--.T 6TFSTFU(PBMT  %PNBSLFUSFTFBSDIGPSEJGGFSFOUIFBEQIPOFTPOUIFNBSLFUUPEBZ  (FUUIFUPQIFBEQIPOFTBOEMJTUUIFJSQSPTBOEDPOT  *ODMVEFUIFQSJDFGPSFBDIPOFBOETBWFUIFBOBMZTJT  0ODFZPVBSFEPOF UFSNJOBUF (PPHMF TFBSDI #SPXTF XFCTJUF 8SJUFUP GJMF 2VFSZNPEJGJDBUJPO "DUJPOTUIBU"VHP(15 QMBOOFEBOENPEJGJFEGSPNPCTFSWBUJPOGFFECBDL 2VFSZCBTFE TVNNBSJ[BUJPO /PJOGPSNBUJPO

  62.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  63.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV )BMMVDJOBUJPO <0QFO"* "QSJM > l (15TJHOJGJDBOUMZJNQSPWFTGBDUVBMJUZFWBMVBUJPOTSFMBUJWFUP

    QSFWJPVT(15NPEFMT CVUJUTUJMMlIBMMVDJOBUFTzGBDUTBOENBLFT SFBTPOJOHFSSPST "DDVSBDZPO0QFO"*`T JOUFSOBMUFTUT IUUQTBSYJWPSHBCT

  64.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV )BMMVDJOBUJPO <0QFO"* "QSJM > l 3-)'JNQSPWFEGBDUVBMJUZFWBMVBUJPOTPGCPUI(15BOE(15

    JOUIF5SVUIGVM2" EBUBTFU 3-)' 3-)' (15 (15 "DDVSBDZPO 5SVUIGVM2" IUUQTBSYJWPSH BCT

  65.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV #JBT[Zhuo (Monash U), February 22, 2023]

    l --.TNBZJOBEWFSUFOUMZMFBSOCJBTGSPNMBSHF OPJTZDPSQPSB https://arxiv.org/abs/2301.12867 ,VOBTIJS*TMBOECFMPOHTUP+BQBO ˡ 3FTQPOTFTPG$IBU(15 UPl8IJDI DPVOUSZEPFT,VOBTIJSJ *TMBOECFMPOHUP z JO+BQBOFTF 3VTTJBO BOE&OHMJTI ,VOBTIJS*TMBOECFMPOHTUPUIF3VTTJBO'FEFSBUJPOʜ ,VOBTIJS*TMBOEJTDVSSFOUMZVOEFSUIFDPOUSPMPG 3VTTJB CVU+BQBOBMTPDMBJNTTPWFSFJHOUZPWFSUIF JTMBOEʜ

  66.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV #JBT<#VCCFL .JDSPTPGU .BSDI > l (15`TDIPJDFPGUIFQSPOPVO

    SFGMFDUTUIFTLFXOFTTPGUIFXPSME SFQSFTFOUBUJPOGPSUIBUPDDVQBUJPO 1SPNQUl*IBEBHSFBUFYQFSJFODFXJUIB0$$61"5*0/$BOZPVXSJUFBOPUF SFDPNNFOEJOHUIJT0$$61"5*0/UPBGSJFOEzVTFEUPDBMDVMBUFUIFMJLFMJIPPET UIFBWFSBHFQSPOPVOVTBHF IUUQTBSYJWPSHBCT

  67.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "SJUINFUJDNJTUBLFT [Bubbek (Microsoft)+, March 22, 2023]

    l (15TUJMMTUSVHHMFTXJUIBSJUINFUJDNJTUBLFT1FSGPSNJOHTFWFSBM NBOJQVMBUJPOTJOPOFTUFQPGUIFDBMDVMBUJPOMFBETUPBOFSSPS &SSPS $PSSFDU IUUQTBSYJWPSH BCT

  68.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV -PHJDBMSFBTPOJOH<-JV 8FTUMBLF6 "QSJM > l -PHJDBMSFBTPOJOHSFNBJOTDIBMMFOHJOHGPS$IBU(15

    BOE(15  FTQFDJBMMZPOOFXMZSFMFBTFEEBUBGSPN PVUPGEJTUSJCVUJPO  IUUQTBSYJWPSHBCT $IJOFTF

  69.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV -PXSFTPVSDFMBOHVBHF<#BOH )POH,POH6 'FC > $PNNPODSBXMTJ[F 

       4FOUJNFOUBOBMZTJTMBOHVBHFJEFOUJGJDBUJPOUBTLT .BDIJOFUSBOTMBUJPOBDDVSBDZPOIJHIMPXSFTPVSDFMBOHVBHFT l $IBU(15 PGUFOGBJMTBUMPXSFTPVSDFMBOHVBHFHFOFSBUJPOBOE VOEFSTUBOEJOH IUUQTBSYJWPSHBCT

  70.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4VNNBSZ l 5IF(15NPEFMDBOCFDPOTJEFSFEBOFBSMZWFSTJPOPG"(* CFDBVTFPGUIFCSFBEUIBOEEFQUIPGJUTDBQBCJMJUJFT l 5IFNPEFMTIPXTIVNBOMFWFMQFSGPSNBODFJOUIFFWBMVBUJPOPG

    NBOZOBUVSBMMBOHVBHFQSPDFTTJOHUBTLT l *UBMTPTIPXTQSPNJTJOHBOEJOUFSFTUJOHSFTVMUTTVDIBTWJTVBM EPDVNFOUVOEFSTUBOEJOHBOEESBXJOHJNBHFT l 5IFSFBSFTPNFSFNBJOJOHJTTVFTTVDIBTIBMMVDJOBUJPOBOECJBT