Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PAKDD2023 Tutorial 2: A Gentle Introduction to ...

Kyosuke Nishida
May 25, 2023
Research
7
1.8k

PAKDD2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT (Parts 3 and 4)

Jun Suzuki, Kyosuke Nishida, Naoaki Okazaki: "PAKDD 2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT," The 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2023) , May 25, 2023.

Parts 1 and 2: https://www.fai.cds.tohoku.ac.jp/research/activities/
Parts 3 and 4: this deck.
Part 5: https://speakerdeck.com/chokkan/efforts-for-responsible-llms-pakdd-2023-tutorial-2

Kyosuke Nishida

May 25, 2023
Tweet

More Decks by Kyosuke Nishida

See All by Kyosuke Nishida
論文紹介 / The Llama 3 Herd of Models
kyoun
7
1.7k
大規模言語モデル入門 / LLM introduction (SES2023)
kyoun
66
21k
論文紹介 / Llama 2: Open Foundation and Fine-Tuned Chat Models
kyoun
5
7.2k
Collaborative AI: 視覚・言語・行動の融合
kyoun
21
8.3k
NLPとVision-and-Languageの基礎・最新動向 (1) / DEIM Tutorial Part 1: NLP
kyoun
25
9.9k
NLPとVision-and-Languageの基礎・最新動向 (2) / DEIM Tutorial Part 2 Vision-and-Language
kyoun
20
11k
論文紹介 / Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
kyoun
2
850
自然言語処理とVision-and-Language / A Tutorial on NLP & Vision-and-Language
kyoun
23
12k
論文紹介 / PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
kyoun
1
750

Other Decks in Research

See All in Research
129 2 th
0325
0
250
Introducing Research Units of Matsuo-Iwasawa Laboratory
matsuolab
0
1.3k
国際会議ACL2024参加報告
chemical_tree
1
350
MetricSifter：クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
140
ECCV2024読み会: Minimalist Vision with Freeform Pixels
hsmtta
1
300
CoRL2024サーベイ
rpc
1
1.1k
EBPMにおける生成AI活用について
daimoriwaki
0
220
TransformerによるBEV Perception
hf149
1
580
最近のVisual Odometryと Depth Estimation
sgk
1
310
PetiteSRE_GenAIEraにおけるインフラのあり方観察
ichichi
0
190
さんかくのテスト.pdf
sankaku0724
0
520
Practical The One Person Framework
asonas
1
1.8k

Featured

See All Featured
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.3k
Why Our Code Smells
bkeepers
PRO
335
57k
The Invisible Side of Design
smashingmag
298
50k
Build The Right Thing And Hit Your Dates
maggiecrowley
33
2.4k
Making Projects Easy
brettharned
116
5.9k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
32
2.7k
Building Better People: How to give real-time feedback that sticks.
wjessup
365
19k
Visualization
eitanlees
146
15k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.9k
A Philosophy of Restraint
colly
203
16k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5k

Transcript

  1. 1BSUTBOE 5FDIOPMPHJFTBOE3FDFOU"DIJFWFNFOUT JO$IBU(15MJLF--.T /55)VNBO*OGPSNBUJDT-BCPSBUPSJFT ,ZPTVLF /JTIJEB .BZ  1",%%5VUPSJBM "(FOUMF*OUSPEVDUJPOUP5FDIOPMPHJFT#FIJOE

    -BOHVBHF.PEFMTBOE3FDFOU"DIJFWFNFOUJO$IBU(15

  2.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU<JOUSPEVDUPSZ>-BOHVBHFNPEFMT -.T l 1BSU-BSHFMBOHVBHFNPEFMT --.T

    u 4QFBLFS 1SPGFTTPS+VO4V[VLJ 5PIPLV6OJWFSTJUZ u 4MJEFT IUUQTXXXGBJDETUPIPLVBDKQSFTFBSDIBDUJWJUJFT l 1BSU5FDIOPMPHJFT6OEFSMZJOH$IBU(15MJLF--.T l 1BSU3FDFOUBDIJFWFNFOUTJO$IBU(15MJLF--.T l 1BSU&GGPSUTGPS3FTQPOTJCMF--.T u 4QFBLFS1SPGFTTPS/BPBLJ 0LB[BLJ 5PLZP*OTUJUVUFPG5FDIOPMPHZ u 4MJEFT IUUQTTQFBLFSEFDLDPNDIPLLBOFGGPSUTGPSSFTQPOTJCMFMMNT QBLEEUVUPSJBM

  3.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU5FDIOPMPHJFT6OEFSMZJOH$IBU(15MJLF--.T

  4.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 0QFO"* NPEFMTSFMFBTFEGPMMPXJOH(15 u $PEFY5SBJOJOHGSPNDPEFBOEUFYU

    u *OTUSVDU(15 BOE$IBU(15*OTUSVDUJPOUVOJOHBOE3-)' u (15BOE$IBU(15QMVHJOT.VMUJNPEBMBOEFYUFSOBMLOPXMFEHF JOUFHSBUJPO l 0QFOTPVSDFNPEFMT u --B.""OPQFO--.USBJOFEPOPWFSPOFUSJMMJPOUPLFOT u "MQBDBBOE7JDVOB'JOFUVOFENPEFMTCBTFEPO--B." u .15"OFXDPNNFSDJBMMZVTBCMFNPEFMDBQBCMFPGIBOEMJOHMPOH DPOUFYU u --B7" BOE.JOJ(15NVMUJNPEBMNPEFMT

  5.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV &WPMVUJPOGSPN(15UP(15 l #FUXFFO(15BOE(15 UIFSFBSFUISFFNPEFMTSFGFSSFEUPBT UIF(15GBNJMZ l

    (15 .BZ QBQFS ˣ 5SBJOJOH-.TXJUITPVSDFDPEF l $PEFY +VMZ QBQFS ˣ 5SBJOJOH-.TUPGPMMPXJOTUSVDUJPOTXJUIIVNBOGFFECBDL l *OTUSVDU(15 +BO  QBQFS ˣ 0QUJNJ[BUJPO GPSDIBU l $IBU(15 /PW  CMPH ˣ .VMUJNPEBMJOUFHSBUJPO l (15 .BS UFDIOJDBMSFQPSUCMPH

  6.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $PEFY<$IFO 0QFO"* +VMZ> l #VJMUPOUIFTBNFGPVOEBUJPOBT(15 l

    5PFOBCMFUIF--.UPHFOFSBUFDPEFJOSFTQPOTFUPVTFSJOTUSVDUJPOT  JUTUSBJOJOHEBUBTFUJODMVEFTCJMMJPOTPGMJOFTPGDPEF l *UTDPOUFYUJTFYQBOEFEUP UPLFOT l 5IFMBUFTU"1*WFSTJPO DPEFEBWJODJ XPSLTJOPWFSUFOMBOHVBHFT UIFQSPNQU NFUIPEEFGJOJUJPOTBOE *OQVUPVUQVUFYBNQMFT QSPWJEFEUPUIFNPEFM NPEFMHFOFSBUFEDPNQMFUJPO IUUQTBSYJWPSHBCT

  7.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (JU)VC$PQJMPU<+VOF> l $PEFYQPXFSTBOl"*QBJSQSPHSBNNFSzPO(JUIVC l 4JODFJUTSFMFBTF JUIBTUSBOTGPSNFEEFWFMPQFSQSPEVDUJWJUZGPS

    NPSFUIBOPOFNJMMJPOQFPQMF IFMQJOHEFWFMPQFSTDPEFVQUP GBTUFS<> <>IUUQTHJUIVCCMPHHJUIVCDPQJMPUGPSCVTJOFTTJTOPXBWBJMBCMF

  8.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 5PLFOJ[FSPG$PEFY l .PEJGJFEGSPN(15 XJUIUIFBEEJUJPOPGUPDPOTFDVUJWF XIJUFTQBDFTUPUIFWPDBCVMBSZ 5IFTFFYBNQMFTXFSFHFOFSBUFEVTJOHIUUQTQMBUGPSNPQFOBJDPNUPLFOJ[FS

    $PEFY`TUPLFOJ[BUJPO SL@CBTF (15 QL@CBTF DPOTFDVUJWFXIJUFTQBDFT BSFDPOWFSUFEJOUPBTJOHMFUPLFO

  9.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV *OTUSVDU(15 <0VZBOH 0QFO"* .BSDI> l 4JHOJGJDBOUMZPVUQFSGPSNT(15JOGPMMPXJOHVTFSJOTUSVDUJPOT

     SFTVMUJOHJOPVUQVUTUIBUBSFNPSFUSVUIGVMBOEMFTTUPYJD IUUQTPQFOBJDPNSFTFBSDIJOTUSVDUJPOGPMMPXJOHTBNQMF VTFSJOTUSVDUJPO 5IJTBOTXFSlBMJHOTzXJUIUIFJOTUSVDUJPO (15GBJMTUPSFTQPOEUPUIFVTFSJOTUSVDUJPOT

  10.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV IUUQTBSYJWPSHBCT *OTUSVDUJPOUVOJOH 3FJOGPSDFNFOU-FBSOJOHGSPN)VNBO'FFECBDL 3-)' -FBSOJOHTUFQTPG*OTUSVDU(15

  11.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV *OTUSVDUJPOUVOJOH<8FJ (PPHMF 4FQ><$IVOH (PPHMF 0DU> l

    'JOFUVOJOHMBOHVBHFNPEFMTPOBDPMMFDUJPOPGUBTLTEFTDSJCFEWJB JOTUSVDUJPOT JNQSPWFT[FSPTIPUQFSGPSNBODFPOVOTFFOUBTLT l 0QFO"* VTFEBCPVULUSBJOJOHQSPNQUT GSPNUIF0QFO"* "1* BOEMBCFMFSXSJUUFO GPSJOTUSVDUJPOUVOJOH u 5IJTTVQFSWJTFEGJOFUVOFENPEFMJTDBMMFEUIF4'5 NPEFM IUUQTBSYJWPSHBCT

  12.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 3FXBSENPEFMJOH l -BCFMFSTXFSFQSFTFOUFEXJUICFUXFFOBOESFTQPOTFTUPFBDI QSPNQUBOEBTLFEUPSBOLUIFN l 5IFSFXBSENPEFM

    TJ[FPG# XBTUSBJOFEXJUIBCPVULUSBJOJOH QSPNQUT GSPNUIF0QFO"* "1*BOEMBCFMFSXSJUUFO  ,SFTQPOTFT ! " DPNQBSJTPOT SBOLJOH

  13.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 3FJOGPSDFNFOU-FBSOJOHGSPN)VNBO'FFECBDL 3-)' l 0SJHJOBMMZEFWFMPQFEGPSUSBJOJOHTJNQMFSPCPUTBOE"UBSJHBNFT <$ISJTUJBOP *CBS[

    > UIFOBQQMJFEUPUIFGJOFUVOJOH PG-.TUPTVNNBSJ[FUFYU<4UJFOOPO > l 0QFO"* IBTBMTPBQQMJFE3-)'UPGJOFUVOFUIF4'5NPEFM VTJOH BCPVULUSBJOJOHQSPNQUT GSPNUIF0QFO"* "1*  l 3-0CKFDUJWF110QUY BWBSJBOUPG110<4DIVMNBO > ",VMMCBDL-JCMFS QFOBMUZGSPNUIF4'5NPEFM UP NJUJHBUFPWFSPQUJNJ[BUJPOPGUIFSFXBSENPEFM 1SFUSBJOJOHNJYUPGJYUIFQFSGPSNBODF SFHSFTTJPOTPOQVCMJD/-1EBUBTFUT

  14.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15<0QFO"* /PW  > l 0QUJNJ[FEGPSDIBUVTJOHUIFTBNFNFUIPETBT*OTUSVDU(15<>

    u /PGVSUIFSEFUBJMTBCPVUUIFBSDIJUFDUVSF NPEFMTJ[F EBUBTFU DPOTUSVDUJPO PSUSBJOJOHNFUIPETIBWFCFFOSFQPSUFE <>IUUQTPQFOBJDPNCMPHDIBUHQU 'PSJOTUSVDUJPOUVOJOH  )VNBO"*USBJOFSTQSPWJEFEDPOWFSTBUJPOT JOXIJDIUIFZQMBZFECPUITJEFT<>

  15.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 5PLFOJ[FSPG$IBU(15 l /FXMZDSFBUFE DBMMFElDML@CBTFz BOEUIFWPDBCVMBSZTJ[FIBT CFFOFYQBOEFEVQUPL

    *OTUSVDU(15 SL@CBTF `T UPLFOJ[BUJPOPGl͜Μʹͪ͸z lIFMMPzJO+BQBOFTF 5PLFOTIBWFCFFOJODSFBTFEGPS QSPHSBNJOHMBOHVBHFTBOEOPO&OHMJTIMBOHVBHFT "QBSUPG lDML@CBTFz WPDBCVMBSZ

  16.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15<0QFO"* .BSDI > l $BOBDDFQUJNBHFBOEUFYUJOQVUTBOEQSPEVDFUFYUPVUQVUT (15`TSFTQPOTF

    IUUQTBSYJWPSHBCT

  17.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15<0QFO"* .BSDI > l "5SBOTGPSNFSTUZMFNPEFMQSFUSBJOFEUPQSFEJDUUIFOFYUUPLFOJO BEPDVNFOU

    XJUIBDPOUFYUPGLUPLFOT BOEGJOFUVOFEVTJOH3-)' u /PGVSUIFSEFUBJMTBCPVUUIFBSDIJUFDUVSF NPEFMTJ[F IBSEXBSF USBJOJOH DPNQVUBUJPO EBUBTFUDPOTUSVDUJPO PSUSBJOJOHNFUIPETIBWFCFFO SFQPSUFE *UJTVODMFBSIPXUIFJNBHFTBOEUFYUGPSN UIFJOQVUTFRVFODFBOEBSFGFEUP(15 *UJTVODMFBSIPXJUQSPDFTTFTMPOHTFRVFODFT PGVQUPLUPLFOTBUIJHITQFFE

  18.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 1MVHJOTBSFFYUFOTJPOT

    JODMVEJOHUIJSEQBSUZBQQMJDBUJPOT UIBU BMMPXNPEFMTUPJOUFSBDUXJUIFYUFSOBM"1*T IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT

  19.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IFl8FC#SPXTFSzQMVHJOBMMPXTBDDFTTUPVQUPEBUFJOGPSNBUJPO

    IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 18 IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 5IFNPEFMJTBCMFUPQSPWJEFVQUPEBUF JOGPSNBUJPO FWFOUIPVHIJUJTPOMZUSBJOFE POEBUBVQUP0DUPCFS

  20.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IFl8FC#SPXTFSz

    QMVHJOBMMPXTBDDFTTUPVQUPEBUFJOGPSNBUJPO IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT 19 IUUQTPQFOBJDPNCMPHDIBUHQUQMVHJOT

  21.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 1MVHJOT<0QFO"* .BSDI > l 5IF8PMGSBNQMVHJOBMMPXTBDDFTTUPQPXFSGVMDPNQVUBUJPO

     BDDVSBUFNBUI DVSBUFELOPXMFEHF BOEWJTVBMJ[BUJPO 20

  22.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUTPGUIJTQBSU l 0QFO"*bT NPEFMT u $PEFYUSBJOJOHGSPNDPEFBOEUFYU

    u *OTUSVDU(15 BOE$IBU(15JOTUSVDUJPOUVOJOHBOE3-)' u (15NVMUJNPEBMJOUFHSBUJPO l 0QFOTPVSDFNPEFMT u --B."BOPQFO--.USBJOFEPOPWFSPOFUSJMMJPOUPLFOT u "MQBDBBOE7JDVOB'JOFUVOFENPEFMTCBTFEPO--B." u .15BDPNNFSDJBMMZVTBCMFNPEFMDBQBCMFPGIBOEMJOHMPOH DPOUFYU u --B7" BOE.JOJ(15NVMUJNPEBMNPEFMT

  23.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV --B." <5PVWSPO .FUB 'FC  >

    l "OPQFOTPVSDF--.USBJOFEXJUIQVCMJDMZBWBJMBCMFEBUBTFUT l 5IFQSFUSBJOJOHEBUBIBTCFFOSFDFOUMZSFQSPEVDFEBT l3FE1BKBNB%BUBz<5PHFUIFS$PNQVUFS>BOEJTBWBJMBCMFGPSBOZPOF IUUQTBSYJWPSHBCT USJMMJPO UPLFOT

  24.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B."#JTDPNQBSBCMFUP(15 # PONPTUCFODINBSLT l 5IFOVNCFSPGUSBJOJOHUPLFOTJO--B."

    5 XBTTJHOJGJDBOUMZ HSFBUFSUIBOUIBUPG(15 #  u DG$IJODIJMMB<)PGGNBO %FFQNJOE > ;FSPTIPUQFSGPSNBODFPODPNNPOTFOTFSFBTPOJOH IUUQTBSYJWPSHBCT

  25.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "SDIJUFDUVSFBOEUSBJOJOHNFUIPE l --B." MFWFSBHFTWBSJPVTJNQSPWFNFOUTUIBUXFSFQSPQPTFE BGUFS UIFPSJHJOBM5SBOTGPSNFSQVCMJDBUJPO

    u (15`TQSFOPSNBMJ[BUJPO u 1B-.`T 4XJ(-6 BDUJWBUJPOGVODUJPO<4IB[FFS> u (15/FP`T3PUBSZ&NCFEEJOHT<4V > l 5IFUSBJOJOHNFUIPEJTUIFTBNFBT(15 JF OFYUUPLFO QSFEJDUJPO  l --B."JUTFMGJTOPUUSBJOFEXJUIJOTUSVDUJPOUVOJOHBOE3-)'

  26.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "MQBDB<4UBOGPSE6 .BSDI > l "NPEFMGJOFUVOFEGSPNUIF--B." #NPEFMPO,JOTUSVDUJPO

    GPMMPXJOHEFNPOTUSBUJPOT l "TFMGJOTUSVDUUFDIOJRVF<8BOH >XBTVTFEUPBVUPNBUJDBMMZ HFOFSBUFUIFEFNPOTUSBUJPOGSPNPOMZIVNBOXSJUUFOTFFEUBTLT u 5IFUFYUEBWJODJ *OTUSVDU(15 NPEFMXBTVTFEJOUIJTTUVEZ IUUQTDSGNTUBOGPSEFEV BMQBDBIUNM

  27.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "MQBDB-P3" <.BSDI > l -P3" <)V

    >GSFF[FTUIFQSFUSBJOFEXFJHIUTBOEJOKFDUT USBJOBCMFBEBQUFST SBOLEFDPNQPTJUJPONBUSJDFT JOUPFBDIMBZFS l 6TJOH--B." -P3" BOEUIF"MQBDBEBUBTFU BOZPOFDBODIFBQMZ QSPEVDF$IBU(15MJLFNPEFMTXJUIJOIPVSTPOBTJOHMF(16 IUUQTBSYJWPSHBCT IUUQTHJUIVCDPNUMPFOBMQBDBMPSB 7BSJPVTBEBQUFSTIBWFCFFO SFMFBTFEUPNBLF"MQBDBDPNQBUJCMF XJUIOPO&OHMJTIMBOHVBHFT

  28.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JDVOB<6$#FSLFMFZ $.6 4UBOGPSE BOE6$4BO%JFHP.BSDI > l

    "NPEFMGJOFUVOFEGSPN--B." PO,VTFSTIBSFE DPOWFSTBUJPOTDPMMFDUFEGSPNUIF4IBSF(15 XFCTJUF l &WBMVBUJPOVTJOH(15BTBKVEHFTIPXTUIBU7JDVOB# BDIJFWFTNPSFUIBOUIFRVBMJUZPG$IBU(15 5XPXFFLTBQBSU 5XPXFFLTBQBSU IUUQTMNTZTPSHCMPHWJDVOB

  29.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JDVOB`T"VUPNBUJDFWBMVBUJPOVTJOH(15 l (15DBOQSPEVDFIJHIMZDPOTJTUFOUSBOLTBOEBEFUBJMFE BTTFTTNFOUXIFODPNQBSJOHDIBUCPUT`BOTXFST IUUQTMNTZTPSHCMPHWJDVOB

  30.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4UBCMF7JDVOB <4UBCJMJUZ"* "QSJM > l "MQBDBBOE7JDVOBVTFEPOMZJOTUSVDUJPOUVOJOH

    l 4UBCMF7JDVOB JTUIFGJSTUMBSHFTDBMFPQFOTPVSDFDIBUCPUUSBJOFE WJB3-)'VTJOHUIFGPMMPXJOHPQFOEBUBTFUT u 0QFO"TTJTUBOU $POWFSTBUJPOT 0"445 <"QSJM > u 4UBOGPSE)VNBO1SFGFSFODFT 4)1 <'FCSVBSZ > u "OUISPQJD))3-)'<#BJ "QSJM >

  31.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .15<MosaicML, May 5, 2023] l "OPQFOTPVSDF

    DPNNFSDJBMMZVTBCMF--. USBJOFEGSPNTDSBUDI PO5UPLFOTPGUFYUBOEDPEF l .15#NBUDIFTUIFRVBMJUZPG--B."#POTUBOEBSEUBTLT IUUQTXXXNPTBJDNM DPNCMPHNQUC

  32.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .15#4UPSZ8SJUFSL <MosaicML, May 5, 2023] l

    #VJMUCZGJOFUVOJOH.15#XJUIBDPOUFYUMFOHUIPGLUPLFOTUP SFBEBOEXSJUFTUPSJFTXJUITVQFSMPOHDPOUFYUMFOHUIT u $BOSFBEUIFFOUJSFUFYUPGl3PNFPBOE+VMJFUz<> XIJDIIBTBCPVUL UPLFOTVTJOH(15`TUPLFOJ[FS DML@CBTF  l 6TFT"-J#J <1SFTT "VHVTU>JOTUFBEPGQPTJUJPOBMFNCFEEJOHT UPIBOEMFTVDIMPOHDPOUFYUT <>IUUQTXXXHVUFOCFSHPSHDBDIFFQVCQHUYU

  33.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "-J#J "UUFOUJPOXJUI-JOFBS#JBTFT l CJBTFTRVFSZLFZBUUFOUJPOTDPSFTXJUIBIFBETQFDJGJDQFOBMUZUIBU JTQSPQPSUJPOBMUPUIFJSEJTUBODF https://arxiv.org/abs/2108.12409

    XIFSFTDBMBSNJTBIFBETQFDJGJDTMPQFGJYFECFGPSFUSBJOJOH <1SFTT 68BTIJOHUPO'"*3 "VHVTU > "MJ#JFOBCMFTUIF53"*/ 4)035 5&45-0/(DPODFQU  XIFSFUIFNPEFMUSBJOFEPOUIF DPOUFYUMFOHUIPGQFSGPSNT XFMMGPSLJOQVUUPLFOTBU JOGFSFODFUJNF UIBOLTUP"MJCJ

  34.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l 5IFSFBSFTFWFSBMPQFOTPVSDFTUVEJFTGPSNVMUJNPEBMNPEFMT l 'PSFYBNQMF --B7" BMMPXTGPSWJTVBMMZHSPVOEFEDIBUTMJLF(15

    IUUQTBSYJWPSHBCT --B7" <-JV 68JTDPOTJOr.BEJTPO "QSJM >

  35.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B7" HFOFSBUFTWJTVBMJOTUSVDUJPOUVOJOHEBUBCZQBTTJOH$0$0 DBQUJPOTBOECPVOEJOHCPYFTUPUIFMBOHVBHFPOMZ(15 l 5IFSFTVMUJOHEBUBTFUJTVTFEUPGJOFUVOFNVMUJNPEBMNPEFMT

    6TJOH-BOHVBHFPOMZ(15 7JTVBM *OTUSVDUJPO5VOJOH EBUBTFU IUUQTBSYJWPSHBCT --B7"`T 7JTVBM*OTUSVDUJPO5VOJOH

  36.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l --B7" DPOTJTUTPG--B." BOEUIFWJTVBMFODPEFSPG$-*1 l 'PSBOJOQVUJNBHF

    UIFQSFUSBJOFE$-*1WJTVBMFODPEFS 7J5-  DBOPVUQVUHSJEGFBUVSFTBTWJTVBMUPLFOT $-*1`THSJEGFBUVSFT <3BEGPSE +BOVBSZ> 8IBUJTVOVTVBMBCPVU UIJTJNBHF  5IFVOVTVBM BTQFDUPG UIJTJNBHF JTBNBO JSPOJOHʜ --B." IUUQTBSYJWPSHBCT .PEFM"SDIJUFDUVSF

  37.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .JOJ(15<;IV ,JOH"CEVMMBI6 "QSJM > IUUQTXXXZPVUVCFDPNXBUDI W@@UGUPYQ#"X

  38.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV .JOJ(15<;IV ,JOH"CEVMMBI6 "QSJM > l $POTJTUTPG7JDVOBBOE#-*1`TWJTVBMFODPEFS<-J

    4BMFTGPSDF b> l 3FRVJSFTPOMZUSBJOJOHUIFMJOFBSQSPKFDUJPOMBZFSUPBMJHOUIFWJTVBM GFBUVSFTXJUIUIF7JDVOB BLIP-2 IUUQTBSYJWPSHBCT

  39.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4VNNBSZ l .PEFSO--.T TUBSUJOHGSPN$PEFY BSFUSBJOFEPONBTTJWF BNPVOUTPGUFYUBOEDPEF

    l *NQPSUBOUUFDIOJRVFTGPSBMJHOJOHNPEFMTUPGPMMPXJOTUSVDUJPOTBSF lJOTUSVDUJPOUVSOJOHzBOElSFJOGPSDFNFOUMFBSOJOHGSPNIVNBO GFFECBDL 3-)' z l 5IFUFDIOJDBMEFUBJMTPG(15IBWFOPUZFUCFFOSFQPSUFE l 4JODFUIFMBVODIPG--B." UIFSFIBTCFFOSFNBSLBCMFQSPHSFTTJO UIFEFWFMPQNFOUPGPQFOTPVSDFNPEFMTBOEEBUBTFUT

  40.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(1  5IV l 1BSU3FDFOUBDIJFWFNFOUTJO$IBU(15MJLF--.T

  41.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $IBU(15 JTUIFGBTUFTUHSPXJOHBQQFWFS l $IBU(15 SFBDIFTNJMMJPOVTFSTUXPNPOUITBGUFSMBVODI l

    *UIBTBUUSBDUFEBHSFBUEFBMPGBUUFOUJPOEVFUPJUTIJHIQFSGPSNBODF IUUQTUXJUUFSDPN&DPOPNZ"QQTUBUVT

  42.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4QBSLTPG"(*<#VCCFDL .JDSPTPGU .BZ > l 0OFXFFLBGUFSUIFSFMFBTFPG(15

    .JDSPTPGUQVCMJTIFEBQBQFS FOUJUMFEl4QBSLTPG"SUJGJDJBM(FOFSBM*OUFMMJHFODF&BSMZ&YQFSJNFOUT XJUI(15z l 5IFBVUIPSTTUBUFEUIBU(15DPVMECFDPOTJEFSFEBOFBSMZ WFSTJPOPGBSUJGJDJBMJOUFMMJHFODF "(* CFDBVTFPGUIFCSFBEUIBOE EFQUIPG(15TDBQBCJMJUJFTJONBUIFNBUJDT DPEJOH WJTJPO  NFEJDJOF MBX QTZDIPMPHZ BOENPSF l 5IFBVUIPSTVTFEUIFUFSNl"(*zUPSFGFSUPTZTUFNTUIBU EFNPOTUSBUFCSPBEJOUFMMJHFODFDBQBCJMJUJFTBUPSBCPWFIVNBO MFWFM

  43.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  44.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15QBTTFTUIFCBSFYBN<,BU[ *MMJOPJT5FDI .BSDI > l (15

    [FSPTIPUFWBMVBUJPOT IBTQBTTFEUIFFOUJSF6OJGPSN#BS &YBNJOBUJPO 6#& JODMVEJOHOPUPOMZNVMUJQMFDIPJDF .#& CVU BMTP PQFOFOEFERVFTUJPOT .&&BOE.15  l (15BDIJFWFEBOPWFSBMMTDPSFPGQPJOUT TJHOJGJDBOUMZ TVSQBTTJOHUIFQBTTJOHTDPSFSBOHFPGUPQPJOUT IUUQTQBQFSTTTSODPNTPMQBQFSTDGN BCTUSBDU@JE

  45.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15 [FSPTIPUFWBM FYDFFETUIFQBTTJOHTDPSFPO64.-& 6OJUFE4UBUFT.FEJDBM-JDFOTJOH&YBNJOBUJPO CZPWFSQPJOUT

    IUUQTBSYJWPSHBCT QBTTUISFTIPMETBQQSPYJNBUFMZPG NVMUJQMFDIPJDFRVFTUJPOT (15QBTTFTNFEJDBMMJDFOTJOHFYBN</PSJ .JDSPTPGU .BZ>

  46.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15JTDPNQBSBCMFUPIVNBOQFSGPSNBODFPOBCFODINBSLPG VOTFFO QSPHSBNNJOHQSPCMFNTPO-FFU$PEF l (15BMTPQBTTFTNPDLUFDIOJDBMJOUFSWJFXTBUNBKPSUFDI

    DPNQBOJFTPO-FFU$PEF (15DPVMECFBTPGUXBSFFOHJOFFS<#VCCFL .JDSPTPGU .BSDI > QBTT!L NFBTVSFTXIFUIFSUIFNPEFM QSPEVDFTBDPSSFDUTPMVUJPOXJUIJOLUIBUUFNQUT (15TPMWFEBMMRVFTUJPOTJOPOMZ NJOVUFTPGUIFBMMPUUFE IPVST https://arxiv.org/abs/2303.12712

  47.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNT(15PONPTUBDBEFNJDBOEQSPGFTTJPOBM FYBNT JODMVEJOHNVMUJQMFDIPJDFBOEGSFFSFTQPOTFRVFTUJPOT l (15`TQFSGPSNBODFJTCFUUFSXIFOJUUBLFTWJTJPOBTJOQVUT

    IUUQTBSYJWPSHBCT 1FSGPSNBODFPOFYBNT<0QFO"* "QSJM >

  48.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNFE TUBUFPGUIFBSUNPEFMTXJUICFODINBSL TQFDJGJDUSBJOJOH PONBOZBDBEFNJDCFODINBSLT (15DBOCFBUGJOFUVOFE405"NPEFMT<0QFO"*

    "QSJM > https://openai.com/ research/gpt-4

  49.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV l (15PVUQFSGPSNTUIF&OHMJTIMBOHVBHFQFSGPSNBODFPGFYJTUJOH --.T GPSUIFWBTUNBKPSJUZPGMBOHVBHFTUFTUFE JODMVEJOHMPX SFTPVSDFMBOHVBHFTTVDIBT-BUWJBO

    8FMTI BOE4XBIJMJ 1FSGPSNBODFPO.VMUJMJOHVBM..-6<0QFO"* "QSJM > .VMUJMJOHVBM ..-6CFODINBSL USBOTMBUFEXJUI "[VSF

  50.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15PVUQFSGPSNTDSPXEXPSLFST<1BO 6$# "QSJM > l (15BVUPNBUJPOPGTDFOBSJPMBCFMJOHJOBTPDJBMEFDJTJPONBLJOH

    UBTLBDIJFWFECFUUFSSFTVMUTUIBOIVNBOBOOPUBUPST IUUQTBSYJWPSHBCT 8IFOBOFOTFNCMFPGUISFF FYQFSUT JTDPSSFDU (15 PVUQFSGPSNTBOFOTFNCMFPG UISFFDSPXEXPSLFSTJO DBUFHPSJFT

  51.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  52.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 1FSGPSNBODFPOBDBEFNJDCFODINBSLT<0QFO"* > l (15BDIJFWFTWFSZIJHITDPSFTPOCFODINBSLTTVDIBTDIBSUT  EJBHSBNT

    BOEJOGPHSBQIJDT BMUIPVHIOPUZFUBUUIFIVNBOMFWFM 51 Human 80.78% 85.01% N/A N/A 98.1% 98.0% 89.61% N/A * Updated 70.5%* (DePlot) 90.1%* (BAIDU-DI) A: 4-shot, CoT prompting https://openai.com/research/gpt-4

  53.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOESBXJNBHFTXJUIQSPHSBNNJOHMBOHVBHFT 1SPNQUl%SBXBVOJDPSOJO5JL;z 5IFSJHIUNPTUJNBHFJTBMBUFSWFSTJPOPG(15JOUSBJOJOH

  54.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOESBXJNBHFTXJUIQSPHSBNNJOHMBOHVBHFT 1SPNQUl1SPEVDF+BWBDSJQU DPEF UIBUDSFBUFTBSBOEPNHSBQIJDBM JNBHFUIBUMPPLTMJLFBQBJOUJOHPG ,BOEJOTLZz $PNQPTJUJPO  BXPSLPGBSUCZ 8BTTJMZ,BOEJOTLZ

  55.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV (15`TJOUFSBDUJWFESBXJOHT[Bubbek(Microsoft)+, March 22, 2023] IUUQTBSYJWPSHBCT l

    (15DBOBMTPJOUFSBDUJWFMZESBXJNBHFT

  56.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JTVBMEPDVNFOUVOEFSTUBOEJOH[OpenAI, March 14, 2023] l 5IJTFYBNQMFSFRVJSFTVOEFSTUBOEJOHBQIZTJDTQSPCMFNXJUIB

    EJBHSBNXSJUUFOJO'SFODI 55 Prompt: Answer question I.1.a. Think step-by-step. GPT-4’s response: IUUQTBSYJWPSHBCT

  57.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 7JTVBMEPDVNFOUVOEFSTUBOEJOH[OpenAI, March 14, 2023] l (15DBOSFBEBOETVNNBSJ[FEPDVNFOUJNBHFTBOEBMTPBOTXFS

    UIFRVFTUJPOBCPVUGJHVSFT l 5IJTBDIJFWFNFOUJTSFNBSLBCMFBTJUTVSQBTTFTUIFDBQBCJMJUJFTPG QSFWJPVTNPEFMTCZBTJHOJGJDBOUNBSHJO 56 User: Below is part of the InstuctGPT paper. Could you read and summarize it to me? GPT-4: The InstructGPT paper focuses on training large language models to follow instructions with human feedback. … User: Could you explain the process described in Figure 2? GPT-4: Figure 2 in the InstructGPT paper illustrates the three steps of their method: 1. Supervised fine-tuning (SFT): … 2. Reward model (RM) training: … 3. Reinforcement learning using Proximal Policy Optimization (PPO): … These three steps are iteratively performed to train the InstructGPT model, which aims to generate outputs that align better with human preferences and follow instructions more effectively. IUUQTBSYJWPSHBCT

  58.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  59.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV MMBNBDQQ <(FPSHJ(FSHBOPW BOEDPOUSJCVUPST .BSDI ʙ> l

    *UFOBCMFT MBOHVBHFNPEFMTUPCFSVOPOB.BD#PPLVTJOHCJU JOUFHFSRVBOUJ[BUJPOXJUIPVUTJHOJGJDBOUQFSGPSNBODFEFHSBEBUJPO https://github.com/AUGMXNT/llm-experiments “q8” and “q4” means 8-bit and 4-bit quantization, respectively.

  60.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "VUP(15<5PSBO#SVDF3JDIBSETBOEDPOUSJCVUPST ʙ> l "OPQFOTPVSDFBQQMJDBUJPOUIBUDIBJOTUPHFUIFS(15`T lUIPVHIUTzUPBDIJFWFVTFSTFUHPBMTBVUPOPNPVTMZ IUUQTUXJUUFSDPNTBJSBIVMTUBUVT

  61.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "VUP(15<5PSBO#SVDF3JDIBSETBOEDPOUSJCVUPST ʙ> l l4FMGQSPNQUJOHz UIFBCJMJUZUPEFWFMPQBOEFYFDVUFOFXQSPNQUT CBTFEPOJOJUJBMJOQVU

    XJMMCFBOFXUSFOEGPS--.T 6TFSTFU(PBMT  %PNBSLFUSFTFBSDIGPSEJGGFSFOUIFBEQIPOFTPOUIFNBSLFUUPEBZ  (FUUIFUPQIFBEQIPOFTBOEMJTUUIFJSQSPTBOEDPOT  *ODMVEFUIFQSJDFGPSFBDIPOFBOETBWFUIFBOBMZTJT  0ODFZPVBSFEPOF UFSNJOBUF (PPHMF TFBSDI #SPXTF XFCTJUF 8SJUFUP GJMF 2VFSZNPEJGJDBUJPO "DUJPOTUIBU"VHP(15 QMBOOFEBOENPEJGJFEGSPNPCTFSWBUJPOGFFECBDL 2VFSZCBTFE TVNNBSJ[BUJPO /PJOGPSNBUJPO

  62.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV $POUFOUT l 1FSGPSNBODFJO/BUVSBM-BOHVBHF1SPDFTTJOHCFODINBSLTBOE FYBNT l *OUFSFTUJOHSFTVMUTJO7JTJPOBOE-BOHVBHF6OEFSTUBOEJOH

    FWBMVBUJPOT l 0QFOTPVSDFBQQMJDBUJPOTQPXFSFECZ--.T l 3FNBJOJOHJTTVFT

  63.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV )BMMVDJOBUJPO <0QFO"* "QSJM > l (15TJHOJGJDBOUMZJNQSPWFTGBDUVBMJUZFWBMVBUJPOTSFMBUJWFUP

    QSFWJPVT(15NPEFMT CVUJUTUJMMlIBMMVDJOBUFTzGBDUTBOENBLFT SFBTPOJOHFSSPST "DDVSBDZPO0QFO"*`T JOUFSOBMUFTUT IUUQTBSYJWPSHBCT

  64.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV )BMMVDJOBUJPO <0QFO"* "QSJM > l 3-)'JNQSPWFEGBDUVBMJUZFWBMVBUJPOTPGCPUI(15BOE(15

    JOUIF5SVUIGVM2" EBUBTFU 3-)' 3-)' (15 (15 "DDVSBDZPO 5SVUIGVM2" IUUQTBSYJWPSH BCT

  65.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV #JBT[Zhuo (Monash U), February 22, 2023]

    l --.TNBZJOBEWFSUFOUMZMFBSOCJBTGSPNMBSHF OPJTZDPSQPSB https://arxiv.org/abs/2301.12867 ,VOBTIJS*TMBOECFMPOHTUP+BQBO ˡ 3FTQPOTFTPG$IBU(15 UPl8IJDI DPVOUSZEPFT,VOBTIJSJ *TMBOECFMPOHUP z JO+BQBOFTF 3VTTJBO BOE&OHMJTI ,VOBTIJS*TMBOECFMPOHTUPUIF3VTTJBO'FEFSBUJPOʜ ,VOBTIJS*TMBOEJTDVSSFOUMZVOEFSUIFDPOUSPMPG 3VTTJB CVU+BQBOBMTPDMBJNTTPWFSFJHOUZPWFSUIF JTMBOEʜ

  66.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV #JBT<#VCCFL .JDSPTPGU .BSDI > l (15`TDIPJDFPGUIFQSPOPVO

    SFGMFDUTUIFTLFXOFTTPGUIFXPSME SFQSFTFOUBUJPOGPSUIBUPDDVQBUJPO 1SPNQUl*IBEBHSFBUFYQFSJFODFXJUIB0$$61"5*0/$BOZPVXSJUFBOPUF SFDPNNFOEJOHUIJT0$$61"5*0/UPBGSJFOEzVTFEUPDBMDVMBUFUIFMJLFMJIPPET UIFBWFSBHFQSPOPVOVTBHF IUUQTBSYJWPSHBCT

  67.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV "SJUINFUJDNJTUBLFT [Bubbek (Microsoft)+, March 22, 2023]

    l (15TUJMMTUSVHHMFTXJUIBSJUINFUJDNJTUBLFT1FSGPSNJOHTFWFSBM NBOJQVMBUJPOTJOPOFTUFQPGUIFDBMDVMBUJPOMFBETUPBOFSSPS &SSPS $PSSFDU IUUQTBSYJWPSH BCT

  68.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV -PHJDBMSFBTPOJOH<-JV 8FTUMBLF6 "QSJM > l -PHJDBMSFBTPOJOHSFNBJOTDIBMMFOHJOHGPS$IBU(15

    BOE(15  FTQFDJBMMZPOOFXMZSFMFBTFEEBUBGSPN PVUPGEJTUSJCVUJPO  IUUQTBSYJWPSHBCT $IJOFTF

  69.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV -PXSFTPVSDFMBOHVBHF<#BOH )POH,POH6 'FC > $PNNPODSBXMTJ[F 

       4FOUJNFOUBOBMZTJTMBOHVBHFJEFOUJGJDBUJPOUBTLT .BDIJOFUSBOTMBUJPOBDDVSBDZPOIJHIMPXSFTPVSDFMBOHVBHFT l $IBU(15 PGUFOGBJMTBUMPXSFTPVSDFMBOHVBHFHFOFSBUJPOBOE VOEFSTUBOEJOH IUUQTBSYJWPSHBCT

  70.  1",%%5VUPSJBM"(FOUMF*OUSPEVDUJPOUP5FDI#FIJOE-.TBOE3FDFOU"DIJFWFNFOUJO$IBU(15  5IV 4VNNBSZ l 5IF(15NPEFMDBOCFDPOTJEFSFEBOFBSMZWFSTJPOPG"(* CFDBVTFPGUIFCSFBEUIBOEEFQUIPGJUTDBQBCJMJUJFT l 5IFNPEFMTIPXTIVNBOMFWFMQFSGPSNBODFJOUIFFWBMVBUJPOPG

    NBOZOBUVSBMMBOHVBHFQSPDFTTJOHUBTLT l *UBMTPTIPXTQSPNJTJOHBOEJOUFSFTUJOHSFTVMUTTVDIBTWJTVBM EPDVNFOUVOEFSTUBOEJOHBOEESBXJOHJNBHFT l 5IFSFBSFTPNFSFNBJOJOHJTTVFTTVDIBTIBMMVDJOBUJPOBOECJBT