Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer
Scatter Lab Inc.
May 22, 2020
Research
0
1.3k
Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer
Scatter Lab Inc.
May 22, 2020
Tweet
Share
More Decks by Scatter Lab Inc.
See All by Scatter Lab Inc.
scatterlab
0
1.4k
scatterlab
0
1.2k
scatterlab
0
1.1k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.2k
scatterlab
0
1.3k
scatterlab
0
1.3k
Other Decks in Research
See All in Research
matsumoto_r
PRO
1
220
chikuwait
2
320
makenowjust
1
110
hkefka385
0
120
yamathcy
0
280
nk35jk
0
910
corgies
0
210
kayceesrk
2
3.9k
matsumoto_r
PRO
7
1.9k
tarugoconf
0
1.7k
ailaboocu
0
180
ysekky
3
1.5k
Featured
See All Featured
brettharned
93
3k
morganepeng
18
1.2k
erikaheidi
14
4.3k
tmm1
61
9.4k
rmw
11
810
destraynor
146
19k
jensimmons
207
10k
malarkey
392
61k
kneath
219
15k
carmenhchung
31
1.5k
chriscoyier
779
240k
eitanlees
112
10k
Transcript
&YQMPSJOHUIF-JNJUTPG5SBOTGFS-FBSOJOH XJUIB6OJGJFE5FYUUP5FYU5SBOTGPSNFS ߅ള .-3FTFBSDI4DJFOUJTU 1JOHQPOH
0WFSWJFX 0WFSWJFX t&YQMPSJOHUIF-JNJUTPG5SBOTGFS-FBSOJOHXJUIB6OJGJFE5FYUUP5FYU5SBOTGPSNFSu 5ۄחܴਵ۽ਬݺೣ 3BGGFMFUBM (PPHMF ֙ਘੌী"SYJWӝ
ݾର ݾର *OUSPEVDUJPO 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS &YQFSJNFOUT 1VUUJOH*U"MM5PHFUIFS
3FGMFDUJPO
*OUSPEVDUJPO *OUSPEVDUJPO
*OUSPEVDUJPO 5SBOTGFS-FBSOJOHJO/-1 • ࠺بणীӝ߈ೠण USBOTGFSMFBSOJOH /-1ীCSFBLUISPVHIܳઁदೣ • ݽ؛ੌࠗী೧8PSE7FD (MPWF 'BTU5FYU
• ݽ؛ী೧6-.'JU #&35 (15 &-&$53"j • ݢۨ࠶হחষؘఠ۽যীೠੌ߈ੋध LOPXMFEHF ܳणೞҊ QSFUSBJOJOH ڙڙ೧ݽ؛ਸۨ࠶חݾUBTLী೧ള۲ೣ GJOFUVOJOH • ৵ੜغחѪੌө • Ѣ ޖೠೠੋఠ֔ࣁ࢚ࣘVTFSHFOFSBUFEUFYU 3FEEJU 8JLJ $PNNPO$SBXMj • ೱ࢚ػDPNQVUBUJPOQPXFS .PPSFsTMBX • ೱ࢚ػࣘب જࢿמਸਤೠݽ؛ҳઑઁউ 5SBOTGPSNFS • ӝઓ3//҅ৌਬী࠺೧эনਸणೡٸ؊ڙڙೞҊ эदрزউ؊݆নਸणೣ
1SPCMFN5IFSBQJESBUFPGQSPHSFTT *OUSPEVDUJPO • 'JFMEߊҗೣԋ ࢎणਸਤೠ݆֤ࣻޙٜݒੌ١ೣ • അઓೞחNFUIPETܳGPMMPXVQೞחѪઑରऔঋ • ппNFUIPEооDPOUSJCVUJPOਸڍয֬Ҋ࠺Үೞӝয۰ •
पਸా೧Ѩૐೞݶغঋਸө • ѐੋࠗೞӝীחࢎणਸਤೠDPTUоցޖఀ • ೞաࢎणNFUIPEо݃աনೠUBTLী೧જपೞӝয۰ • ࢎѾਸਤೠपਸݒߣೡࣻহݶ न܉ೡ݅ೠSFGFSFODFਃࢿழ
4PMVUJPO6OJGJFE'SBNFXPSL $PPSEJOBUF%FTDFOU *OUSPEVDUJPO • ݢনೠUBTLܳࣻ೯ೡࣻחCBTFMJOFGSBNFXPSLٜܳ݅Ҋ 6OJGJFE'SBNFXPSL • ৈ۞/-1UBTLܳUFYUUPUFYUഋక۽ജೣ •
ೞաݽ؛ਸݽٚUBTLীࢎਊೞҊ नݽ؛ੑ۱ਸా೧UBTLܳݺदೣ • ࣇਸೞաঀ߄Լࠁݶࢲ ݽ؛ࢿמযڌѱ߄Շחঌইࠁ $PPSEJOBUFEFTFDFOU • ݽ؛ҳઑ &OD%FDWT%FDPOMZ ݾೣࣻ ؘఠࣇ 4DBMJOH١ • അपਵ۽ݽٚ҃ࣻܳઑ೧ࢲೡࣻחহਵפӬ ࠺ҮझாੌࣇࠗఠೞաঀѾೞݶࢲ೯
*OUSPEVDUJPO
$POUSJCVUJPO *OUSPEVDUJPO • ৈ۞/-1UBTLܳࣻ೯ೡࣻחೞա6OJGJFE'SBNFXPSLܳઁद • ৈ۞ࢎणࣇٜпUBTLࢿמীחೱਸೞաೞաઑࢎೞҊҕਬ • оજࣇٜ݅ݽইࢲपѐѐUBTLী೧405" • ӏݽؘఠࣇ
$ ߂णݽ؛ҕѐ
55FYUUP5FYU5SBOTGFS 5SBOTGPSNFS 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS
5ݽ؛ҳઑ 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS • ୭ઙੋݽ؛ҳઑחपਸా೧PQUJNBMೠࣇਸࢎਊפ ইউঌ۰ષ • ݾೣࣻ DPSSVQUFESBUF١بই •
пपਸਤ೧ࢲחCBTFMJOFݽ؛ &OD%FD ਸࢎਊೞѢա ীࢶఖೠPQUJNBMPQUJPOਸࢎਊפ • ೞ݅ҕాਵ۽5FYUܳੑ۱߉Ҋ۱ೡࣻযঠפ • ઁա5SBOTGPSNFSӝ߈ҳઑܳࢎਊפ • &ODPEFS%FDPEFS %FDPEFSPOMZ 1SFGJY-.
ࢎणಌझ 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS • $PMPTTBM$MFBO$PNNPO$SBXM $ ಌझܳ࢜܂ѱઁद • )FVSJTUJDGJMUFSJOHਸా೧ӝઓ$PNNPO$SBXMоޙઁਸ৮ചೣ • 0GGFOTJWFMBOHVBHF
OPOFOHMJTI +BWBTDSJQUҙ۲ޙࢲઁѢ١ • 8FCFYUSBDUFEؘఠܳ୶о • ୨(#
%PXOTUSFBN5BTLT 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS • /-6ಣо#FODINBSLੋ(-6& TVQFS(-6& • 4FOUFODFQBJSTJNJMBSJUZ 454C 2" .VMUJ3$
/-* $PSFGFSFODFSFTPMVUJPO 8/-* ١ • "CTUSBDUJWF4VNNBSJ[BUJPO • $//%BJMZ.BJM • .BDIJOF5SBOTMBUJPO • &OHMJTI(FSNBO 'SFODI 3PNBOJBO • 2VFTUJPO"OTXFSJOH • 42V"%
*OQVUBOE0VUQVU'PSNBU 55FYUUP5FYU5SBOTGFS5SBOTGPSNFS • 5BTLTQFDJGJDTFRVFODFܳ*OQVUTFRVFODFখীࠢৈࢲݽ؛ীѱUBTLܳݺदೣ • tযীࢲೠҴয۽ߣ*UIJOLZPVutաחցܳࢤп೧u • $MBTTJGJDBUJPO • ޙઁܳ*OQVUਵ۽Ҋ
ӒীೠMBCFMਸPVUQVUীࢲHSFFEZೞѱ٣٬ೞب۾פ • tTUTCTFOUFODFয়ט j TFOUFODFযઁ j utu • 3FHSFTTJPOҗэ۱GMPBUৈب ೞաషਵ۽ܻ • 'JOFUVOJOHغҊաݶ UBTLীݏঋחష٣٬غח҃חহҊפ • (FOFSBUJPO • ੌ߈ੋࢤࢿݽ؛ۢള۲פ
&YQFSJNFOUT &YQFSJNFOUT
3FDBQ֤ޙݾ &YQFSJNFOUT • 5SBOTGPSNFS ӝ߈ ݽ؛ ण 5SBOTGFS -FBSOJOH
җ ҙ۲ػ ৈ۞ పץ ࢎ FNQJSJDBMTVSWFZ • (-6& 42V"%э/-6UBTL ߣ ਃড١/-(UBTL۽ಣоפ ੌ࢚ചח9 • पೞחղਊٜ • ݽ؛ҳઑ ݾೣࣻ • ؘఠࣇӝ߂ઙܨ • ള۲ߑߨ ബਯੋ4DBMJOHVQ • ݽٚઑਸ೧ࠅࣻחহਵפӬ ѢࠗఠೞաঀपೞҊઁੌજ২࣌ਸࢶఖೣ
&YQFSJNFOUT • &OD%FDܻ࠙ػҳઑੑ۱షՙܻחনߑೱBUUFOUJPO٣؊חੋ؊۱җUJNFTUFQషীBUUFOUJPO • -.ੑ۱द௫झܳೞա۽োѾ೧ࢲೞա٣؊ী֍Ҋ ઁաUJNFTUFQషী݅BUUFOUJPO • 1SFGJY-.ੑ۱द௫झܳনߑೱਵ۽ࠁޅೞח-BOHVBHF.PEFMױਸࠁ৮ೞӝਤ೧ઁউػߑߨҳઑח -BOHVBHF.PEFMҗزੌೞ݅ BUUFOUJPONBTLܳా೧ੑ۱షՙܻחࢲ۽ܳࠅࣻ
"ݽ؛ҳઑ
&YQFSJNFOUT • %FOPJTJOH.BTLੌࠗܳാࣚ DPSSVQU दఅद௫झܳੑ۱ਵ۽֍Ҋ ാࣚػࠗ࠙ਸࠂਗ • -.द௫झղషٜਸೞաঀ֍ਵݶࢲ UJNFTUFQױযܳஏ "0CKFDUJWF%FOPJTJOHWT-.
&YQFSJNFOUT • 1SFUSBJOJOH?TUFQ ?UPLFOTFR ?TFRCBUDI?UPLFO # • #&35 #
ա3P#&35B 5 ࠁؘఠࣇ • 'JOFUVOJOHݽٚUBTLܳ?TUFQزউള۲ೞҊ оࢿמજನੋࢎਊ • ޙઁҕೠಣоܳਤ೧ࢲחݽ؛ӝ 1 ৬҅ . زੌ೧ঠೣ • &OD%FDо 1 . ੌ҃-BOHVBHF.PEFM 1 . 1 . ؾפ &OD%FDоܻ࠙غয • فѐ࠺ҮೞҊ &OD%FDрۄఠܳTIBSFೞחҳઑبೣԋ࠺Үೣ "5SBJOJOH%FUBJMT
"3FTVMUT &YQFSJNFOUT • %FOPJTJOHPCKFDUJWFо-.PCKFDUJWFࠁજࢿמਸࠁ • &OD%FD1SFGJY-.-BOHVBHF.PEFMࣽਵ۽જࢿמਸࠁ • ੑ۱द௫झܳনߑೱীࢲࠁחѱਃೞঋਸөࢤпؽ • &OD
%FDрXFJHIUܳTIBSFೞחݽ؛ب࠙֫ࢿמਸࠁ • ੑ۱۱ҙ҅ܳFYQMJDJUೞѱܻ࠙ೞחѪਃೞঋਸөࢤпؽ
"%FOPJTJOH0CKFDUJWF୶оप &YQFSJNFOUT • %FOPJTJOHPCKFDUJWFо-.PCKFDUJWFࠁજҊоೞҊ ҙ۲WBSJBUJPOٜਸ୶оपפ • ৽ଃ #&35ۢੌࠗషਸ݃झఊೠܳࠂਗೞחPCKFDUJWFо-.աEFTIVGGMJOHࠁ؊֫ࢿמ • য়ܲଃ
#&35TUZMFࣁࠗ২࣌ FH TQBOױਤWTషױਤ DPSSVQUJPO࠺ਯ١ ী೧प೧ࠁও݅ ٜࢎਬೞѱରחաఋաঋ
#ࢎള۲ؘఠࣇ &YQFSJNFOUT • ࢎणਊಌझઙܨীٮܲEPXOTUSFBNUBTLࢿמ߸ചܳ࠙ࢳפ • पೞחಌझ • $ • $VGJMUFSFEIFVSJTUJDఠ݂ਸೞঋಌझ
• 3FBM/FXTMJLFझಌझ • 8FC5FYUMJLFۨ٪ӝ߈ಌझ • 8JLJQFEJBৡۄੋߔҗࢎಌझ • 8JLJQFEJB 5PSPOUP#PPL$PSQVT8JLJQFEJBTQFDJGJDೠౠࢿਸ৮ചೞӝਤ೧
#पѾҗ &YQFSJNFOUT • ఠ݂ೠ$ಌझоજࢿמਸࠁ • ࢎणಌझ৬UBTLبݫੋ࠺तೞݶ؊જࢿמਸࠁ • 8JLJQFEJBӝ߈ݽ؛42V"%ীࢲࢿמજҊ (-6&UBTLੌࠗب࠺तೠ҃ೱਸࠁ •
۞ೠഅ࢚ਸѐࢲ۽ܲҙীࢲࠅࣻणפ • *OEPNBJOؘఠীࢲࢎणਸೞחѪࢿמೱ࢚ীبػ • पઁ۽֤ޙٜਸࠁݶ UBTLী߄۽GJOFUVOJOHೞӝীѦ۽ೠߣQSFUSBJOJOHೞݶࢿמೱ࢚ػҊ פ • ࢜۽EPNBJOযਸٸ ৈӝীࡅܰѱ BEBQU ೞחੌ߈ചػࢎणݽ؛ള۲
#ؘఠࣇӝ &YQFSJNFOUT • ࢎणؘఠࣇӝоઁೠػ࢚ടীࢲ ܳSFQFBUೞח҃ী೧पפ • ؘఠ۽FQPDIWTؘఠ۽FQPDIWTOؘఠ۽OFQPDI • ؘఠࣇযٜݶӒ݅ఀSFQFBU೧ࢲ TUFQࣻחزੌೞѱ࠺Үפ
#पѾҗ &YQFSJNFOUT • ؘఠܳೠߣࠁחѱоࢿמજणפ • ੌࣻળөחৈ۞ߣࠁইبࢿמೞۅѢহणפ • ߣ ࢚ ߈ࠂೞݶ
ݽ؛ ള۲ ؘఠࣇ ܳ ৻ਕߡ۰ࢲ 5SBJOJOH -PTTח যٜҊ UBTLQFSGPSNBODFחәхפ য়ߡೖ
$'JOFUVOJOH4USBUFHZ &YQFSJNFOUT • ࢎण৮ܐػ࢚ടীࢲ पઁUBTLীযڌѱGJOFUVOJOHೞחѪજਸө • оߑߨۿਸ࠺Үפ • "MMQBSBNFUFSTݽ؛ݽٚQBSBNFUFSܳGJOFUVOJOH
• "EBQUFS MBZFST ࢎ णػ 5SBOTGPSNFSח GSFF[FೞҊ ݽ؛ ղࠗ GFFEGPSXBSE MBZFS ী <E@NPEFM E@BEBQUFS>3F-6<E@BEBQUFS E@NPEFM>ഋక'FFE'PSXBSEMBZFSܳ୶оೠٍѪ݅ णפ • (SBEVBM6OGSFF[JOHо࢚ਤۨযࠗఠର۹۽VOGSFF[Fפ֫ۨযੌࣻ۾t؏uੌ߈ੋ যधਸߓחоীࢲߊפ
$'JOFUVOJOH4USBUFHZ &YQFSJNFOUT • ݽ؛ܳ߄۽GJOFUVOJOHೞחѱоࢿמજणפ • ݅ࣘبஏݶীࢲחݽ؛ܳUVOJOHೞחѪજঋणפ • "EBQUFSMBZFS৬ҙ۲೧ࢲ MPXSFTPVSDFUBTL FH
42V"% חMBZFSରਗইبغ݅IJHISFTPVSDF UBTL FH .5 חѱਵݶGJOFUVOJOHࢿמഅڄযפ
%4DBMJOH &YQFSJNFOUT • ࢿמਸৢܾࣻחऔҊഛपೠߑߨೞաח ؊ݽ؛ਸ؊ؘ݆ఠ۽؊য়ۖزউणೞחѪੑפ • ೞ݅$PNQVUBUJPOSFTPVSDFחೠغযਵ۽ ৈ۞GBDUPSযוѪਸ݃աఃѾ೧ঠפ • ৬ҙ۲೧֤ޙীࢲ؍חޙҗэणפtܻоॶࣻח҅ߓ۽טযլݶ
ܳযڌѱ ࢎਊ೧ঠೡө u • ޙী ೧ ࠁ ੌ߈ചػ ਸ ҳೞҊ ೠ ֤ޙ ೞա۽ॄ ী ৢۄৡ 4DBMJOH MBXT GPS OFVSBM MBOHVBHFNPEFMTu ,BQMBO +BSFE FUBMणפ
%3FTVMU &YQFSJNFOUT • טযդ҅ਸݽ؛ӝীࠗైೞѢա ള۲झయࣻ৬زੌೞѱա־חѱоજࢿמਸࠁੑפ • ݽ؛Ӓ۽فҊߓӝաള۲झయࣻܳטܻחѪبযוبࢿמೱ࢚ਸࠁੑפ • لؘఠࣇӝܳߓ݅ఀטܻחബҗоणפ •
ৈӝࢲೡ ೞաGBDUPSܳఃٸܲGBDUPSٜ۞ೠ࢚थীCPUUMFOFDLغঋਸب۽ח ࠙ழঠೠחੑפ • ࢎणݽ؛'SFF[FೞҊGJOFUVOJOH݅ৈ۞ߣ೧ࢲFOTFNCMFೞחѪبডрࢿמೱ࢚णפ
1VUUJOHJU"MM5PHFUIFS 1VUUJOHJU"MM5PHFUIFS
1VUUJOHJU"MM5PHFUIFS 5$POGJHVSBUJPO • Әөજ২ٜ࣌ਸӝ߈ਵ۽ݽ؛ਸणפ • ੋ؊٣؊ҳઑ • %FOPJTJOHPCKFDUJWFషઁѢಣӐӡTQBOਸ݃झఊ • UPLFOTFR
?TFRCBUDI .TUFQUSJMMJPOష • ѐݽ؛ࢎૉ • #BTF. • 4NBMM. • -BSHF. • &YUSFNFMBSHF# # • *OEJWJEVBM .VMUJUBTL۽GJOFUVOJOHೞҊ ઁੌજࢿמਸSFQPSUפ
ઁݾఫझ 3FTVMUT • 24ѐ 17ѐ taskী ೧ SOTA •
GLUE, superGLUEী ೧ о ֫ ࢿמ • ߣ taskীࢲח ࢚ਵ۽ ծ ࢿמ • Monolingual ࢎण ؘఠ + Scaling ઑਵ۽ח ই ֫ ࢿמਸ ࠁ ޅೣ.
3FGMFDUJPO 3FGMFDUJPO
3FGMFDUJPO 5BLFXBZT • 5BTLTQFDJGJDೞѱࠂೠݽ؛ҳઑহب TDBMFழݶજࢿמਸࠁ • &OD%FDҳઑоഅࣁীࢲоજࢿמ • زੌೠۨয%FDPOMZী࠺೧ߓݽ؛ࢎૉܳо݅ ࠺तೠ$PNQVUBUJPODPTUܳо
• &OD%FDрਝܳҕਬೞݶ ݽ؛ࢎૉо۽ݶࢲࢿמೞۅبঋ • %FOPJTJOHPCKFDUJWFо-.ী࠺೧જࢿמ • ࢿמ݅Ҋ۰ೞݶ ݽ؛ܳೠߣীGJOFUVOJOHೞחѱо֫ࢿמਸࠁ • ୶оੋDPNQVUBUJPOоמೡٸ tݽ؛ ؘఠࣇutݽ؛ ؘఠࣇuࠁ જࢿמਸࠁ
3FGMFDUJPO 0VUMPPL • ࢎपݽ؛ݶ؊ੜػחѪ־ҳաࢤпೡࣻೞ݅ѱࠛоמೞݶ • -PXSFTPVSDFMBOHVBHF ݽ؛ਸणೞѢաࢎਊೡࣻহਸٸ١١j •
%JTUJMMBUJPO 1BSBNFUFS4IBSJOH 1SVOJOHীೠোҳо҅ࣘਃೣ • ࠁബਯੋLOPXMFEHFFYUSBDUJPO • ઑѐۄఠܳࢎਊೞঋইبযীೠੌ߈ੋधਸੜߓࣻחߑߨਃೞ • അEFOPJTJOHCBTFEPCKFDUJWF࢚ޖоj • &-&$53"оೞաറࠁ۽ॄәؽ
3FGMFDUJPO 0VUMPPL • 5BSHFUUBTLޙٜਸࢎणಌझীࢴযࢲള۲ೞݶࢿמೱ࢚ؽ • *OEPNBJOఫझܻܳ݅աחബҗܳӝೡࣻ • %POsU4UPQ1SFUSBJOJOH"EBQU-BOHVBHF.PEFMTUP%PNBJOTBOE5BTLT "$-
ীࢲࠁੜ࠙ ࢳ • بݫੋ UBTLрਬࢎبীӝ߈ೠࢿמೱ࢚بӝೡࣻ • -BOHVBHFBHOPTUJDNPEFMT • যؘఠ۽݅ࢎण೧ࢲӒ۠ ߣUBTLীࢲ405"ܳب׳ೞޅ೮ • &OHMJTI/BUVSBM-BOHVBHFੋؘ ੋোҳزೱӒۧঋ • যীޖҙೠױযਸ݅٘חѪب൜۽ઁ
ӝఋ • #BTFMJOFQFSGPSNBODF • .VMUJUBTL-FBSOJOH • ࢎणदীUBTLؘఠبэॳݶࢿמೱ࢚ؽ • ؘఠࣇӝоܲৈ۞UBTLܳೠߣীणೠݶ пUBTLࢠ࠺ਯਸযڌѱઑೡ
ܖޅೠղਊٜ
хࢎפ✌ ୶оޙژחҾӘೠݶઁٚইېোۅ۽োۅࣁਃ ߅ള .-3FTFBSDI4DJFOUJTU 1JOHQPOH &NBJMDIBFIVO!TDBUUFSMBCDPLS