Optimization as a Model for Few-Shot Learning - ICLR 2017 reading seminar

!%F/" 4IJCVZB)JLBSJF )PLVUP ,BHBZB !@IPLLVO@ 0QUJNJ[BUJPOBTB.PEFM GPS'FX4IPU-FBSOJOH

5-%3 • 1VSQPTF • #FUUFSJOGFSFODFGPSGFXTIPUPOFTIPUMFBSOJOH QSPCMFN • .FUIPE • .FUBMFBSOJOHCBTFEPO-45.PGEFFQOFVSBM
OFUXPSL • 3FTVMU • $PNQFUJUJWFXJUIEFFQNFUSJDMFBSOJOHUFDIOJRVFT

#BDLHSPVOE • 8IZEFFQMFBSOJOHTVDDFFEFE • NBDIJOFQPXFS • BNPVOUPGEBUB • -BSHF%BUBTFU
• *NBHF/FU *NBHF • .JDSPTPGU$0$0$BQUJPOT *NBHF$BQUJPO • :PV5VCF. 7JEFP • 8JLJ5FYU 5FYU

#BDLHSPVOE • )PXFWFS *ONBOZGJFMET UPDPMMFDUBMBSHF BNPVOUPGUSBJOJOHTBNQMFTJT • EJGGJDVMU •
&Y'JOFHSBJOFESFDPHOJUJPO DBS CJSE GPPE • UJNFDPOTVNJOH • TDSBQJOH DSBXMJOH BOOPUBUJOHʜ • "DUVBMMZIVNBOCFJOHTDBOHFOFSBMJ[FVTJOH GFXTBNQMFTPGUBSHFUT

1SPCMFN1VSQPTF • )PXDBOXFBDRVJSFHFOFSBMJ[FENPEFMVTJOH GFXTBNQMFTBOEBTFUOVNCFSPGVQEBUFT • &YJTUFEHSBEJFOUCBTFEUSBJOJOHBMHPSJUIN 4(% "%".
"EB(MBE EPFTOPUGJUUIFQSPCMFNXJUIB TFUOVNCFSPGQBSBNFUFSVQEBUFT • *OPUIFSTJNQMFXPSET BVUIPSTXBOUUPGJOE HPPEJOJUJBMQBSBNFUFSTPG// • DG SFWJFXDPNNFOUTJUJTNVDICFUUFSUPCFBCMFUP GJOEBSDIJUFDUVSBMQBSBNFUFSTPG//

1SPCMFN1VSQPTF • )PX • .FUBMFBSOJOH • -FBSOJOHUPMFBSO5SBJOMFBSOFSJUTFMG • "WBSJFUZPGNFUBMFBSOJOH
• 5SBOTGFSMFBSOJOH • 6TFUIFFYQFSJFODFTPGEJGGFSFOUEPNBJO • 1PQVMBSJOUIFGJFMEPGJNBHFDMBTTJGJDBUJPO FTQFDJBMMZGPS GJOFHSBJOFEWJTVBMDMBTTJGJDBUJPO • &OTFNCMFDMBTTJGJFS • DPNCJOFNVMUJQMFDMBTTJGJFS 5IJTBSUJDMFJTWFSZHPPEUPVOEFSTUBOENFUBMFBSOJOH IUUQIUUQXXXTDIPMBSQFEJBPSHBSUJDMF.FUBMFBSOJOH

1SPQPTFENFUIPE • -45.CBTFENFUBMFBSOJOH

1SFSFRVJTJUFT • 8IBUJT-45. • -POHUJNF4IPSU5FSN.FNPSZ • ࣌ܥྻΛѻ͍͍ͨɺͰ΋ޡ͕ࠩൃࢄফࣦͪ͠Ό͏ • աڈͷσʔλͷॏΈΛ̍ʹͯ͠๨Εͳ͍Α͏ʹ্ͨ͠ Ͱɺબ୒తʹೖྗग़ྗΛߦ͏Α͏ʹͨ͠
b • ͔͠͠ٸܹͳঢ়گͷมԽʢʁʣʹରԠͰ͖ͳ͔ͬͨͷ Ͱɺ๨٫ήʔτΛઃஔ͢Δ͜ͱͰબ୒తʹաڈͷσʔ λͷهԱΛফڈͰ͖ΔΑ͏ʹͨ͠ ` • ࢀߟʢ೔ຊޠʣ • IUUQRJJUBDPNU@4JHOVMMJUFNTCCFCGEC

%BUB4FQBSBUJPO • NFUBUSBJOEBUBTFU • NFUBUFTUEBUBTFU • NFUBTBNQMF UBSHFU USBJOJOH
TBNQMFT UBSHFU UFTUJOH TBNQMFT POFNFUBTBNQMF BLBFQJTPEF

1SPQPTFE.FUIPE " = "$% + )*+, ℒ" " =
" ⨀"$% + " ⨀̅" XIFSF " = (6 7 ？ + b9 ) " = ; 7 ？ + b< XIFSF ？ DVSSFOUHSBEJFOUT DVSSFOUMPTT QSFWJPVT QSFWJPVTJUTFMG , /PSNBM4(% .FUBQIPS OPUDPOTUBOU UP FTDBQFGSPNCBE MPDBMPQUJNB

1SPQPTFE.FUIPE ←meta learner‘s iteration ←learner‘s iteration (meta) loss
value is computed by final state of LSTM (= parameters of target model) and "?@"’s data and labels.

1SPQPTFE.FUIPE • 'SPNBVUIPS`TTMJEF

1SPQPTFE.FUIPE • 8IBUXJMMCFJNQSPWFEHSBEVBMMZ • 'JSTU-45.QBSBNFUFS BLBNFUBMFBSOFS QBSBNFUFST • UIBUJT
zIPXTIPVMEXFVQEBUFUBSHFUNPEFMT z • 4FDPOE-45.TUBUFT PVUQVUT • 'JOBMA JTTIBSFEBNPOHFBDICBUDI TPMFBSOJOHQSPDFFET SBQJEMZUIBOLTGPSHPPEJOJUJBMJ[BUJPO

0UIFS5PQJDT • DPPSEJOBUFXJTF-45. • 1SFQSPDFTTJOHUP-45.JOQVUT • BCPVUCPUIUPQJDT TFF<"OESZDIPXJD[ /*14> QSFQSPDFTTJOHJTJOBQQFOEJY
• BEKVTUUIFTDBMJOHPGHSBEJFOUTBOEMPTTFT • TFQBSBUFJOGPPGNBHOJUVEFBOETJHO • #BUDIOPSNBMJ[BUJPO • BWPJEzEBUBTFUz FQJTPEF MFWFMMFBLBHFPG JOGPSNBUJPO • 3FMBUFEXPSLNFUSJDMFBSOJOH • FY4JBNFTFOFUXPSL

&WBMVBUJPO.FUIPE • #BTFMJOFOFBSFTUOFJHICPS • NFUBUSBJOUSBJOOFVSBMOFUXPSLVTJOHBMMTBNQMF • NFUBUFTUUSBJOJOHTBNQMFΛ//ʹͿͪ͜Μͩ݁Ռͱ UFTUJOHTBNQMFͷͦΕΛൺֱ • #BTFMJOFGJOFUVOF
• NFUBUSBJOʹՃ͑ͯɺ NFUBWBMJEBUJPO EBUBTFU Λ IZQFS QBSBNFUFSͷ୳ࡧʹ࢖͍ɺ ͷ OFUXPSLΛ GJOFUVOF ͢Δ • #BTFMJOF.BUDIJOHOFUXPSL • ڑ཭ֶशͷ405"

&WBMVBUJPO3FTVMU

7JTVBMJ[BUJPOBOE*OTJHIU

7JTVBMJ[BUJPOBOE*OTJHIU • JOQVUHBUFT • EJGGFSFOUBNPOHEBUBTFUT • NFUBMFBSOFSJTO`UTJNQMZMFBSOJOHBGJYFEPQUJNJ[BUJPO TUSBUFHZ • EJGGFSFOUBNPOHUBTLT
• NFUBMFBSOFSIBTVTFEEJGGFSFOUXBZTUPTPMWFFBDI TFUUJOH • GPSHFUHBUFT • TJNQMFEFDBZ • ݁ہ΄ͱΜͲ DPOTUBOU

$PODMVTJPO • 'PVOE-45.CBTFENPEFMUPMFBSOBMFBSOFS XIJDIJTJOTQJSFECZBNFUBQIPSCFUXFFO 4(%VQEBUFTBOE-45. • 5SBJONFUBMFBSOFSUPEJTDPWFS • (PPEJOJUJBMJ[BUJPOPGMFBSOFS
• (PPENFDIBOJTNGPSVQEBUJOHMFBSOFS`T QBSBNFUFST • DPNQFUJUJWFFYQFSJNFOUBMSFTVMUXJUI405" NFUSJDMFBSOJOHNFUIPET

'VUVSFXPSL • GFXTBNQMFTMPUTPGDMBTTFT • NPSFDIBMMFOHJOHTDFOBSJPT • GSPNSFWJFXDPNNFOU • JUJTNVDICFUUFSUPCFBCMFUPGJOEBSDIJUFDUVSBM QBSBNFUFSTPG//

ॴײ • USBOTGFSMFBSOJOHʹ͓͚ΔʮผυϝΠϯͷܦݧ Λ׆͔͢ʯͱ͍͏࡞ۀΛʮ࣌ܥྻͷֶशʯతʹଊ ͑ͯ -45.Ϟσϧͱֶͯ͠शͨ͠ɺͱ͍͏ͷ͸ ࣗવʹࢥ͑ͨ • ͢Ͱʹ͋ͬͨൃ૝ʁ࣌ؒͳؔ͘࿈ݚڀ·ͰಡΈࠐΊ ͣɻɻ
• SFWJFXDPNNFOUʹ͋ͬͨɺߏ଄ͷ࠷దԽ·Ͱ Ͱ͖Δͱ͘͢͝Αͦ͞͏ͩͱࢥͬͨ • γϯϓϧͳϑΟϧλΛͨ͘͞ΜॏͶΔͱ͍͍ͱ͍͏࿩ ΋͋Δ͕ɻɻ • ֶ෦࣌୅ DVEBDPOWOFU Λ࢖ͬͯͨ͘͞ΜϋΠύύϥ ϝʔλΛࢼͨۤ͠࿑͕ોͬͨ

ଟ෼Θ͔ͬͯͳ͍͜ͱ • ݁ہɺ͜ͷ࿦จͰॳΊͯΘ͔ͬͨͷ͸Ͳ͜ʁ -45.Λ MFBSOJOHUPMFBSOʹ࢖ͬͨͷ͸ଟ෼ॳ Ίͯ͡Όͳ͍ʁ • ྫ͑͹ "OESZDIPXJD[ `
Ͱ͸ɺޯ഑Λೖྗʹͯ͠ UBSHFUMFBSOFSͷ QBSBNFUFSVQEBUFT Λग़ྗ͢Δ -45.Λֶश • ύϥϝλͦͷ΋ͷΛ௚઀ग़ྗͯ͠Δͱ͜Ζʁ

Optimization as a Model for Few-Shot Learning - ICLR 2017 reading seminar

Optimization as a Model for Few-Shot Learning - ICLR 2017 reading seminar

Hokuto Kagaya

More Decks by Hokuto Kagaya

Other Decks in Science

Featured

Transcript

!%F/" 4IJCVZB)JLBSJF )PLVUP ,BHBZB !@IPLLVO@ 0QUJNJ[BUJPOBTB.PEFM GPS'FX4IPU-FBSOJOH

5-%3 • 1VSQPTF • #FUUFSJOGFSFODFGPSGFXTIPUPOFTIPUMFBSOJOH QSPCMFN • .FUIPE • .FUBMFBSOJOHCBTFEPO-45.PGEFFQOFVSBM

#BDLHSPVOE • 8IZEFFQMFBSOJOHTVDDFFEFE • NBDIJOFQPXFS • BNPVOUPGEBUB • -BSHF%BUBTFU

#BDLHSPVOE • )PXFWFS *ONBOZGJFMET UPDPMMFDUBMBSHF BNPVOUPGUSBJOJOHTBNQMFTJT • EJGGJDVMU •

1SPCMFN1VSQPTF • )PXDBOXFBDRVJSFHFOFSBMJ[FENPEFMVTJOH GFXTBNQMFTBOEBTFUOVNCFSPGVQEBUFT • &YJTUFEHSBEJFOUCBTFEUSBJOJOHBMHPSJUIN 4(% "%".

1SPCMFN1VSQPTF • )PX • .FUBMFBSOJOH • -FBSOJOHUPMFBSO5SBJOMFBSOFSJUTFMG • "WBSJFUZPGNFUBMFBSOJOH

1SPQPTFENFUIPE • -45.CBTFENFUBMFBSOJOH

1SFSFRVJTJUFT • 8IBUJT-45. • -POHUJNF4IPSU5FSN.FNPSZ • ࣌ܥྻΛѻ͍͍ͨɺͰ΋ޡ͕ࠩൃࢄফࣦͪ͠Ό͏ • աڈͷσʔλͷॏΈΛ̍ʹͯ͠๨Εͳ͍Α͏ʹ্ͨ͠ Ͱɺબ୒తʹೖྗग़ྗΛߦ͏Α͏ʹͨ͠

%BUB4FQBSBUJPO • NFUBUSBJOEBUBTFU • NFUBUFTUEBUBTFU • NFUBTBNQMF UBSHFU USBJOJOH

1SPQPTFE.FUIPE " = "$% + )*+, ℒ" " =

1SPQPTFE.FUIPE ←meta learner‘s iteration ←learner‘s iteration (meta) loss

1SPQPTFE.FUIPE • 'SPNBVUIPS`TTMJEF

1SPQPTFE.FUIPE • 8IBUXJMMCFJNQSPWFEHSBEVBMMZ • 'JSTU-45.QBSBNFUFS BLBNFUBMFBSOFS QBSBNFUFST • UIBUJT

0UIFS5PQJDT • DPPSEJOBUFXJTF-45. • 1SFQSPDFTTJOHUP-45.JOQVUT • BCPVUCPUIUPQJDT TFF<"OESZDIPXJD[ /*14> QSFQSPDFTTJOHJTJOBQQFOEJY

&WBMVBUJPO.FUIPE • #BTFMJOFOFBSFTUOFJHICPS • NFUBUSBJOUSBJOOFVSBMOFUXPSLVTJOHBMMTBNQMF • NFUBUFTUUSBJOJOHTBNQMFΛ//ʹͿͪ͜Μͩ݁Ռͱ UFTUJOHTBNQMFͷͦΕΛൺֱ • #BTFMJOFGJOFUVOF

&WBMVBUJPO3FTVMU

7JTVBMJ[BUJPOBOE*OTJHIU

7JTVBMJ[BUJPOBOE*OTJHIU • JOQVUHBUFT • EJGGFSFOUBNPOHEBUBTFUT • NFUBMFBSOFSJTO`UTJNQMZMFBSOJOHBGJYFEPQUJNJ[BUJPO TUSBUFHZ • EJGGFSFOUBNPOHUBTLT

$PODMVTJPO • 'PVOE-45.CBTFENPEFMUPMFBSOBMFBSOFS XIJDIJTJOTQJSFECZBNFUBQIPSCFUXFFO 4(%VQEBUFTBOE-45. • 5SBJONFUBMFBSOFSUPEJTDPWFS • (PPEJOJUJBMJ[BUJPOPGMFBSOFS

'VUVSFXPSL • GFXTBNQMFTMPUTPGDMBTTFT • NPSFDIBMMFOHJOHTDFOBSJPT • GSPNSFWJFXDPNNFOU • JUJTNVDICFUUFSUPCFBCMFUPGJOEBSDIJUFDUVSBM QBSBNFUFSTPG//

ॴײ • USBOTGFSMFBSOJOHʹ͓͚ΔʮผυϝΠϯͷܦݧ Λ׆͔͢ʯͱ͍͏࡞ۀΛʮ࣌ܥྻͷֶशʯతʹଊ ͑ͯ -45.Ϟσϧͱֶͯ͠शͨ͠ɺͱ͍͏ͷ͸ ࣗવʹࢥ͑ͨ • ͢Ͱʹ͋ͬͨൃ૝ʁ࣌ؒͳؔ͘࿈ݚڀ·ͰಡΈࠐΊ ͣɻɻ

ଟ෼Θ͔ͬͯͳ͍͜ͱ • ݁ہɺ͜ͷ࿦จͰॳΊͯΘ͔ͬͨͷ͸Ͳ͜ʁ -45.Λ MFBSOJOHUPMFBSOʹ࢖ͬͨͷ͸ଟ෼ॳ Ίͯ͡Όͳ͍ʁ • ྫ͑͹ "OESZDIPXJD[ `