Pruning Basics on Multi Head Attention-based Models

1SVOJOH#BTJDTPO .VUJ)FBE"UUFOUJPOCBTFE.PEFMT ഘथജ .-4PGUXBSF&OHJOFFS 1JOHQPOH

ݾର ݾର "SF4JYUFFO)FBET3FBMMZ#FUUFSUIBO0OF <.JDIFMFUBM> "OBMZ[JOH.VMUJ)FBE4FMG"UUFOUJPO 4QFDJBMJ[FE)FBET%PUIF)FBWZ-JGUJOH UIF3FTU$BO#F1SVOFE
<7PJUBFUBM> 3FEVDJOH5SBOTGPSNFS%FQUIPO%FNBOEXJUI4USVDUVSFE%SPQPVU <'BOFUBM> 2

"SF4JYUFFO)FBET 3FBMMZ#FUUFSUIBO0OF 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CTUSBDU • ୭Ӕ5SBOTGPSNFSCBTFE.PEFMٜী௾৔ೱਸ઱Ҋ੓ח.VMUJIFBE"UUFOUJPO • ৈ۞ѐ੄)FBEٜ۽೟णೠറ*OGFSFODF5JNFী؀ࠗ࠙੄)FBEٜਸহগࠆ • ௾ࢿמೞۅহ঻਺ ݻѐ੄-BZFSٜ਷ই৘ೠѐ۽઴ৈبҡଳও׮

"SF"MM"UUFOUJPO)FBET*NQPSUBOU • ѐঀ݅.BTLJOHೞחप೷ ѐࡐҊ׮.BTLJOHೞחप೷ਸ૓೯ೣ • فѐ੄ݽ؛ࢎਊ • 8.5 • 7BTXBOJFUBM੄-BSHF5SBOTGPSNFS
-BZFS )FBETQFS-BZFS • 0UUFUBM GBJSTFR 1SFUSBJOFE.PEFM newstest2013#-&64DPSF • ѐ੄"UUFOUJPO.FDIBOJTNࢎਊ &OD&OD &OD%FD %FD%FD • #&35 • %FWMJOFUBM -BZFS "UUFOUJPO)FBET • #BTF .VMUJ/-*.BUDIFE%BUBTFU'JOFUVOJOH೧ࢲࢎਊ "DDVSBDZӝળ࠺Ү • ѐ੄"UUFOUJPO.FDIBOJTNࢎਊ п-BZFS߹TFMGBUUFOUJPO 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

"SF"MM"UUFOUJPO)FBET*NQPSUBOU • ח഑਷ • ౠ੿)FBEܳ.BTLJOHೞӝਤ೧೧׼чਸઑ੘೧ࢲप೷ೣ • чਸ੹ࠗ۽ݏ୶ݶੌ߈੸ੋ.VMUJIFBE"UUFOUJPOҗэ਺ ξh 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

"CMBUJOH0OF)FBE ਗࠄ4DPSFܳ೧஖૑ঋחࢶীࢲ؀ࠗ࠙੄"UUFOUJPO)FBEٜ੉ࢤۚоמ೮਺ 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

"CMBUJOH0OF)FBE ୨ѐ઺য়૒ѐ݅4UBUJTUJDBMMZ4JHOJGJDBOU೮਺ ब૑যӒ઺ѐח4DPSFоৢۋ਺ 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

"CMBUJOH0OF)FBE "UUFTUUJNF NPTUIFBETBSFSFEVOEBOU HJWFOUIFSFTUPGUIFNPEFM 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

"CMBUJOH"MM)FBETCVU0OF 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

• पઁ۽о੢*NQPSUBOUೠ)FBEೞա݅ਵ۽୽࠙ೠࢿמ੉ա৳਺ • $SPTTUBTL&WBMVBUJPOীࢲب࠺तೠѾҗܳࠁ੐ • ҭ੢൤Ҋޖ੸ੋѾҗ • ੹୓1BSBNFUFS੄ ഑਷ ࠙੄݅ࢎਊ೧ࢲࢿמ੉ࠁઓؽ
• ೞ૑݅ݻѐ੄-BZFSীࢲח.VMUJIFBEо೙ਃ೮਺ • &ODPEFS%FDPEFS"UUFOUJPOীࢲೞաो؊פڄয૗ "CMBUJOH"MM)FBETCVU0OF 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • ૑Әө૑੄ղਊ਷ౠ੿5FTU4FUী੄ઓػഋక੄प೷Ѿҗ • ׮ܲ%BUBTFUী؀ೠੌ߈ചооמೠо • )FBEо6OJWFSTBMMZ*NQPSUBOUೠഋక۽ઓ੤ೞחо •
৮੹൤؀ઑ੸ੋ৉ೡਸೞח0VUPGEPNBJO5FTU4VJUF۽ппप೷ೣ • 8.5ী.5/5&OHMJTIUP'SFODI5FTU4FU • #&35ী./-*.JTNBUDIFE%BUBTFU7BMJEBUJPO4FU "SF*NQPSUBOU)FBETUIF4BNF"DSPTT%BUBTFUT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "SF*NQPSUBOU)FBETUIF4BNF"DSPTT%BUBTFUT ঴୶࠺तೠਤ஖੄࠙ನ ೠ%PNBJOীࢲ઺ਃೠ)FBEо ׮ܲ%PNBJOীࢲب6OJWFSTBMMZ઺ਃ೮਺

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • ૑Әө૑ח-BZFS݃׮ٮ۽ٮ۽प೷೮਺ • ೠߣীৈ۞-BZFSীࢲ)FBEܳ.BTLJOHೞחप೷૓೯ • )FBE*NQPSUBODF4DPSFܳ҅࢑೧ࢲӒчਵ۽੹୓)FBEܳ੿۳ • )FBEܳೞաঀহগоݶࢲࢿמਸஏ੿ೣ *UFSBUJWF1SVOJOHPG"UUFOUJPO)FBET
)FBE*NQPSUBODF4DPSF ౠ੿)FBEо-PTTী৔ೱਸՙ஖ח੿بܳӝળਵ۽҅࢑

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT &GGFDUPG1SVOJOHPO#-&6"DDVSBDZ ੹୓੸ਵ۽ࠌਸٸ*OEJWJEVBM0SBDMFࠁ׮؊જ਷ࢿמਸࠁ৓਺

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT &GGFDUPG1SVOJOHPO&GGJDJFODZ 1SVOFE.PEFM੄*OGFSFODF4QFFEо#BUDI4J[Fоழ૕ࣻ۾ࡈۄ઎਺ .FNPSZDPOTUSBJOUFE4FUUJOHীࢲ੄%FQMPZNFOUীӝৈ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • ইө)FBEܳೠѐ݅Ҏۋਸٸц੘झۣѱࢿמ੉%SPQغח$BTFо੓঻਺ • যڃ҃਋ী੉۞ೠഅ࢚੉ੌযաח૑प೷ • )FBE*NQPSUBODF4DPSFীٮۄ)FBEܳ૒੹प೷җڙэ੉ੜۄն • 8.5੄о૑"UUFOUJPOߑधী؀೧ࢲппࣻ೯ೞҊ࠺Үೣ 8IFO"SF.PSF)FBET*NQPSUBOU
5IF$BTFPG.BDIJOF5SBOTMBUJPO

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 8IFO"SF.PSF)FBET*NQPSUBOU 5IF$BTFPG.BDIJOF5SBOTMBUJPO ਬդ൤&OD%FDߑध݅_ҳрীࢲәѺೞѱ#-&64DPSFоڄয૗

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • 5SBJOJOHҗ੿ীࢲ)FBE*NQPSUBODFоযڌѱ਑૒੉ח૑ঌӝਤೠप೷ਸ૓೯ • 1SVOJOHਸ૓೯ೡٸ4DPSFоযڌѱڄয૑ח૑ܳп&QPDIীࢲஏ੿ೣ • _1SVOJOH೮ਸٸпп#-&6оযڌѱ߸ചೞח૑ܳ3FQPSU • ੷੗о૒੽݅ٚ੘਷ߡ੹੄8.5ݽ؛ਸ೟ण೧ࠆ •
-BZFS п-BZFS׼ѐ੄)FBE • *84-5%BUBTFUࢎਊ %ZOBNJDTPG)FBE*NQPSUBODFEVSJOH5SBJOJOH

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT %ZOBNJDTPG)FBE*NQPSUBODFEVSJOH5SBJOJOH _&QPDIद੼ীࢲח1SVOJOH੿بীٮۄࢿמ੉-JOFBSೞѱхࣗ ઺ਃೠ)FBEחള۲ୡӝ ૒റחইפҊઑӘ૑աࢲ ীѾ੿ؽ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • .VMUJIFBE"UUFOUJPO਷੉ۿ੸ਵ۽಴അࢿ੉ڪযա૑݅ ׮ࢎਊೞ૑ঋח׮חࢎपਸߊѼ • ࢿמਸ௼ѱ੷ೞदః૑ঋਵݶࢲ)FBEܳ׮ࣻহচࣻ੓਺ਸࠁ੐ • .5࠙ঠীࢲ &ODPEFS%FDPEFSݽ؛੉.VMUJIFBEী݆੉੄ઓೣਸߊѼೣ •
п)FBE੄࢚؀੸઺ਃࢿ਷ള۲ୡӝীҙ଴ؽ $PODMVTJPO

"OBMZ[JOH.VMUJ)FBE4FMG"UUFOUJPO 4QFDJBMJ[FE)FBET%PUIF)FBWZ-JGUJOH UIF3FTU$BO#F1SVOFE 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CTUSBDU • .VMUJIFBE"UUFOUJPOীࢲп)FBE੄઺ਃࢿҗ৉ೡী؀೧োҳ • ઺ਃೠ)FBEח$POTJTUFOUೞҊ-JOHVJTUJDBMMZJOUFSQSFUBCMFೠౠࢿਸ૑שਸߊѼ • 4UPDIBTUJD(BUF৬ QFOBMUZܳ੉ਊೠ%JGGFSFOUJBCMF3FMBYBUJPOਸࢎਊೞৈ 1SVOJOH೮Ҋ
઺ਃೠ)FBEٜ੉о੢ա઺ী1SVOJOHؽ • ؀ࠗ࠙੄)FBEܳহগҊبࢿמਸࠁઓೣਸࠁ੐ • &OHMJTIUP3VTTJBO8.5%BUBTFUࢎਊ • )FBEѐ઺ѐզܻҊ#-&6ࣚ೧ࠆ L0

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT *EFOUJGZJOH*NQPSUBOU)FBET • 5SBOTMBUJPOѾҗীৈ۞ߧਤ۽৔ೱਸ޷஖חѐ߹)FBEٜী؀ೠ࠙ࢳ • )FBE੄$POGJEFODFܳ੿੄ • &044ZNCPMਸઁ৻ೠࠗ࠙੄.BYJNVN"UUFOUJPO8FJHIU • $POGJEFOUೠ)FBEח੹୓"UUFOUJPO੄؀ࠗ࠙ਸౠ੿5PLFOী֍חࢿೱਸ૑ש
• 5SBOTMBUJPOী৔ೱਸ݆੉઴Ѫ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFSXJTF3FMFWBODF1SPQBHBUJPO -31 • %JOHFUBM ীࢲ୊਺ઁदؽ • 8FJHIU3BUJPܳഝਊೞৈ/FUXPSL࢚੄ౠ੿/FVSPOٜࢎ੉੄ 3FMBUJWF$POUSJCVUJPOਸ҅࢑ೞחߑߨ •
.PEFMী੄೧৘ஏػ୭࢚ਤ5PQ-PHJUীп)FBEо঴݃աҙৈೞ৓חоܳ ҅࢑ೞחؘী੉.FUIPEܳࢎਊೡѪਸઁदೣ • ੉чਸӝળਵ۽઺ਃبܳಣо

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFSXJTF3FMFWBODF1SPQBHBUJPO -31

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFSXJTF3FMFWBODF1SPQBHBUJPO -31 -31ח_ѐ੄)FBE݅੉઺ਃೞ׮חѾҗܳն $POGJEFODF৬-31חࢲ۽࠺तೠন࢚੄чਸࠁ੐ ୐ߣ૩-BZFSীࢲ݅-31о$POGJEFODF৬׮ܰѱಣоܳೞח੉ਬח

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT $IBSBDUFSJ[JOH)FBET • -31ী੄೧֫਷੼ࣻܳ߉਷)FBEٜਸઑࢎೠѾҗ׮਺੄о૑۽࠙ܨؽ • 1PTJUJPOBM • ੋ੽ೠ5PLFOਸоܻఃח)FBE • 4ZOUBDUJD
• ౠ੿4ZOUBDUJD3FMBUJPOਸо૑ח5PLFOਸоܻఃח)FBE • 3BSF8PSE • -FBTU'SFRVFOU5PLFOਸоܻఃח)FBE

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1PTJUJPOBM)FBET • ੉࢚੄ഛܫ۽੗नҗੋ੽ೠ5PLFOਸоܻఃח)FBEٜ • .BYJNVN"UUFOUJPO8FJHIU੉഑਷ ߣ૩5PLFOীࠗৈؽ • -31߂$POGJEFODFীࢲ֫਷чਸ঳਷ࠗ࠙җэ਷ࠗ࠙ীߓ஖ؽ •
8FJHIU੄оౠ੿5PLFOীନ൧੿ب۽ъبо֫਷҃਋о݆ও਺ ࠁۄ࢝ࠗ࠙

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 4ZOUBDUJD)FBET • 5SBOTMBUJPO5BTL૓೯઺&ODPEFSо4ZOUBDUJD4USVDUVSFܳ೟णೠ׮חоࢸਸઁद • ౠ੿)FBEо4ZOUBDUJDೠ੄޷о੓ח5PLFOਸоܻఆѪэ਺ • ࠙ࢳѾҗ׮਺о૑%FQFOEFODZ3FMBUJPOਸഛੋೣ • /PNJOBM4VCKFDU
OTVCK %JSFDU0CKFDU EPCK "EKFDUJWBM.PEJGJFS BNPE "EWFSCJBM.PEJGJFS BEWNPE ୡ۾࢝ࠗ࠙

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 4ZOUBDUJD)FBETn.FUIPEPMPHZ • $PSF/-1 .BOOJOHFUBM ܳࢎਊ೧ࢲ%FQFOEFODZ4USVDUVSFܳ݅ٞ • "UUFOUJPO8FJHIUਸ୶୹ೠറਤ%FQFOEFODZ4USVDUVSF৬࠺Ү • খীࢲݺदೠޙߨ੸ਃࣗܳоܻఃח%FQFOEFODZо٘۞աח૑ܳഛੋೞҊर਺
• ౠ੿%FQFOEFODZ3FMBUJPO੉ౠ੿%JSFDUJPOਸоܻఃח҃਋ী؀ೠ"DDVSBDZܳ҅࢑ • "DDVSBDZоইېౠ੿ޙߨਃࣗী؀ೠ1PTJUJPOBM3FMBUJPOࠁ׮੉࢚֫ਸ҃਋4ZOUBDUJDೞ׮Ҋ੿੄ пޙߨਃࣗ߹۽1PTJUJPOBM3FMBUJPOਸ࠺Үೣ ਤ஖੸҃ೱࢿژೠэ੉о૑חѪਸഛੋೣ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 4ZOUBDUJD)FBETn3FTVMU • 4ZOUBDUJD3FMBUJPOਸఐ૑ೞח)FBE੄"DDVSBDZо Ӓۧ૑ঋ਷҃਋ࠁ׮֫ও਺ • &ODPEFSо4ZOUBDUJD%JTBNCJHVBUJPOਸࣻ೯ೠ׮ח оࢸਸّ߉ஜೞחѾҗ۽೧ࢳೡࣻ੓਺

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 3BSF8PSET • ݽٚݽ؛ীࢲ ୐-BZFS੄ೠ)FBEоҭ੢൤*NQPSUBODFо֫׮חѾҗܳഛੋ೮਺ • ઑࢎѾҗ -FBTU'SFRVFOUೠ5PLFOਸоܻఃחѪਵ۽ࠁ੐ • חо੢؏١੢ೠ5PLFOਸ
חо੢؏١੢ೠف5PLFO઺ೞաܳоܻௌ਺ ઱ട࢝ࠗ࠙

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOH"UUFOUJPO)FBET • .VMUJIFBE"UUFOUJPOਸ׮਺ҕधਵ۽߄Է • • חп)FBE݃׮׮ܰѱࠗৈغח(BUF • *OQVU4FOUFODF৬ח*OEFQFOEFOUೣ
• п)FBEܳ%PXO8FJHIUJOHೞחѱইפҊই৘%JTBCMFदఃҊर਺ • ী 3FHVMBSJ[BUJPOਸ੸ਊ • .VMUJ)FBE(Q, K, V) = $PODBUi (gi ⋅ )FBEi )WO gi gi L0 L0 (g1 , …, gi ) = h ∑ i=1 (1 − [[gi = 0]])

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOH"UUFOUJPO)FBET • Ӓ۞ա 3FHVMBUJPO਷޷࠙ࠛоמೞ޲۽ 4UPDIBTUJD3FMBYBUJPOਸࢎਊೣ • ܳ3BOEPN7BSJBCMF۽فҊ)FBETQFDJGJD%JTUSJCVUJPOীٮۄة݀੸ਵ۽੿ೣ • )BSE$PODSFUF%JTUSJCVUJPOਸࢎਊೣ
$PODSFUF੄#JOBSZߡ੹ L0 gi LC (ϕ) = h ∑ i=1 (1 − P(gi = 0|ϕi )) L(θ, ϕ) = Lxent (θ, ϕ) + λLC (ϕ) -/PSN੄3FMBYBUJPO L0 (g1 , …, gi ) = h ∑ i=1 (1 − [[gi = 0]]) ୭ઙ-PTTೣࣻ 5SBOTMBUJPO.PEFM੄$SPTT&OUSPQZ 3FHVMBSJ[BUJPO$PFGGJDJFOU թਸ)FBE੄іࣻઑ੺ Lxent λ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 2VBOUJUBUJWF3FTVMUT#-&64DPSF )FBEѐթӝҊ #-&64DPSFхࣗ )FBEѐթӝҊ #-&64DPSFхࣗ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT )FBET*NQPSUBODF • .PEFM਷&ODPEFS4FMG"UUFOUJPOਸઁੌ ݢ੷1SVOJOHೞח҃ೱ੉੓਺ • %FDPEFS4FMG"UUFOUJPO਷ %PNBJOীٮۄ׮ܴ • 8.5੄-POH4FOUFODFח઺ਃب֫਺
• 0QFO4VCUJUMFT੄4IPSU4FOUFODFח ઺ਃبծ਺ • &ODPEFS%FDPEFS"UUFOUJPO਷ 5SBOTMBUJPOী೙ࣻ੸

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT )FBET*NQPSUBODF • %FDPEFS&ODPEFS"UUFOUJPO਷ ծ਷ױ҅-BZFSীࢲ؊ਬ૑ؽ • %FDPEFS4FMG"UUFOUJPO਷ ֫਷ױ҅-BZFSীࢲ؊ਬ૑ؽ 5SBOTGPSNFS੄-BZFSח ծ਷ױ҅ীࢲ-BOHVBHF.PEFMJOHਸೞҊ
֫਷ױ҅ীࢲ4PVSDF4FOUFODF$POEJUJPOJOHਸೣ

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT $PODMVTJPO • -BZFSXJTF3FMFWBODF1SPQBHBUJPOਸࢎਊ೧ࢲ)FBE੄઺ਃࢿਸಣо • 5SBOTMBUJPO5BTLীࢲ)FBEٜ੄4VCTFU݅੉઺ਃೞ׮חࢎप੉ҙ଴ؽ • ઺ਃ)FBEח೧ࢳоמೠౠ੿ӝמਸೠ׮חѪਸߊѼ • ઺ਃೞ૑ঋ਷)FBEٜ੉1SVOJOHغযঠೠ׮חѐ֛ਸӝୡ۽1SVOJOH.FUIPEܳઁद
• &ODPEFS4FMG"UUFOUJPOਸਤदೠ؀ࠗ࠙੄)FBEٜ਷ࢿמࣚ೧Ѣ੄হ੉হচࣻ੓਺ਸࠁ੐

3FEVDJOH5SBOTGPSNFS%FQUIPO %FNBOEXJUI4USVDUVSFE%SPQPVU 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CTUSBDU • -BZFS%SPQਸࣗѐೣ • 5SBJOJOH઺3FHVMBSJ[BUJPO&GGFDUܳоೞҊ *OGFSFODF5JNFীബ਎੸ਵ۽1SVOJOH • ௾֎౟ਕ௼ীࢲ'JOFUVOJOHҗ੿হ੉ೞա੄4VCOFUXPSLܳࢶఖ೮ਸٸ ௾ࢿמ%SPQহ੉୶ۿоמೠҳઑܳࠁ੐

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFS%SPQ • Өѱऺੋ5SBOTGPSNFSܳ3FHVMBSJ[FೞҊ 5SBJOJOHਸউ੿ചदఇ • ׮নೠ#FODINBSLীࢲ405"1FSGPSNBODFܳਬ૑ೞب۾ೣ • ೞա੄௾1SFUSBJOFE.PEFMীࢲౠ੿Ө੉ܳо૓੘Ҋബਯ੸ੋݽ؛ਸ୶୹ೣ •
5FTU5JNFী੗زਵ۽੉ܖয૗ • 'JOFUVOJOHҗ੿੉೙ਃೞ૑ঋ਺ • %SPQPVUҗ࠺तೠࣻળ੄ए਍*NQMFNFOUBUJPOդ੉بܳо૗

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CTUSBDU

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 3BOEPNMZ%SPQQJOH4USVDUVSFTBU5SBJOJOH5JNF • п8FJHIUоة݀੸ਵ۽߬ܰ־੉࠙ನ ী੄೧Ѿ੿ػ%SPQ3BUFܳٮܴ • .੉\ ^উীࢲ3BOEPNMZ4BNQMFػ.BTL.FUSJYੌٸ
8FJHIU.BUSJYח p > 0 Wd = M ⊙ W

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 3BOEPN4USVDUVSFE%SPQPVU • э਷Ҋਬҳઑܳо૓$PNQPOFOUীౠ੿8FJHIUٜ੄(SPVQਸഋࢿೞҊ Ӓ(SPVQীٮۄࢲп$PNQPOFOUܳ.BTLJOHೣ • )FBEա''/.FUSJDFT -BZFS৬э੉0WFSMBQQJOHೞחҳઑ۽غয੓ח ݽٚ8FJHIUী੸ਊೡࣻ੓਺

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOHBU*OGFSFODF5JNFn4FMFDUJOH-BZFSTUP1SVOF • 1SVOJOHೡ-BZFSܳҊܰחо૑4USBUFHZ • &WFSZ0UIFS • 3BUF о઱য઎ਸٸ %FQUI
ী؀೧ࢲܳ1SVOJOHೣ • о੢૒ҙ੸੉Ҋ#BMBODFE/FUXPSLٜܳ݅ࣻ੓חߑߨ ݽٚ҃਋ীੜ੘زೣ • 4FBSDIPO7BMJE • -BZFSٜ੄ৈ۞$PNCJOBUJPOਸ҅࢑ೞҊ7BMJEBUJPO4FUਵ۽ппѨૐೣ • 4USBJHIUGPSXBSEೞ૑݅ো࢑۝੉࠺ऱҊ7BMJEBUJPO4FUী0WFSGJUUJOHೡࣻ੓਺ • %BUB%SJWFO1SVOJOH • п-BZFS੄%SPQ3BUF ܳ೟णೞݴ -BZFSٜ੄"WFSBHF3BUFܳ ৬эѱೣ • -BZFS੄"DUJWBUJPOਵ۽ ܳ೟णೞҊ4PGUNBYܳ҅࢑ೞחഋక • *OGFSFODJOH5JNFী4PGUNBY0VUQVU઺5PQLܳࡳইࢲ೧׼-BZFSٜਸࢎਊೣ p d pd p pd d ≡ 0(mod⌊ 1 p ⌋)

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOHBU*OGFSFODF5JNFn4FUUJOHUIFESPQSBUFGPSPQUJNBMQSVOJOH • (SPVQ੉ ѐ੓Ҋ 'JYFE%SPQ3BUJP о੓ਸٸ (SPVQٜ੄ಣӐіࣻח ੉޲۽ ѐ੄(SPVQਸ1SVOJOHೠ׮Ҋೞݶ
0QUJNBM%SPQ3BUF ח • ֫਷1SVOJOH3BUFח੘਷.PEFMীࢲ؊ա਷ࢿמਸࠁ੐ • -BZFS%SPQীࢲח ܳࢎਊ೮ਵա ੘਷.PEFMীࢲח ܳӂ੢ೣ N p N(1 − p) r p* p* = 1 − r N p = 0.2 p = 0.5

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 3FTVMU

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CMBUJPO4UVEJFT

1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT $PODMVTJPO • 4USVDUVSFE%SPQPVU੉3FHVMBSJ[BUJPOীب਑੉݆੉ؽ • ؋࠙ী1SVOJOHद੼ী/FUXPSLо؊3PCVTUೡࣻ੓঻਺ • যڃ%FQUI۽1SVOJOHೞ؊ۄبযו੿ب&GGJDJFOUೞب۾ࣁ౴ೣ • Ө਷/FUXPSLܳ೟णदఆٸب-BZFS%SPQ੉4UBCJMJUZীب਑ਸ݆੉઱঻਺

хࢎ೤פ׮✌ ഘथജ .-4PGUXBSF&OHJOFFS 1JOHQPOH &NBJMTFVOHIXBO!TDBUUFSMBCDPLS

Pruning Basics on Multi Head Attention-based Mo...

Pruning Basics on Multi Head Attention-based Models

More Decks by Scatter Lab Inc.

Other Decks in Research

Featured

Transcript