Share
Embed
Pruning Basics on Multi Head Attention-based Models
Scatter Lab Inc.
May 18, 2020
Transcript
56 slides
-
ݾର ݾର "SF4JYUFFO)FBET3FBMMZ#FUUFSUIBO0OF <.JDIFMFUBM> "OBMZ[JOH.VMUJ)FBE4FMG"UUFOUJPO 4QFDJBMJ[FE)FBET%PUIF)FBWZ-JGUJOH UIF3FTU$BO#F1SVOFE
<7PJUBFUBM> 3FEVDJOH5SBOTGPSNFS%FQUIPO%FNBOEXJUI4USVDUVSFE%SPQPVU <'BOFUBM> 2 -
"SF"MM"UUFOUJPO)FBET*NQPSUBOU • ѐঀ݅.BTLJOHೞחप ѐࡐҊ.BTLJOHೞחपਸ೯ೣ • فѐݽ؛ࢎਊ • 8.5 • 7BTXBOJFUBM-BSHF5SBOTGPSNFS
-BZFS )FBETQFS-BZFS • 0UUFUBM GBJSTFR 1SFUSBJOFE.PEFM newstest2013#-&64DPSF • ѐ"UUFOUJPO.FDIBOJTNࢎਊ &OD&OD &OD%FD %FD%FD • #&35 • %FWMJOFUBM -BZFS "UUFOUJPO)FBET • #BTF .VMUJ/-*.BUDIFE%BUBTFU'JOFUVOJOH೧ࢲࢎਊ "DDVSBDZӝળ࠺Ү • ѐ"UUFOUJPO.FDIBOJTNࢎਊ п-BZFS߹TFMGBUUFOUJPO 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -
• पઁ۽о*NQPSUBOUೠ)FBEೞա݅ਵ۽࠙ೠࢿמա৳ • $SPTTUBTL&WBMVBUJPOীࢲب࠺तೠѾҗܳࠁ • ҭҊޖੋѾҗ • 1BSBNFUFS ࠙݅ࢎਊ೧ࢲࢿמࠁઓؽ
• ೞ݅ݻѐ-BZFSীࢲח.VMUJIFBEоਃ೮ • &ODPEFS%FDPEFS"UUFOUJPOীࢲೞաो؊פڄয "CMBUJOH"MM)FBETCVU0OF 1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • Әөղਊౠ5FTU4FUীઓػഋకपѾҗ • ܲ%BUBTFUীೠੌ߈ചооמೠо • )FBEо6OJWFSTBMMZ*NQPSUBOUೠഋక۽ઓೞחо •
৮ઑੋೡਸೞח0VUPGEPNBJO5FTU4VJUF۽ппपೣ • 8.5ী.5/5&OHMJTIUP'SFODI5FTU4FU • #&35ী./-*.JTNBUDIFE%BUBTFU7BMJEBUJPO4FU "SF*NQPSUBOU)FBETUIF4BNF"DSPTT%BUBTFUT -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT • 5SBJOJOHҗীࢲ)FBE*NQPSUBODFоযڌѱחঌӝਤೠपਸ೯ • 1SVOJOHਸ೯ೡٸ4DPSFоযڌѱڄযחܳп&QPDIীࢲஏೣ • _1SVOJOH೮ਸٸпп#-&6оযڌѱ߸ചೞחܳ3FQPSU • о݅ٚߡ8.5ݽ؛ਸण೧ࠆ •
-BZFS п-BZFSѐ)FBE • *84-5%BUBTFUࢎਊ %ZOBNJDTPG)FBE*NQPSUBODFEVSJOH5SBJOJOH -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT "CTUSBDU • .VMUJIFBE"UUFOUJPOীࢲп)FBEਃࢿҗೡী೧োҳ • ਃೠ)FBEח$POTJTUFOUೞҊ-JOHVJTUJDBMMZJOUFSQSFUBCMFೠౠࢿਸשਸߊѼ • 4UPDIBTUJD(BUF৬ QFOBMUZܳਊೠ%JGGFSFOUJBCMF3FMBYBUJPOਸࢎਊೞৈ 1SVOJOH೮Ҋ
ਃೠ)FBEٜоաী1SVOJOHؽ • ࠗ࠙)FBEܳহগҊبࢿמਸࠁઓೣਸࠁ • &OHMJTIUP3VTTJBO8.5%BUBTFUࢎਊ • )FBEѐѐզܻҊ#-&6ࣚ೧ࠆ L0 -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFSXJTF3FMFWBODF1SPQBHBUJPO -31 • %JOHFUBM ীࢲઁदؽ • 8FJHIU3BUJPܳഝਊೞৈ/FUXPSL࢚ౠ/FVSPOٜࢎ 3FMBUJWF$POUSJCVUJPOਸ҅ೞחߑߨ •
.PEFMী೧ஏػ୭࢚ਤ5PQ-PHJUীп)FBEо݃աҙৈೞחоܳ ҅ೞחؘী.FUIPEܳࢎਊೡѪਸઁदೣ • чਸӝળਵ۽ਃبܳಣо -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT $IBSBDUFSJ[JOH)FBET • -31ী೧֫ࣻܳ߉)FBEٜਸઑࢎೠѾҗо۽࠙ܨؽ • 1PTJUJPOBM • ੋೠ5PLFOਸоܻఃח)FBE • 4ZOUBDUJD
• ౠ4ZOUBDUJD3FMBUJPOਸоח5PLFOਸоܻఃח)FBE • 3BSF8PSE • -FBTU'SFRVFOU5PLFOਸоܻఃח)FBE -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 4ZOUBDUJD)FBET • 5SBOTMBUJPO5BTL೯&ODPEFSо4ZOUBDUJD4USVDUVSFܳणೠחоࢸਸઁद • ౠ)FBEо4ZOUBDUJDೠоח5PLFOਸоܻఆѪэ • ࠙ࢳѾҗо%FQFOEFODZ3FMBUJPOਸഛੋೣ • /PNJOBM4VCKFDU
OTVCK %JSFDU0CKFDU EPCK "EKFDUJWBM.PEJGJFS BNPE "EWFSCJBM.PEJGJFS BEWNPE ୡ۾࢝ࠗ࠙ -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 4ZOUBDUJD)FBETn.FUIPEPMPHZ • $PSF/-1 .BOOJOHFUBM ܳࢎਊ೧ࢲ%FQFOEFODZ4USVDUVSFܳ݅ٞ • "UUFOUJPO8FJHIUਸ୶ೠറਤ%FQFOEFODZ4USVDUVSF৬࠺Ү • খীࢲݺदೠޙߨਃࣗܳоܻఃח%FQFOEFODZо٘۞աחܳഛੋೞҊर
• ౠ%FQFOEFODZ3FMBUJPOౠ%JSFDUJPOਸоܻఃח҃ীೠ"DDVSBDZܳ҅ • "DDVSBDZоইېౠޙߨਃࣗীೠ1PTJUJPOBM3FMBUJPOࠁ࢚֫ਸ҃4ZOUBDUJDೞҊ пޙߨਃࣗ߹۽1PTJUJPOBM3FMBUJPOਸ࠺Үೣ ਤ҃ೱࢿژೠэоחѪਸഛੋೣ -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOH"UUFOUJPO)FBET • .VMUJIFBE"UUFOUJPOਸҕधਵ۽߄Է • • חп)FBE݃ܰѱࠗৈغח(BUF • *OQVU4FOUFODF৬ח*OEFQFOEFOUೣ
• п)FBEܳ%PXO8FJHIUJOHೞחѱইפҊই%JTBCMFदఃҊर • ী 3FHVMBSJ[BUJPOਸਊ • .VMUJ)FBE(Q, K, V) = $PODBUi (gi ⋅ )FBEi )WO gi gi L0 L0 (g1 , …, gi ) = h ∑ i=1 (1 − [[gi = 0]]) -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOH"UUFOUJPO)FBET • Ӓ۞ա 3FHVMBUJPO࠙ࠛоמೞ۽ 4UPDIBTUJD3FMBYBUJPOਸࢎਊೣ • ܳ3BOEPN7BSJBCMF۽فҊ)FBETQFDJGJD%JTUSJCVUJPOীٮۄة݀ਵ۽ೣ • )BSE$PODSFUF%JTUSJCVUJPOਸࢎਊೣ
$PODSFUF#JOBSZߡ L0 gi LC (ϕ) = h ∑ i=1 (1 − P(gi = 0|ϕi )) L(θ, ϕ) = Lxent (θ, ϕ) + λLC (ϕ) -/PSN3FMBYBUJPO L0 (g1 , …, gi ) = h ∑ i=1 (1 − [[gi = 0]]) ୭ઙ-PTTೣࣻ 5SBOTMBUJPO.PEFM$SPTT&OUSPQZ 3FHVMBSJ[BUJPO$PFGGJDJFOU թਸ)FBEіࣻઑ Lxent λ -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT )FBET*NQPSUBODF • .PEFM&ODPEFS4FMG"UUFOUJPOਸઁੌ ݢ1SVOJOHೞח҃ೱ • %FDPEFS4FMG"UUFOUJPO %PNBJOীٮۄܴ • 8.5-POH4FOUFODFחਃب֫
• 0QFO4VCUJUMFT4IPSU4FOUFODFח ਃبծ • &ODPEFS%FDPEFS"UUFOUJPO 5SBOTMBUJPOীࣻ -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT -BZFS%SPQ • Өѱऺੋ5SBOTGPSNFSܳ3FHVMBSJ[FೞҊ 5SBJOJOHਸউചदఇ • নೠ#FODINBSLীࢲ405"1FSGPSNBODFܳਬೞب۾ೣ • ೞա1SFUSBJOFE.PEFMীࢲౠӨܳоҊബਯੋݽ؛ਸ୶ೣ •
5FTU5JNFীزਵ۽ܖয • 'JOFUVOJOHҗਃೞঋ • %SPQPVUҗ࠺तೠࣻળए*NQMFNFOUBUJPOդبܳо -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOHBU*OGFSFODF5JNFn4FMFDUJOH-BZFSTUP1SVOF • 1SVOJOHೡ-BZFSܳҊܰחо4USBUFHZ • &WFSZ0UIFS • 3BUF оযਸٸ %FQUI
ী೧ࢲܳ1SVOJOHೣ • оҙҊ#BMBODFE/FUXPSLٜܳ݅ࣻחߑߨ ݽٚ҃ীੜزೣ • 4FBSDIPO7BMJE • -BZFSٜৈ۞$PNCJOBUJPOਸ҅ೞҊ7BMJEBUJPO4FUਵ۽ппѨૐೣ • 4USBJHIUGPSXBSEೞ݅ো࠺ऱҊ7BMJEBUJPO4FUী0WFSGJUUJOHೡࣻ • %BUB%SJWFO1SVOJOH • п-BZFS%SPQ3BUF ܳणೞݴ -BZFSٜ"WFSBHF3BUFܳ ৬эѱೣ • -BZFS"DUJWBUJPOਵ۽ ܳणೞҊ4PGUNBYܳ҅ೞחഋక • *OGFSFODJOH5JNFী4PGUNBY0VUQVU5PQLܳࡳইࢲ೧-BZFSٜਸࢎਊೣ p d pd p pd d ≡ 0(mod⌊ 1 p ⌋) -
1SVOJOH#BTJDTPO.VMUJ)FBE"UUFOUJPOCBTFE.PEFMT 1SVOJOHBU*OGFSFODF5JNFn4FUUJOHUIFESPQSBUFGPSPQUJNBMQSVOJOH • (SPVQ ѐҊ 'JYFE%SPQ3BUJP оਸٸ (SPVQٜಣӐіࣻח ۽ ѐ(SPVQਸ1SVOJOHೠҊೞݶ
0QUJNBM%SPQ3BUF ח • ֫1SVOJOH3BUFח.PEFMীࢲ؊աࢿמਸࠁ • -BZFS%SPQীࢲח ܳࢎਊ೮ਵա .PEFMীࢲח ܳӂೣ N p N(1 − p) r p* p* = 1 − r N p = 0.2 p = 0.5