8-bit Quantization of Transformer Model

Slide 1

Slide 1 text

#JU2VBOUJ[BUJPOPG5SBOTGPSNFS.PEFM ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS 1JOHQPOH

Slide 2

Slide 2 text

ݾର ݾର &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM "CTUSBDU *OUSPEVDUJPO .FUIPE 3FTVMU

Slide 3

Slide 3 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS /FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM

Slide 4

Slide 4 text

"CTUSBDU &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM • *$.- *OUFM • ߓ੄ࣘبUISPVHIQVUೱ࢚ਸ঳ਵݶࢲ੿ب੄#-&6TDPSFBDDVSBDZ݅ڄয૗ • *OUFMDQVী୭੸ച • 5FOTPS'MPX • '1*OU 4

Slide 5

Slide 5 text

*OUSPEVDUJPO &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM • ୭न੄*OUFM$16ٜ਷WFDUPSJ[FEOFVSBMOFUXPSLJOTUSVDUJPO 7//* ٜਸನೣ • ѐ੄CJUܳ'." 'VTFE.VMUJQMZBOE"EE 0QFSBUJPOਸجܻחѪਸ$ZDMF۽ࣻ೯ • .BJO$POUSJCVUJPO • '1*/5RVBOUJ[BUJPOਸ޷݅੄405"#-&64DPSFೞۅ݅ਵ۽੉ܖযն • 1FSGPSNBODF0QUJNJ[BUJPO • .BU.VM • 2VBOUJ[FE.BU.VM(SBQI0QUJNJ[BUJPO • *OQVU1JQFMJOF0QUJNJ[BUJPO • 1BSBMMFM&YFDVUJPO 5

Slide 6

Slide 6 text

.PEFM%FTDSJQUJPO &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM • ೨ब'1*/5۽QSFDJTJPOਸ൞ࢤೞ؊ۄب୽࠙൤ࢿמ੉ੜաৢஹನք౟ܳ2VBOUJ[F • 4PGUNBY -BZFS/PSNBMJ[BUJPOҗэ਷҃਋ח*/5۽աఋղӝী൨ٚч੉݆ই  ֫਷BDD൞ࢤ੉৘࢚ؽ • .FBO 7BSJBODF &YQ١੄҅࢑੉*/5۽աఋղӝী൨ٝ 6

Slide 7

Slide 7 text

/BJWF2VBOUJ[BUJPO &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM • ӝઓ݆੉ॳ੉؍-JOFBS2VBOUJ[BUJPOߑधࢎਊ • .BY .JOਸ҅࢑೧ঠೞ޲۽-JOFBSTDBOਸ೙ਃ۽ೣ • 2VBOUJ[BUJPO0WFSIFBE0 / 7

Slide 8

Slide 8 text

/BJWF2VBOUJ[BUJPO &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 8

Slide 9

Slide 9 text

/BJWF2VBOUJ[BUJPO &GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 9 .JHBD[ 4CJUJOGFSFODFXJUIUFOTPSSU 63-IUUQPOEFNBOEHQVUFDIDPOGDPNHUDQSFTFOUBUJPOTCJUJOGFSFODFXJUIUFOTPSSUQEG

Slide 10

Slide 10 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM ,-%JWFSHFODFGPSPQUJNBMUISFTIPME • 2VBOUJ[BUJPO਷যରೖNBQQJOHਸযڌѱੜೞוջоޙઁ • '1UFOTPSEJTUSJCVUJPO_*/5UFOTPSEJTUSJCVUJPO • ߈ࠂ೧оݶࢲ*/5߸ജਸਤೠ0QUJNBM.JO .BYܳ଺ח׮ • 0QUJNBM౸ױਸ,-%JWFSHFODF۽҅࢑ • 7BMJEBUJPO%BUBTFUѐ੄ޙ੢઺ѐ੿بSBOEPNTBNQMJOH • .JO .BY5ISFTIPMEفѐܳ੿೧ঠೞחؘ Ӓߑߨਸࣁо૑੿ب۽ա־যࠆ 10 4ZNNFUSJD $POKVHBUF Ӓրٮ۽҅࢑

Slide 11

Slide 11 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM ,-%JWFSHFODFGPSPQUJNBMUISFTIPME 11

Slide 12

Slide 12 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM ,-%JWFSHFODFGPSPQUJNBMUISFTIPME • /BJWF2VBOUJ[BUJPO਷45015PLFOਸյࣻহ঻ӝٸޙী/" • Ӕؘࣘب࢚0GGTFU੉;FSPоغחಞ੉޷ࣁೞѱࣘبо؊ࡅܰ׮ • Ӓېࢲ4ZNNFUSJDࢎਊೣ 12

Slide 13

Slide 13 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM ,-%JWFSHFODFGPSPQUJNBMUISFTIPME 13 .JHBD[ 4CJUJOGFSFODFXJUIUFOTPSSU 63-IUUQPOEFNBOEHQVUFDIDPOGDPNHUDQSFTFOUBUJPOTCJUJOGFSFODFXJUIUFOTPSSUQEG

Slide 14

Slide 14 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO • 7//**OTUSVDUJPOࢎਊ • 0QFSBUJPO୨іࣻ઴੉ӝ • 3FPSEFS0QFSBUJPO • .,-۽0QUJNJ[BUJPO૓೯ • 1BSBMMFMJ[FCBUDIJOHFYFDVUJPO 14

Slide 15

Slide 15 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO2VBOUJ[FE.BU.VMT • "79 ࠺౟੄'."ো࢑$ZDMFীоמ • '1ѐ */5ѐ • $BTDBEF-BLF$16ࠗఠ*/57//*ܳ؊୭੸ച • 7//*ࢎਊೠ*/5.BU.VM਷"79ࢎਊೠ'1.BU.VMࠁ׮Yࡅܴ • 7//*ࢎਊೠ*/5.BU.VM਷"79ࢎਊೠ*/5.BU.VMࠁ׮Yࡅܴ 15

Slide 16

Slide 16 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO2VBOUJ[FE.BU.VMT 16

Slide 17

Slide 17 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO2VBOUJ[FE.BU.VMT • ೞ૑݅5FOTPS'MPXחJOUFHFS.BU.VMਊਵ۽(&..-081ۄחPQFOTPVSDFܳࢎਊೣ • (&..-081ח*/57//*ܳࢎਊೞ૑ঋҊ */5ো࢑द߸ജҗ੿ژೠ೙ਃೣ • Ӓېࢲ.,-#-"4ೣٜࣻ۽૒੽2VBOUJ[BUJPO4UFQ੘ࢿ • /PO;FSP0GGTFUبহছ 17

Slide 18

Slide 18 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO2VBOUJ[FE.BU.VMT 18 • HPPHMFHFNNMPXQ • UFOTPSGMPXUFOTPSGMPXীࢲחই૒ࢎਊೞחѪਸഛੋ • UFOTPSGMPXDPSFLFSOFMTRVBOUJ[FE@NBUNVM@PQTDD • UFOTPSGMPXMJUFLFSFOFMTDQV@CBDLFOE@HFNN@HFNNMPXQI

Slide 19

Slide 19 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 19 UFOTPSGMPXDPSFLFSOFMTRVBOUJ[FE@NBUNVM@PQTDD

Slide 20

Slide 20 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 20 UFOTPSGMPXDPSFLFSOFMTRVBOUJ[FE@NBUNVM@PQTDD

Slide 21

Slide 21 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 21 UFOTPSGMPXDPSFLFSOFMTEFRVBOUJ[F@PQDD

Slide 22

Slide 22 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 22 UFOTPSGMPXDPSFLFSOFMTEFRVBOUJ[F@PQDD

Slide 23

Slide 23 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO2VBOUJ[FE.BU.VMT 23 • /PO;FSP0GGTFUੌ҃਋HFNN@TVT੄ো࢑

Slide 24

Slide 24 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO(BUIFS/% 24 • (BUIFS/%חݫݽܻ*0о઺ਃೠো࢑ੋ݅ఀ */5۽઴੉؊ۄبো࢑ࣘبীח੉ٙ੉হ਺ • ೞ૑݅'1ࠁ׮*/5੉EBUBTJ[Fоߓ पઁ۽חY ੘ਵ޲۽*0੄ೱ࢚ਸӝ؀ೡࣻ੓਺ • (FOFSBUJPO-PPQীࢲ੹ױ҅੄Ѿҗীࢲ%FRVBOUJ[FEܳউೞҊ(BUIFS/%ܳ߄۽ࣻ೯ೡ ࣻ੓ਵ޲۽Y੄ࢿמೱ࢚੉੓঻਺

Slide 25

Slide 25 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO*OQVU1JQFMJOF 25 • 1BE5PLFO੉ҭ੢൤௾0WFSIFBE੉ӝٸޙী೧׼ష௾ਸহগࠁ۰חदب • 5PSDI੄1BDLFE4FRVFODFܳࢤп೮חؘ ӒѤইצ٠ • ੉֤ޙীࢲחৈ۞ߓ஖ܳTPSUJOH೧ࢲQBEUPLFO੉୭ࣗ۽ٜযоب۾ೣ • ੿ب੄ࢿמೱ࢚ਸ঳਺ • ੉ࠗ࠙਷5PSDI੄1BDLFE4FRVFODFэ਷ো࢑җэ਷ѐ֛ਸҳഅ೧ࢲॳݶ੿۳ೡ೙ਃبহ ੉഻ঁࡅܳѪэ׮

Slide 26

Slide 26 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO(SBQI0QUJNJ[BUJPO 26 • ੉੹ীࢲࣿ೮٠,-%JWFSHFODFܳ੉ਊ೧ UISFTEIPMETܳ଺חߑध਷NJO NBYܳো࢑ೞח दрਸহগળ׮$POTU0QFSBUJPOਵ۽߄Պ • 3FRVBOUJ[F৬3FRVBOUJ[BUJPO3BOHF 0QFSBUJPOਸ(SBQIীࢲহঞҊ */5ীࢲ '-0"5۽߄۽߄Բب۾ো࢑ਸ߸҃೮׮ ӝઓ ো࢑਷*/5*/5'-0"5

Slide 27

Slide 27 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO(SBQI0QUJNJ[BUJPO 27

Slide 28

Slide 28 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO1BSBMMFM#BUDIJOH 28 • &YFDVUJPOUJNF਷CBUDIউ੄TFOUFODFMFOHUIী੄ઓ੸੉׮ • -POHFSTFOUFODFח$16഻ܳঁബਯ੸ਵ۽ॳחѪਸҙ଴ೡࣻ੓঻Ҋ • 4FSJBMFYFDVUJPOदীח഻ঁ࠺ബਯ੸ਵ۽ॳחѪਸࠅࣻ੓঻׮ • ੉ѦQBSBMMFMFYFDVUJPOೞݶYࢿמೱ࢚੉ઓ੤ೣ • *NQMFNFOUBUJPO • '*'02VFVFܳҙܻೞח1BSFOU5FOTPS'MPX4FTTJPO੘ࢿ • ౠ੿$16௏য৬MPDBMNFNPSZীBGGJOJUJ[FEػ /6." DIJMEQSPDFTTGPSL • $IJMEQSPDFTTח2VFVFীࢲ"TZODISPOPVTೞѱো࢑૓೯

Slide 29

Slide 29 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO1BSBMMFM#BUDIJOH 29

Slide 30

Slide 30 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO$PODMVTJPO 30

Slide 31

Slide 31 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO$PODMVTJPO 31

Slide 32

Slide 32 text

&GGJDJFOU#JU2VBOUJ[BUJPOPG5SBOTGPSNFS/FVSBM.BDIJOF-BOHVBHF5SBOTMBUJPO.PEFM 1FSGPSNBODF0QUJNJ[BUJPO$PODMVTJPO 32

Slide 33

Slide 33 text

хࢎ೤פ׮✌ ୶о૕ޙژחҾӘೠ੼੉੓׮ݶ঱ઁٚইېোۅ୊۽োۅ઱ࣁਃ ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS 1JOHQPOH &NBJMVLKBF!TDBUUFSMBCDPLS 'BDFCPPL!KFPOHVLKBF -JOLFEJO!KFPOHVLKBF