Ligtweight Deep Learning Roadmap with Mobile-BERT

2019 MODUCON - 기술, 교육, 예술의 컨볼루션(convolution)
http://moducon.kr/

On-Device ML(모바일·임베디드 장치에 딥러닝 모델 이식)을 위한 모델 경량화 가이드 로드맵을 소개합니다. 실제 Mobile-BERT 모델의 Application Case를 기반으로 실제 개발 과정에서 발생할 수 있는 시행착오와 노하우들을 함께 살펴봅니다.

1. -JHIUXFJHIU
   %FFQ
   -FBSOJOH
   3PBENBQ
   ੉ܴ न੿ই &NBJM KFPOHBIBSJF!HNBJMDPN ઱ݠפ ࣘ "*
   m ݽ߄ੌ ٩۞׬
2. 
    %PZPVOH (XBL %POH4FPL :BOH 5BFLNJO ,JN +BFXPPL ,BOH :POHHFVO -FF +FPOHBI 4IJO :PVOHDIBF +J .BDIJOF
   -FBSOJOH
   PG
   5IJOHT झ݃౟ಪ ࣘ੄ ٩۞׬ .PEFM 0QUJNJ[BUJPO )BSEXBSF .PCJMF न੿ই KFPOHBISJ উ٘۽੉٘ ੹׸ োҳਗ • 5FOTPSGMPX -JUF
   .-
   ,JU
   1Z5PSDI .PCJMF
   "OESPJE
   //
   "1*
   'FEFSBUFE
   -FBSOJOH • ૒੽ ݽ؛ਸ ࢸ҅ೞҊ ӫҊ ୭੸ച ೟णदఃҊ ࢲࡂ 4FSWJOH
   ೞח оղࣻҕসਸ ૌѹਃ • ૑಴ ஏ੿җ పझ౟ ࢤഝച • ઱۽ 7JTJPO 1PTF
   &TUJNBUJPO
   0CKFDU
   %FUFDUJPO
   'BDF
   %FUFDUJPO
   5FYU
   %FUFDUJPO
   j • ਃ્੄ ҙबࢎח 3// җ 5SBOTGPSNFS m য়ט੄ ߊ಴ب #&35 ✨
3. $0/5&/54  -JHIUXFJHIU
   %FFQ
   -FBSOJOH  148JLJ XJUI
   .PCJMF#&35   TUFQT
   GPS
   -JHIUXFJHIU m )PX
   UP
   NBLF
   148JLJ  4VNNBSZ
4. -JHIUXFJHIU
   %FFQ
   -FBSOJOH ٩۞׬ ݽ؛ਸ ࢲࡂ TFSWJOH ೞח ߑߨীח 5FOTPSGMPX -JUF .-
   ,JU 0O%FWJDF ($1 .JDSPTPGU
   "[VSF $MPVE 4FSWFS
5. -JHIUXFJHIU
   %FFQ
   -FBSOJOH • ݽ߄ੌ੐߬٣٘ ੢஖੄ ੷੢ ਊ۝ %JTL
   4J[F ী Ѧݏח ੸਷ ݽ؛ ࢎ੉ૉ • ݽ߄ੌ "1੄ यӖ ௏য ജ҃ীࢲ ਗഝ൤ ز੘ೡ ࣻ ੓ب۾ ੸਷ ো࢑۝ • ؀׮ࣻ੄ ݽ߄ੌ জٜ੉ ݣ౭௏যܳ ૑ਗೞ૑ ঋҊ ੓਺ • וܽ 3". 3BOEPN
   "DDFTT
   .FNPSZ
   ࣘبী ؀਽ೞח ੸਷ ݫݽܻ ࢎਊ • ݽ߄ੌ੐߬٣٘ ӝӝ m ಕ੉૚ 1BHJOH ੄ ࠺ਊ੉ ࠺श • উ٘۽੉٘ জীࢲ ࢎਊೞח ݽٚ ё୓ח 3".ী ੷੢ 2VBMDPNN
   4OBQESBHPO
   
   4%. • ௪٘௏য ૑ਗ • $16੄ ࢿמ  ()[ 
   
   
   ()[  • Y
   GBTUFS
   UIBO
   J6 • 3".
   TQFFE
   
   
   .)[
   • .)[
   MPXFS
   UIBO
   J6 • ˯
   .FNPSZ
   TJ[F
   (PPHMF
   1JYFM
   9- 5FOTPSGMPX -JUF .-
   ,JU 0O%FWJDF 1Z5PSDI .PCJMF
6. -JHIUXFJHIU
   %FFQ
   -FBSOJOH • য়ߡ೻٘ 0WFSIFBE о ઴যٜݶ ࢲߡ ਬ૑ ࠺ਊب хࣗ • $16
   WT (16 • $16ীࢲ উ੿੸ਵ۽ ݽ؛ਸ ࢲࡂೣਵ۽ॄ ঳ਸ ࣻ ੓ח ࠺ਊ ੺х ബҗ • ױ (16
   ୭੸ച 5FOTPS35 ഝਊ о ੜػ ݽ؛੄ ҃਋ ৘৻о ੓ਸ ࣻ ੓਺ • .JDSPTFSWJDF
   "SDIJUFDUVSF
   m ݽ؛ ࢎ੉ૉ ੉ग • ੸੤ೡ ࣻ ੓ח ݽ؛ ࢎ੉ૉ੄ ઁড੉ ੓ח 4FWFSMFTT 'VODUJPOٜ ($1 .JDSPTPGU
   "[VSF $MPVE 4FSWFS "[VSF
   'VODUJPO
7. -JHIUXFJHIU
   %FFQ
   -FBSOJOH ৵ ਋ܻ੄ ݽ؛਷ о߶ਕঠ ೡө ! 0O%FWJDF
   ]]
   $MPVE 4FSWFS
   ࢎ੉ૉо ௼Ҋ ࠂ੟ೠ ݽ؛਷ • ݆਷ ੷੢ ҕр ٣झ௼ .FNPSZ ١ ਸ ର૑ೠ׮ • ҅࢑ ࠺ਊ੉ ֫׮ • ীց૑ ߓఠܻ ߂ ੹ӝ ܳ ݆੉ ࣗ࠺ೠ׮ #VU
   য়ט ࣗѐೡ ۽٘ݗ਷ 0O%FWJDF
   .-ਸ ਤೠ ҃۝ചী ؊ ૘઺
8. 148JLJ XJUI
   .PCJMF
   #&35 #&35 #JEJSFDUJPOBM
   &ODPEFS
   3FQSFTFOUBUJPOT
   GSPN
   5SBOTGPSNFST IUUQTBSYJWPSHBCT ױߑೱ ঩਷ নߑೱ Ө਷ নߑೱ EFFQMZ
   CJEJSFDUJPOBM • ؀ਊ۝ ఫझ౟ ௏ಌझܳ ੉ਊೞৈ ߧਊ ݾ੸੄ -BOHVBHF
   6OEFSTUBOEJOHਸ ࣻ೯ೞח ݽ؛ ള۲ • Ө਷ নߑೱࢿਸ ૑פҊ ੓਺ • ࣻরѐ .
   . ੄ QBSBNFUFS
   m Ѣ؀ೠ ݽ؛ ࢎ੉ૉ !
9. 148JLJ XJUI
   .PCJMF
   #&35 P S Wiki ಜ ࣻ %&.0 য়ט੄ ઱ઁܳ ׸਷
10. 148JLJ XJUI
    .PCJMF
    #&35 #&35
    XJUI
    4UBOGPSE
    2VFTUJPO
    "OTXFSJOH
    %BUBTFU
    42V"% • ਤఃೖ٣ই BSUJDMF BSUJDMFী ؀ೠ 2VFTUJPO  "OTXFS
    TFU • ੿ࠁо ࣻ۾ػ ޙױ 8JLJQFEJB Ѩ࢝ Ѿҗ җ ૕ޙ ޙ੢ਸ JOQVUਵ۽ ߉ইࢲ ޙױ ղীࢲ о੢ ਬ۱ೠ ੿׹ TFHNFOUܳ SFUVSO
    ೧઱ח ߑध যڌѱ ٜ݅঻ਸөਃ #&35 .PCJMF
11. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݾ಴ ࢸ੿ ୶ۿ ࣘب ੿ഛب ݽ؛ ௼ӝ ݫݽܻ ࢎਊ۝ ߓఠܻ ࢎਊ۝ • पदр ࢿמ ࢎਊ੗ ୓х _NT • NC
    ੉ೞ੄ ݽ؛ ࢎ੉ૉ • ੸਷ ߓఠܻ ࢎਊ۝ 5BTL
    ݃׮ ׮ܲ ਋ࢶࣽਤ $1.#BTFE
    1PTF
    &TUJNBUJPO !
12. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݾ಴ ࢸ੿ ୶ۿ ࣘب ੿ഛب ݽ؛ ௼ӝ ݫݽܻ ࢎਊ۝ ߓఠܻ ࢎਊ۝ 14
    8JLJ • ࢎਊ੗ ૑োਸ וՙ૑ ޅೡ݅ೠ ࢿמ _NT • (PPHMF
    1MBZ
    4UPSFী ١੤ೡ ࣻ ੓ח ݽ؛ ࢎ੉ૉ • NC
    ੉ೞ੄ জ ࢎ੉ૉ ݽ؛ ࢎ੉ૉ NC
    ੉ೞ • ੿ഛب •  ੉࢚ 2"
    ݽ؛੉޲۽ ੿ഛبо ઺ਃ !
13. 
    4UFQT
    GPS
    -JHIUXFJHIU #&35 .PCJMF 45&1
    
    ҃۝ ঌҊ્ܻ ଺ӝ • ޖࣻೠ प೷җ োҳо ೙ਃೠ ৔৉ • োҳ ࢶ۹ܳ ࢓ಝࠁݶ  .PEFM
    "SDIJUFDUVSF • 3FT/FU
    %FOTF/FU
    4RVFF[F/FU  0QUJNJ[JOH
    $POWPMVUJPO
    'JMUFS • .PCJMF/FU
    4IVGGMF/FU  /FVSBM
    "SDIJUFDUVSF
    4FBSDI • /FU"EBQU
    ./BT/FU
14. 
    4UFQT
    GPS
    -JHIUXFJHIU #&35 .PCJMF 45&1
    
    ҃۝ ঌҊ્ܻ ଺ӝ "-#&35
    "
    -JUF
    #&35
    GPS
    4FMGTVQFSWJTFE
    -FBSOJOH
    PG
    -BOHVBHF
    3FQSFTFOUBUJPOT https://arxiv.org/abs/1909.11942 • ౵ۄ޷ఠ хࣗ • GBDUPSJ[FE
    FNCFEEJOH
    QBSBNFUFSJ[BUJPO • DSPTTMBZFS
    QBSBNFUFS
    TIBSJOH • *OUFSDPIFSFODF
    ܳ ਤೠ ࢜۽਍ MPTT
    ࢎਊ • TFOUFODFPSEFS
    QSFEJDUJPO 401 • ഴܯೠ UG 
    ҳഅ୓ ଵҊ • IUUQTHJUIVCDPNLQFCFSUGPSUG
15. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO • 8FJHIU
    1SVOJOH
    о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU
    4IBSJOH
    о઺஖ ҕਬ • ,OPXMFEHF
    %JTUJMMBUJPO
    ૑ध ૐܨ • 5SBOTGFS
    -FBSOJOH
    ੹੉ ೟ण j
16. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO • 8FJHIU
    1SVOJOH
    о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU
    4IBSJOH
    о઺஖ ҕਬ • ,OPXMFEHF
    %JTUJMMBUJPO
    ૑ध ૐܨ • 5SBOTGFS
    -FBSOJOH
    ੹੉ ೟ण j  
               
    
17. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO • 8FJHIU
    1SVOJOH
    о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU
    4IBSJOH
    о઺஖ ҕਬ • ,OPXMFEHF
    %JTUJMMBUJPO
    ૑ध ૐܨ • 5SBOTGFS
    -FBSOJOH
    ੹੉ ೟ण j 5BTL
    ౠࢿ࢚ ࠗ೤ೞ૑ 9 "-#&35
    $PODFQU • 1SFUSBJOFEػ -BSHF
    NPEFM
    ਸ ೟णೞৈ ੘਷ ݽ؛ী EJTUJMMBUJPO
    ೞח җ੿ হ੉ ੘਷ ݽ؛ਸ ߄۽ ೟ण दр੉ য়ې Ѧܻח प೷ /0 !
18. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO • 8FJHIU
    1SVOJOH
    о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO ন੗ച )PX
    UP 5FOTPSGMPX ীࢲ ૑ਗೞח ҃۝ച୭੸ച GFBUVSFܳ ࢎਊ
19. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO
    m 8FJHIU
    1SVOJOH 8FJHIU
    1SVOJOH
    о઺஖ о૑஖ӝ
    ۆ ӝઓ न҃ݎ੉ о૑Ҋ ੓ח о઺஖ 8FJHIUT
    ઺ पઁ ୶ۿਸ ਤ೧ ೙ਃೠ ч਷ ࠺Ү੸ ੘਷ чٜী ؀ೠ ղࢿਸ о૑޲۽
    ੘਷ о઺஖ чਸ ݽف ਵ ۽ ೞৈ ֎౟ਕ௼੄ ݽ؛ ௼ӝܳ ઴੉ח ӝࣿ import tensorflow_model_optimization as tfmot # Sparsity 0% to 50% while training pruning_schedule = tfmot.sparsity.keras.PolynomialDecay( initial_sparsity=0.0, final_sparsity=0.5, begin_step=2000, end_step=4000) # From 2000 to 4000 step, runs every 100 steps # If it runs every steps, the accuracy would be fallen ! model_for_pruning = tfmot.sparsity.keras.prune_low_magnitude( model, pruning_schedule=pruning_schedule) model_for_pruning.fit(...)
20. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO
    m 2VBOUJ[BUJPO 2VBOUJ[BUJPO
    ন੗ച
    ۆ न҃ݎ੄ ࠗز ࣗࣻ੼ ࣻܳ ౠ੿ࣻ۽ ઴੉ח ੘স
    ઱۽ ݽ߄ੌীࢲח GMPBU
    ! VJOU • %VSJOHUSBJOJOH
    RVBOUJ[BUJPO • "-#&35
    QSFUSBJOFEXFJHIUܳ ߉ই RVBOUJ[BUJPOਸ ऺইоח ߑೱਵ۽ ࣻ೯ • 1PTUUSBJOJOH
    RVBOUJ[BUJPO • 8FJHIU0/-: • ݽ؛ ࢎ੉ૉ 
    хࣗ
    ೞ૑݅ पઁ ୶ۿद੄ ো࢑਷ GMPBU
    ୓҅۽ SFDPOTUSVDUJPO
    غয ੉ܖয૑ӝ ٸޙী ࣘ ب ߸ച 9 • 8FJHIU
    
    "DUJWBUJPO
    'VMMZ • 'VMMZ۽ QPTURVBOU
    ࣻ೯ೞӝ ਤ೧ࢲח 5FOTPSGMPX -JUF
    ࢚ীࢲ VJOU
    ୶ۿ ੹ਊ 2VBOUJ[FE0QTܳ ૑ਗ೧ঠೣ " "
21. 5FOTPSGMPX
    .PEFM
    0QUJNJ[BUJPO
    5PPMLJU ✓ 1PTUUSBJOJOH
    2VBOUJ[BUJPO  8FJHIU
    0/-:
22. 5FOTPSGMPX
    .PEFM
    0QUJNJ[BUJPO
    5PPMLJU ✓ 1PTUUSBJOJOH
    2VBOUJ[BUJPO  'VMMZ
    2VBOUJ[FE
    XFJHIUBDUJWBUJPO
23. 
    4UFQT
    GPS
    -JHIUXFJHIU 45&1
    
    ࢲࡂ 4FSWJOH ୭੸ച m (16
    %FMFHBUJPO
    "1*
    ࢎਊ (16 (SBQIJDT
    1SPDFTTJOH
    6OJU
    
    ݆਷ ୊ܻܳ ਃೞ૑݅ ࡈܻ ୊ܻ೧ঠೞח ݽ؛ $16 $FOUSBM
    1SPDFTTJOH
    6OJU
    
    рױೠ .-
    ݽ؛ী੄ ୭ࢶ੄ ২࣌ %41 %JHJUBM
    4JHOBM
    1SPDFTTPS
    
    ੸਷ ୊ܻ۝ਸ ਃೞݶࢲب ࠂ੟ೠ ݽ؛
    ষ୒աѱ ࡈܻ ୊ܻ೧ঠೞח ݽ؛
24. 5FOTPSGMPX -JUF 'BTUFS
    XJUI
    .PCJMF
    (16 ୹୊
    
    IUUQTNFEJVNDPNUFOTPSGMPXUFOTPSGMPXMJUFOPXGBTUFSXJUINPCJMFHQVTEFWFMPQFSQSFWJFXFFEFF 5IF
    OFX
    (16
    CBDLFOE  1FSGPSNT
    
    mY GBTUFS 5IF
    GMPBUJOH
    QPJOU
    $16
    JNQMFNFOUBUJPO
25. (16
    %FMFHBUJPO
    "1* IUUQTXXXUFOTPSGMPXPSHMJUFQFSGPSNBODFHQV@BEWBODFE ✓ .PCJMF
    (16ܳ оࣘೡ ࣻ ੓ח 5FOTPSGMPX
    0QFSBUPSח ೠ ੿غয ੓਺ ✓ (16
    %FMFHBUFীࢲ ૑ਗೞ૑ ঋח 0QFSBUPSо ࢎਊػ ࠗ ࠙ী ೠীࢲ $16৬੄ ߽೯੸ੋ ࢎਊ੉ оמೞ૑݅ $16݅ ࢎਊ ೡ ٸ ࠁ׮ غ۰ ࢿמ੉ ೞۅؼ ࣻ ੓਺ ✓ .PCJMF#&35
    ݽ؛ীࢲח (16
    %FMFHBUJPOਸ ࢎਊೞח Ѫ ࠁ׮ $16
    ജ҃ীࢲ .VMUJUISFBEJOH
    ೞחѱ ؊ ബҗ੸੉঻਺
26. 4VNNBSZ Ѿҗ • *OGFSFODF
    UJNF
     ಣӐ NT • .PEFM
    4J[F  .# • "DDVSBDZ • UGFTUJNBUPS XJUI
    4BWFE.PEFM 'SP[FO
    (SBQI
    m  • UG*OUFQSFUFS XJUI
    5'-JUF m 
27. 4VNNBSZ 45&1
     45&1
     45&1
     45&1
     ݾ಴ ࢸ੿ೞӝ m ୶ۿ ࣘب
    ݽ؛ ࢎ੉ૉ
    ݫݽܻ ࢎਊ۝
    ߓఠܻ ࢎਊ۝ j
    ҃۝ ঌҊ્ܻ ଺ӝ m թ੉ ݅ٚ պ੢Ҋ ഝਊೞӝ ݽ؛ ঑୷ .PEFM
    $PNQSFTTJPO ࢲࡂ 4FSWJOH ୭੸ച
28. 3FGFSFODF 3FDFOU
    3%
    5SFOET
    GPS
    -JHIUXFJHIU
    %FFQ
    -FBSOJOH m &53* IUUQTFUUSFOETFUSJSFLSFUUSFOET