モデル高速化百選 内田祐介 AIシステム部 株式会社ディー・エヌ・エー

લఏ • ओʹԼهͷ৚݅Λຬͨ͢ख๏Λ঺հ • ಛఆͷϋʔυ΢ΣΞʹґଘͤͣʹ࣮ݱՄೳ • ৞ΈࠐΈχϡʔϥϧωοτϫʔΫ $// ͕ର৅ • ਪ࿦࣌ͷߴ଎Խ͕ର৅ • ඦબ͠·͕ͨ͠Ұ෦͚ͩ঺հ͠·͢ • "QQFOEJYʹϦετ͕͋Γ·͢

ߴ଎Խʁ • Ϟσϧύϥϝʔλ਺ͷ࡟ݮ • '-01T ."$T ਺ͷ࡟ݮ • ϞσϧϑΝΠϧαΠζͷ࡟ݮ • ਪ࿦࣌ؒͷ࡟ݮ • ܇࿅࣌ؒͷ࡟ݮ ඍົʹҧ͏ͷͰɺ࢖͏ͱ͖͸ԿΛॏࢹ͢΂͖͔ɺ ࿦จΛಡΉͱ͖͸Կ͕վળ͍ͯ͠Δͷ͔Λؾʹ͢Δ

Ϟσϧߴ଎Խ • ৞ΈࠐΈͷ෼ղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4 • ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO

畳み込みの分解 (Factorization)

৞ΈࠐΈ૚ͷܭࢉྔ • ೖྗϨΠϠαΠζɿ)Y8Y/ • ৞ΈࠐΈΧʔωϧɿ,Y,Y/Y. DPOW,Y, .ͱදه FHDPOWY • ग़ྗϨΠϠαΠζɿ)Y8Y. • ৞ΈࠐΈͷܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ.ʢόΠΞε߲Λແࢹʣ 8 ) / . , , 8 ) ⼊⼒特徴マップ 畳み込み カーネル / 出⼒特徴マップ ˎ ࿨ ཁૉੵ × . DPOW,º, . 畳み込み層の計算量は • 画像/特徴マップのサイズ（HW） • ⼊出⼒チャネル数（NM） • カーネルサイズ（K2） に⽐例

ۭؒํ޲ͷ෼ղ • େ͖ͳ৞ΈࠐΈΧʔωϧΛখ͞ͳ৞ΈࠐΈΧʔωϧʹ෼ղ • ྫ͑͹Yͷ৞ΈࠐΈΛYͷ৞ΈࠐΈͭʹ෼ղ • ͜ΕΒ͸ಉ͡αΠζͷड༰໺Λ͕࣋ͭ෼ղ͢Δͱܭࢉྔ͸ • *ODFQUJPOW<>Ͱ͸࠷ॳͷY৞ΈࠐΈΛY৞ΈࠐΈͭʹ෼ղ • Ҏ߱ͷ4&/FU΍4IV⒐F/FU7౳ͷ࣮૷Ͱ΋ར༻͞Ε͍ͯΔ<> ಛ௃Ϛοϓ conv5x5 conv3x3 - conv3x3 [4] C. Szegedy, et al., "Rethinking the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016. [18] T. He, et al., "Bag of Tricks for Image Classification with Convolutional Neural Networks," in Proc. of CVPR, 2019.

ۭؒํ޲ͷ෼ղ • OYOΛYOͱOYʹ෼ղ͢Δ͜ͱ΋ [4] C. Szegedy, et al., "Rethinking the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016.

ۭؒํ޲ͱνϟωϧํ޲ͷ෼ղ TFQBSBCMFDPOW • ۭؒํ޲ͱνϟωϧํ޲ͷ৞ΈࠐΈΛಠཱʹߦ͏ • %FQUIXJTF৞ΈࠐΈʢۭؒํ޲ʣ • ಛ௃Ϛοϓʹର͠νϟωϧຖʹ৞ΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ. ./ )ɾ8ɾ,ɾ/ • 1PJOUXJTF৞ΈࠐΈʢνϟωϧํ޲ʣ • Yͷ৞ΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ. , )ɾ8ɾ/ɾ. • %FQUIXJTFQPJOUXJTF TFQBSBCMF • ܭࢉྔɿ)ɾ8ɾ/ɾ ,. 㲈)ɾ8ɾ/ɾ. ˞., • )ɾ8ɾ/ɾ,ɾ.͔Βେ෯ʹܭࢉྔΛ࡟ݮ W H W H N 1 1 M W H W H N K K N W H W H N M K K 通常 depthwise pointwise

9DFQUJPO<> • 4FQBSBCMFDPOWΛଟ༻ͨ͠Ϟσϧ [6] F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in Proc. of CVPR, 2017.

.PCJMF/FU<> • EFQUIXJTFQPJOUXJTFDPOWΛଟ༻ • վྑ൛ͷ.PCJMF/FU7<>7<>΋͋Δ 通常の畳み込み MobileNetの1要素 [7] A. Howard, et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," in arXiv:1704.04861, 2017. [13] M. Sandler, et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks," in Proc. of CVPR, 2018. [20] A. Howard, et al., "Searching for MobileNetV3," in arXiv:1905.02244, 2019.

4IV⒐F/FU<> • .PCJMF/FUͷϘτϧωοΫͱͳ͍ͬͯΔDPOWYΛ HSPVQDPOWYDIBOOFMTIV⒐Fʹஔ׵ • HSPVQDPOWೖྗͷಛ௃ϚοϓΛ(ݸʹάϧʔϓԽ͠ ֤άϧʔϓ಺Ͱݸผʹ৞ΈࠐΈΛߦ͏ ʢܭࢉྔ)ɾ8ɾ/ɾ,ɾ.ˠ)ɾ8ɾ/ɾ,ɾ.(ʣ • DIBOOFMTIV⒐FνϟωϧͷॱংΛೖΕସ͑Δ SFTIBQFUSBOTQPTFͷૢ࡞Ͱ࣮ݱՄೳ DTIVGGMF EFQUIXJTFDPOW HDPOWY spatial channel HDPOWY [8] X. Zhang, et al., "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," in arXiv:1707.01083, 2017.

$IBOOFM/FU<> • νϟωϧํ޲ʹ࣍ݩͷ৞ΈࠐΈΛߦ͏ [11] H. Gao, Z. Wang, and S. Ji, "ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions", in Proc. of NIPS, 2018.

枝刈り (Pruning)

ࢬמΓ 1SVOJOH • ৞ΈࠐΈ૚΍શ݁߹૚ͷॏΈͷҰ෦Λʹ͢Δ͜ͱͰ ύϥϝʔλ਺ɾܭࢉྔΛ࡟ݮ ωοτϫʔΫΛֶश ࢬמΓʢਫ਼౓௿Լʣ ωοτϫʔΫΛ࠶ֶशʢਫ਼౓Λ͋Δఔ౓ճ෮ʣ ͱ͍͏ϑϩʔ͕Ұൠత

6OTUSVDUVSFEWT4USVDUVSFE1SVOJOH • 1SVOJOHલͷ৞ΈࠐΈϑΟϧλ • 6OTUSVDUVSFEQSVOJOH • 4USVDUVSFEQSVOJOHʢϑΟϧλʢνϟωϧʣQSVOJOH͕Ұൠతʣ K K … … … M（出⼒チャネル）個 計算量vs.精度のtrade-offは優れているが 専⽤のハードウェアでないと⾼速化できない 単にチャネル数が減少したネットワークに 再構築が可能で⾼速化の恩恵を受けやすい

%FFQ$PNQSFTTJPO< > • 6OTUSVDUVSFEͳQSVOJOH • - ਖ਼ଇԽΛՃֶ͑ͯश͠ɺઈର஋͕খ͍͞XFJHIUΛʹ • ࣮ࡍʹߴ଎ʹಈ͔͢ʹ͸ઐ༻ϋʔυ͕ඞཁ<> [23] S. Han, et al., "Learning both Weights and Connections for Efficient Neural Networks," in Proc. of NIPS, 2015. [25] S. Han, et al., "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," in Proc. of ICLR, 2016. [26] S. Han, et al., "EIE: Efficient Inference Engine on Compressed Deep Neural Network," in Proc. of ISCA, 2016.

1SVOJOH'JMUFSTGPS&⒏DJFOU$POW/FUT<> • 4USVDUVSFEQSVOJOHʢνϟωϧϨϕϧͷQSVOJOHʣ • ֤ϨΠϠʹ͍ͭͯɺϑΟϧλͷॏΈͷઈର஋ͷ૯࿨͕ খ͍͞΋ͷ͔ΒQSVOJOH • ֤ϨΠϠͷQSVOJOH཰͸QSVOJOH΁ͷTFOTJUJWJUZ͔Β ਓखͰௐ੔ • 1SVOJOHޙʹGJOFUVOF [30] H. Li, et al., "Pruning Filters for Efficient ConvNets," in Proc. of ICLR, 2017.

/FUXPSL4MJNNJOH<> • #BUDIOPSNͷύϥϝʔλЍʹ- ϩεΛֶ͔͚ͯश • ֶशޙɺЍ͕খ͍͞νϟωϧΛ࡟আ͠ɺpOFUVOF νϟωϧຖʹೖྗΛฏۉ0෼ࢄ1ʹਖ਼نԽɺγͱβͰscale & shift νϟωϧi … … Batch normalization [33] Z. Liu, et al., "Learning Efficient Convolutional Networks through Network Slimming," in Proc. of ICCV, 2017.

L0 Ͱ͸ͳ͘Lasso ʹ؇࿨ͯ͠ղ͘ $IBOOFM1SVOJOH<> • ͋ΔGFBUVSFNBQͷνϟωϧ࡟আͨ͠৔߹ʹ ࣍ͷGFBUVSFNBQͷޡ͕ࠩ࠷খͱͳΔΑ͏νϟωϧΛબ୒ [34] Y. He, et al., "Channel Pruning for Accelerating Very Deep Neural Networks," in Proc. of ICCV, 2017.

5IJ/FU<> • લड़ͷख๏ͱಉ͡Α͏ʹɺ࣍ͷGFBUVSFNBQͷޡ͕ࠩ ࠷খͱͳΔϨΠϠΛHSFFEZ࡟আ • ࡟আޙʹɺ৞ΈࠐΈͷॏΈΛޡ͕ࠩ࠷খʹͳΔΑ͏ʹ ௐ੔ˠpOFUVOF [35] J. Luo, et al., "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," in Proc. of ICCV, 2017.

"VUP.-GPS.PEFM$PNQSFTTJPOBOE"DDFMFSBUJPO ".$ <> • ڧԽֶशʢP⒎QPMJDZBDUPSDSJUJDʣʹΑΓ ֤ϨΠϠຖͷ࠷దͳQSVOJOH཰Λֶश • ೖྗ͸ର৅ϨΠϠͷ৘ใͱͦΕ·ͰͷQSVOJOH݁Ռɺ ใु͸rΤϥʔ཰ºMPH '-01T PSMPH 1BSBNT [41] Y. He, et al., "AMC - AutoML for Model Compression and Acceleration on Mobile Devices," in Proc. of ECCV, 2018.

-PUUFSZ5JDLFU)ZQPUIFTJT *$-3`#FTU1BQFS <> • //ʹ͸ɺʮ෦෼ωοτϫʔΫߏ଄ʯͱʮॳظ஋ʯͷ ૊Έ߹Θͤʹʮ౰ͨΓʯ͕ଘࡏ͠ɺͦΕΛҾ͖౰ͯΔͱ ޮ཰తʹֶश͕Մೳͱ͍͏Ծઆ • 6OTUSVDUVSFEͳQSVOJOHͰͦͷߏ଄ͱॳظ஋Λݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ https://www.slideshare.net/YosukeShinya/the-lottery-ticket-hypothesis-finding-small-trainable-neural-networks [44] Jonathan Frankle, Michael Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," in Proc. of ICLR, 2019.

/FUXPSL1SVOJOHBT"SDIJUFDUVSF4FBSDI<> • 4USVDUVSFEͳQSVOJOHޙͷωοτϫʔΫΛTDSBUDI͔Βֶशͤͯ͞΋ pOFUVOFͱಉ౳͔ͦΕΑΓྑ͍݁Ռ͕ಘΒΕΔͱ͍͏ओு • ͭ·ΓQSVOJOH͸ɺॏཁͳॏΈΛ୳ࡧ͍ͯ͠ΔͷͰ͸ͳ͘ ֤ϨΠϠʹͲͷఔ౓ͷνϟωϧ਺ΛׂΓ౰ͯΔ͔ͱ͍͏ /FVSBM"SDIJUFDUVSF4FBSDI /"4 Λ͍ͯ͠ΔͱΈͳͤΔ • -PUUFSZ5JDLFU)ZQPUIFTJTͰ͸VOTUSVDUVSFEͰɺ௿-3ͷΈɺ ࣮ݧ΋খن໛ωοτϫʔΫͷΈ [45] Z. Liu, et al., "Rethinking the Value of Network Pruning," in Proc. of ICLR, 2019.

アーキテクチャ探索 (Neural Architecture Search; NAS)

ΞʔΩςΫνϟ୳ࡧ /"4 • //ͷΞʔΩςΫνϟΛࣗಈઃܭ͢Δख๏ • ୳ࡧۭؒɺ୳ࡧख๏ɺਫ਼౓ධՁख๏Ͱେ·͔ʹ෼ྨ͞ΕΔ • ୳ࡧۭؒ • (MPCBM DFMMCBTFE • ୳ࡧख๏ • ڧԽֶशɺਐԽతΞϧΰϦζϜɺHSBEJFOUϕʔεɺSBOEPN • ਫ਼౓ଌఆख๏ • શֶशɺ෦෼ֶशɺXFJHIUTIBSFɺࢬמΓ୳ࡧ T. Elsken, J. Metzen, and F. Hutter, "Neural Architecture Search: A Survey," in JMLR, 2019. M. Wistuba, A. Rawat, and T. Pedapati, "A Survey on Neural Architecture Search," in arXiv:1905.01392, 2019. https://github.com/D-X-Y/awesome-NAS

/"4/FU<> • ୳ࡧۭؒɿDFMMɺ ୳ࡧख๏ɿڧԽֶश 1SPYJNBM1PMJDZ0QUJNJ[BUJPO • (MPCBMͳઃܭʹυϝΠϯ஌ࣝΛ׆༻ɺ ߏ੒͢ΔDFMMͷΈΛࣗಈઃܭ ˠ୳ࡧۭؒΛେ෯ʹ࡟ݮ • /PSNBMDFMMY/ͱSFEVDUJPODFMMͷελοΫ • 3FEVDUJPODFMM͸࠷ॳʹTUSJEF෇͖ͷ01Ͱ ಛ௃ϚοϓΛμ΢ϯαϯϓϧ • 3FEVDUJPODFMMҎ߱ͰνϟωϧΛഒʹ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.

/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ୒ ͦΕΒ΁ͷ01TΛબ୒˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.

/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ୒ ͦΕΒ΁ͷ01TΛબ୒˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.

/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ୒ ͦΕΒ΁ͷ01TΛબ୒˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.

/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ୒ ͦΕΒ΁ͷ01TΛબ୒˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ concat sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.

&/"4<> • ୳ࡧۭؒɿDFMMɺ୳ࡧख๏ɿڧԽֶश 3&*/'03$& • $FMMͷߏ଄Λग़ྗ͢Δ3//ίϯτϩʔϥͱɺ ίϯτϩʔϥʔ͕ग़ྗ͢ΔશͯͷωοτϫʔΫΛαϒάϥϑͱͯ͠อ ࣋Ͱ͖ΔڊେͳܭࢉάϥϑʢωοτϫʔΫʣΛಉ࣌ʹֶश ˠੜ੒ͨ͠ωοτϫʔΫͷֶश͕ෆཁʹʢ(16GPSEBZTʣ • 4JOHMFTIPU XFJHIUTIBSF • ৄࡉ͸ਆࢿྉΛࢀর [54] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and Jeff Dean, "Efficient Neural Architecture Search via Parameter Sharing," in Proc. of ICML, 2018. * https://www.slideshare.net/tkatojp/efficient-neural-architecture-search-via-parameters- sharing-icml2018

&/"4ͷֶश • ίϯτϩʔϥʔͷύϥϝʔλВͱ ڊେͳωοτϫʔΫͷύϥϝʔλXΛަޓʹֶश • Xͷֶश • ВΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • αϒάϥϑΛGPSXBSECBDLXBSE͠XΛߋ৽ • Вͷֶश • XΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • WBMJEBUJPOσʔλͰਫ਼౓Λଌఆ͠ใुΛऔಘɺ3&*/'03$&ͰВΛߋ৽

%"354<> • ୳ࡧۭؒɿDFMMɺ୳ࡧख๏ɿHSBEJFOU • άϥϑͷ઀ଓ΍01ͷબ୒ΛTPGUNBYͰ࣮ݱ͢Δ͜ͱͰɺ ߏ଄୳ࡧ΋GPSXBSECBDLXBSEͰ࣮ݱ • &/"4ͱಉ͘͡TIBSFEQBSBNɺXͱߏ଄Λަޓʹ࠷దԽ [57] H. Liu, K. Simonyan, and Y. Yang, "DARTS: Differentiable Architecture Search," in Proc. of ICLR, 2019.

'#/FU<> • %"354ͱಉ͘͡HSBEJFOUCBTFE • ֤01ͷ࣮σόΠε্Ͱͷॲཧ࣌ؒΛMPPLVQUBCMFʹอ࣋ • ॲཧ࣌ؒΛߟྀͨ͠ϩεΛ͔͚Δ [61] B. Wu, et al., "FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search", in Proc. of CVPR, 2019. ΫϩεΤϯτϩϐʔ ॲཧ࣌ؒ

ଞʹ΋ <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF 4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3 <>.5BO #$IFO 31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27 -F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713 <>9%BJ FUBM $IBN/FU5PXBSET&⒏DJFOU/FUXPSL%FTJHOUISPVHI 1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713 <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&⒏DJFOU$POW/FU %FTJHO JO1SPDPG*$.-8

早期終了、動的計算グラフ (Early Termination, Dynamic Computation Graph)

ૣظऴྃ &BSMZUFSNJOBUJPO • ೖྗʹԠͯ͡ωοτϫʔΫͷ్தͰ݁ՌΛग़ྗ͠ɺ ͦΕҎ߱ͷॲཧΛߦΘͳ͍ʢૣظऴྃʣ • ೖྗʹԠͯ͡ωοτϫʔΫͷߏ଄Λಈతʹม͑Δ ʢಈతܭࢉάϥϑEZOBNJDDPNQVUBUJPOHSBQIʣ • ʮฏۉॲཧ࣌ؒʯΛ࡟ݮ͢Δ

#SBODIZ/FU<> • ωοτϫʔΫͷ్தʹ݁Ռͷग़ྗ૚Λ௥Ճ • ֶश࣌ʹ͸͢΂ͯͷग़ྗ૚ʹద౰ͳXFJHIUΛֶ͔͚ͯश • ͦͷTPGUNBYͷΤϯτϩϐʔ͕ᮢ஋ҎԼͷ৔߹ʹ&YJU [65] S. Teerapittayanon, et al., "BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks," in Proc. of ICPR, 2016.

4QBUJBMMZ"EBQUJWF$PNQVUBUJPO5JNF 4"$5 <> • "$5֤3FT#MPDL͕IBMUJOHTDPSFΛग़ྗɺ߹ܭ͕Λ௒͑Δͱ Ҏ߱ͷॲཧΛεΩοϓʢۭؒྖҬͰ΋ߦ͏ͱ4"$5ʣ ܭࢉྔʹؔ͢Δޯ഑Λ௥Ճ [66] M. Figurnov, et al., "Spatially Adaptive Computation Time for Residual Networks," in Proc. of CVPR, 2017.

3VOUJNF/FVSBM1SVOJOH<> • ֤ϨΠϠຖʹɺ௚લ·Ͱͷಛ௃ϚοϓΛೖྗͱ͢Δ3//͕ ར༻͢Δ৞ΈࠐΈϑΟϧλू߹Λܾఆ • ,FFQͨ͠৞ΈࠐΈϑΟϧλ਺ͱݩλεΫͷଛࣦؔ਺ʢ࠷ऴ૚ͷ৔߹ ʣΛෛͷใुͱͯ͠2ֶशͰ3//Λֶश [68] J. Lin, et al., "Runtime Neural Pruning," in Proc. of NIPS, 2017.

#MPDL%SPQ<> • 1PMJDZOFUXPSLʹը૾ΛೖྗɺͲͷ#MPDLΛεΩοϓ͢Δ͔Λग़ྗ • ,FFQͱͳͬͨ3FT#MPDLͷΈΛGPSXBSE • ೝ͕ࣦࣝഊͨ͠৔߹͸ෛͷใुΛɺ੒ޭͨ͠৔߹ʹ͸εΩοϓ཰ʹԠ ͨ͡ਖ਼ͷใुΛ༩͑Δ͜ͱͰQPMJDZOFUXPSLΛֶश [73] Z. Wu, et al., "BlockDrop: Dynamic Inference Paths in Residual Networks," in Proc. of CVPR, 2018.

蒸留 (Distillation)

ৠཹ %JTUJMMBUJPO • େ͖ͳϞσϧ΍ɺෳ਺ͷωοτϫʔΫͷΞϯαϯϒϧΛ ʮڭࢣϞσϧʯͱ͠ɺখ͞ͳʮੜెϞσϧʯΛֶश • ڭࢣͷग़ྗ΍தؒಛ௃Λੜె͕໛ٖ͢ΔΑ͏ͳϩεΛ͔͚Δ 1. アンサンブルモデルや⼤ きなモデルを学習 2. 学習済みモデルを利⽤して ⼩さなモデルを学習

%JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL<> … … 学習画像 学習済みモデル 学習するモデル … 正解ラベル （ハード ターゲット） ௨ৗT = 1ͷsoftmaxͷTΛେ͖ͨ͘͠ ιϑτλʔήοτΛར༻ … ソフトターゲット ソフト ターゲット ハード ターゲット ਖ਼ղϥϕϧͱ ֶशϞσϧग़ྗͷ ྆ํΛར༻ [77] G. Hinton, et al., "Distilling the Knowledge in a Neural Network," in Proc. of NIPS Workshop, 2014.

'JU/FU<> • ڭࢣΑΓ΋EFFQ͔ͭUIJOͳੜెΛֶश͢Δ • ੜెͷHVJEFEMBZFS͕ɺڭࢣͷIJUMBZFSͷग़ྗΛ ਖ਼֬ʹ໛ٖ͢Δ SFHSFTTJPO ϩεΛ௥Ճ [79] A. Romero, et al., "FitNets: Hints for Thin Deep Nets," in Proc. of ICLR, 2015.

量子化 (Quantization)

ྔࢠԽ • ωοτϫʔΫͷύϥϝʔλ౳ΛྔࢠԽ͢Δ͜ͱͰ ϞσϧαΠζΛ࡟ݮɺֶश΍ਪ࿦Λߴ଎Խ • ྔࢠԽର৅ • ॏΈɺΞΫςΟϕʔγϣϯʢಛ௃Ϛοϓʣɺޯ഑ɺΤϥʔ • ྔࢠԽख๏ • ઢܗɺMPHɺඇઢܗεΧϥɺϕΫτϧɺ௚ੵྔࢠԽ • ྔࢠԽϏοτ • CJUʢόΠφϦʣɺ஋ ɺCJUɺCJUɺ೚ҙCJU • ઐ༻ϋʔυ͕ͳ͍ͱԸܙΛड͚ΒΕͳ͍ࣄ͕ଟ͍ • ൒ਫ਼౓ࠞ߹ਫ਼౓͸൚༻ϋʔυˍϑϨʔϜϫʔΫͰ΋αϙʔτ * https://github.com/NVIDIA/apex

8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST & ͷશͯΛྔࢠԽ [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018.

8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST & バイナリ [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018.

2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ *OGFSFODF<> • ਪ࿦࣌ʹVJOUͷԋࢉ͕ϝΠϯͱͳΔΑ͏ʹ ֶश࣌ʹྔࢠԽΛγϛϡϨʔγϣϯ͠ͳ͕Βֶश • 5FOTPS'MPXެࣜʹ࣮૷͕ଘࡏ [97] B. Jacob, et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference," in Proc. of CVPR, 2018. * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/quantize/README.md

まとめ

൚༻తͳߴ଎Խख๏Λ঺հ • ৞ΈࠐΈͷ෼ղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4 • ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO

5BLFIPNF.FTTBHF • ܰྔͳϞσϧʢ৞ΈࠐΈͷ෼ղʣΛ1SVOJOH͢Δͷ͕ खͬऔΓૣ͍ • /"4͕ॸຽͷखʹ • ΞʔΩςΫνϟͱϞσϧͷಉֶ࣌शʢ4JOHMFTIPUԽʣ • '-01TͰ͸ͳ࣮͘σόΠεͰͷ଎౓ΛϑΟʔυόοΫ • ࠓޙ • 1SVOJOHͱ/"4ͷҰମԽ • ࢀߟ • ୈճεςΞϥϘਓ޻஌ೳηϛφʔ ʮ৞ΈࠐΈχϡʔϥϧωοτϫʔΫͷߴਫ਼౓Խͱߴ଎Խʯ

百選

৞ΈࠐΈͷ෼ղ <>-4JGSFBOE4.BMMBU 3PUBUJPO 4DBMJOHBOE%FGPSNBUJPO*OWBSJBOU4DBUUFSJOHGPS5FYUVSF%JTDSJNJOBUJPO JO1SPDPG$713 <>-4JGSF 3JHJENPUJPO4DBUUFSJOHGPS*NBHF$MBTTJpDBUJPO JO1I%UIFTJT <>.-JO 2$IFO BOE4:BO /FUXPSLJO/FUXPSL JO1SPDPG*$-3 <>$4[FHFEZ FUBM 3FUIJOLJOHUIF*ODFQUJPO"SDIJUFDUVSFGPS$PNQVUFS7JTJPO JO1SPDPG$713 <>'/*BOEPMB 4)BO .8.PTLFXJD[ ,"TISBG 8+%BMMZ BOE,,FVU[FS 4RVFF[F/FU"MFY/FUMFWFMBDDVSBDZXJUIYGFXFSQBSBNFUFSTBOE.# NPEFMTJ[F JOBS9JW <>'$IPMMFU 9DFQUJPO%FFQMFBSOJOHXJUIEFQUIXJTFTFQBSBCMFDPOWPMVUJPOT JO1SPDPG$713 <>")PXBSE FUBM .PCJMF/FUT&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF7JTJPO"QQMJDBUJPOT JOBS9JW <>9;IBOH FUBM 4IV⒐F/FU"O&YUSFNFMZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLGPS.PCJMF%FWJDFT JOBS9JW <>#8V FUBM 4IJGU";FSP'-01 ;FSP1BSBNFUFS JOBS9JW <>/.B 9;IBOH );IFOH BOE+4VO 4IV⒐F/FU71SBDUJDBM(VJEFMJOFTGPS&⒏DJFOU$//"SDIJUFDUVSF%FTJHO JO1SPDPG&$$7 <>)(BP ;8BOH BOE4+J $IBOOFM/FUT$PNQBDUBOE&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTWJB$IBOOFM8JTF$POWPMVUJPOT JO1SPDPG/*14 <>()VBOH 4-JV -.BBUFO BOE,8FJOCFSHFS $POEFOTF/FU"O&⒏DJFOU%FOTF/FUVTJOH-FBSOFE(SPVQ$POWPMVUJPOT JO1SPDPG$713 <>.4BOEMFS FUBM .PCJMF/FU7*OWFSUFE3FTJEVBMTBOE-JOFBS#PUUMFOFDLT JO1SPDPG$713 <>(9JF +8BOH 5;IBOH +-BJ 3)POH BOE(2J *($7*OUFSMFBWFE4USVDUVSFE4QBSTF$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713

৞ΈࠐΈͷ෼ղ <>5;IBOH (2J #9JBP BOE+8BOH *OUFSMFBWFEHSPVQDPOWPMVUJPOTGPSEFFQOFVSBMOFUXPSLT JO1SPDPG*$$7 <>;2JO ;;IBOH 9$IFO BOE:1FOH '%.PCJMF/FU*NQSPWFE.PCJMF/FUXJUIB'BTU%PXOTBNQMJOH4USBUFHZ JO1SPDPG*$*1 <>,4VO .-J %-JV BOE+8BOH *($7*OUFSMFBWFE-PX3BOL(SPVQ$POWPMVUJPOTGPS&⒏DJFOU%FFQ/FVSBM/FUXPSLT JO#.7$ <>5)F FUBM #BHPG5SJDLTGPS*NBHF$MBTTJpDBUJPOXJUI$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713 <>:$IFO FUBM %SPQBO0DUBWF3FEVDJOH4QBUJBM3FEVOEBODZJO$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI0DUBWF$POWPMVUJPO JOBS9JW <>")PXBSE FUBM 4FBSDIJOHGPS.PCJMF/FU7 JOBS9JW <>+;IBOH 4FFTBX/FU$POWPMVUJPO/FVSBM/FUXPSL8JUI6OFWFO(SPVQ$POWPMVUJPO JOBS9JW

ࢬמΓ <>:-F$VO +%FOLFS BOE44PMMB 0QUJNBM#SBJO%BNBHF JO1SPDPG/*14 <>4)BO +1PPM +5SBO BOE8%BMMZ -FBSOJOHCPUI8FJHIUTBOE$POOFDUJPOTGPS&⒏DJFOU/FVSBM/FUXPSLT JO1SPDPG/*14 <>88FO $8V :8BOH :$IFO BOE)-J -FBSOJOH4USVDUVSFE4QBSTJUZJO%FFQ/FVSBM/FUXPSLT JO1SPDPG/*14 <>4)BO FUBM %FFQ$PNQSFTTJPO$PNQSFTTJOH%FFQ/FVSBM/FUXPSLTXJUI1SVOJOH 5SBJOFE2VBOUJ[BUJPOBOE)V⒎NBO$PEJOH JO1SPDPG*$-3 <>4)BO +1PPM +5SBO BOE8%BMMZ &*&&⒏DJFOU*OGFSFODF&OHJOFPO$PNQSFTTFE%FFQ/FVSBM/FUXPSL JO1SPDPG*4$" <>4"OXBS ,)XBOH BOE84VOH 4USVDUVSFE1SVOJOHPG%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO+&5$ <>4$IBOHQJOZP .4BOEMFS BOE";INPHJOPW 5IF1PXFSPG4QBSTJUZJO$POWPMVUJPOBM/FVSBM/FUXPSLT JOBS9JW <>44DBSEBQBOF %$PNNJOJFMMP ")VTTBJO BOE"6ODJOJ (SPVQ4QBSTF3FHVMBSJ[BUJPOGPS%FFQ/FVSBM/FUXPSLT JO/FVSPDPNQVUJOH <>)-J FUBM 1SVOJOH'JMUFSTGPS&⒏DJFOU$POW/FUT JO1SPDPG*$-3 <>1.PMDIBOPW 45ZSFF 5,BSSBT 5"JMB BOE+,BVU[ 1SVOJOH$POWPMVUJPOBM/FVSBM/FUXPSLTGPS3FTPVSDF&⒏DJFOU*OGFSFODF JO1SPDPG*$-3 <>%.PMDIBOPW ""TIVLIB BOE%7FUSPW 7BSJBUJPOBM%SPQPVU4QBSTJpFT%FFQ/FVSBM/FUXPSLT JO1SPDPG*$.- <>;-JV FUBM -FBSOJOH&⒏DJFOU$POWPMVUJPOBM/FUXPSLTUISPVHI/FUXPSL4MJNNJOH JO1SPDPG*$$7 <>:)F FUBM $IBOOFM1SVOJOHGPS"DDFMFSBUJOH7FSZ%FFQ/FVSBM/FUXPSLT JO1SPDPG*$$7 <>+-VP FUBM 5IJ/FU"'JMUFS-FWFM1SVOJOH.FUIPEGPS%FFQ/FVSBM/FUXPSL$PNQSFTTJPO JO1SPDPG*$$7 <>$-PVJ[PT ,6MMSJDI BOE.8FMMJOH #BZFTJBO$PNQSFTTJPOGPS%FFQ-FBSOJOH JO1SPDPG/*14

ࢬמΓ <>,JSJMM/FLMZVEPW %NJUSZ.PMDIBOPW "STFOJJ"TIVLIB %NJUSZ7FUSPW 4USVDUVSFE#BZFTJBO1SVOJOHWJB-PH/PSNBM.VMUJQMJDBUJWF/PJTF JO1SPDPG/*14 <>.;IVBOE4(VQUB 5PQSVOF PSOPUUPQSVOFFYQMPSJOHUIFF⒏DBDZPGQSVOJOHGPSNPEFMDPNQSFTTJPO JO1SPDPG*$-38 <>5:BOH :$IFO BOE74[F %FTJHOJOH&OFSHZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLT6TJOH&OFSHZ"XBSF1SVOJOH JO1SPDPG$713 <>:)F (,BOH 9%POH :'V BOE::BOH 4PGU'JMUFS1SVOJOHGPS"DDFMFSBUJOH%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG*+$"* <>:)F FUBM ".$"VUP.-GPS.PEFM$PNQSFTTJPOBOE"DDFMFSBUJPOPO.PCJMF%FWJDFT JO1SPDPG&$$7 <>5:BOH ")PXBSE #$IFO 9;IBOH "(P .4BOEMFS 74[F BOE)"EBN /FU"EBQU1MBUGPSN"XBSF/FVSBM/FUXPSL"EBQUBUJPOGPS.PCJMF "QQMJDBUJPOT JO1SPDPG&$$7 <>+-VPBOE+8V "VUP1SVOFS"O&OEUP&OE5SBJOBCMF'JMUFS1SVOJOH.FUIPEGPS&⒏DJFOU%FFQ.PEFM*OGFSFODF JOBS9JW <>+'SBOLMFBOE.$BSCJO 5IF-PUUFSZ5JDLFU)ZQPUIFTJT'JOEJOH4QBSTF 5SBJOBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3 <>;-JV FUBM 3FUIJOLJOHUIF7BMVFPG/FUXPSL1SVOJOH JO1SPDPG*$-3 <>+:V -:BOH /9V +:BOH BOE5)VBOH 4MJNNBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3 <>4-JO 3+J $:BO #;IBOH -$BP 2:F ')VBOH BOE%%PFSNBOO 5PXBSET0QUJNBM4USVDUVSFE$//1SVOJOHWJB(FOFSBUJWF"EWFSTBSJBM-FBSOJOH JO 1SPDPG$713 ("/ <>+:VBOE 5)VBOH 6OJWFSTBMMZ4MJNNBCMF/FUXPSLTBOE*NQSPWFE5SBJOJOH5FDIOJRVFT JOBS9JW <>+:VBOE 5)VBOH /FUXPSL4MJNNJOHCZ4MJNNBCMF/FUXPSLT5PXBSET0OF4IPU"SDIJUFDUVSF4FBSDIGPS$IBOOFM/VNCFST JOBS9JW <>;-JV ).V 9;IBOH ;(VP 9:BOH 5$IFOH BOE+4VO .FUB1SVOJOH.FUB-FBSOJOHGPS"VUPNBUJD/FVSBM/FUXPSL$IBOOFM1SVOJOH JO BS9JW

ΞʔΩςΫνϟ୳ࡧ <>#;PQIBOE27-F /FVSBMBSDIJUFDUVSFTFBSDIXJUISFJOGPSDFNFOUMFBSOJOH JO1SPDPG*$-3 <>#;PQI 77BTVEFWBO +4IMFOT BOE27-F -FBSOJOHUSBOTGFSBCMFBSDIJUFDUVSFTGPSTDBMBCMFJNBHFSFDPHOJUJPO JO1SPDPG$713 <>$-JV FUBM 1SPHSFTTJWF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG&$$7 <>)1IBN .:(VBO #;PQI 27-F BOE+FGG%FBO &GGJDJFOU/FVSBM"SDIJUFDUVSF4FBSDIWJB1BSBNFUFS4IBSJOH JO1SPDPG*$.- <>)-JV ,4JNPOZBO 07JOZBMT $'FSOBOEP BOE,,BWVLDVPHMV )JFSBSDIJDBM3FQSFTFOUBUJPOTGPS&GGJDJFOU"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3 <>&3FBM ""HHBSXBM :)VBOH 27-F 3FHVMBSJ[FE&WPMVUJPOGPS*NBHF$MBTTJGJFS"SDIJUFDUVSF4FBSDI JO1SPDPG"""* <>)-JV ,4JNPOZBO BOE::BOH %"354%JGGFSFOUJBCMF"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3 <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3 <>.5BO #$IFO 31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27-F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713 <>9%BJ FUBM $IBN/FU5PXBSET&GGJDJFOU/FUXPSL%FTJHOUISPVHI1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713 <>#8V FUBM '#/FU)BSEXBSF"XBSF&GGJDJFOU$POW/FU%FTJHOWJB%JGGFSFOUJBCMF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG$713 <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&GGJDJFOU$POW/FU%FTJHO JO1SPDPG*$.-8 <>--JBOE"5BMXBMLBS 3BOEPNTFBSDIBOESFQSPEVDJCJMJUZGPSOFVSBMBSDIJUFDUVSFTFBSDI JOBS9JW

ૣظऴྃɺಈతܭࢉάϥϑ <>:(VP ":BP BOE:$IFO %ZOBNJD/FUXPSL4VSHFSZGPS&⒏DJFOU%//T JO1SPDPG/*14 <>45FFSBQJUUBZBOPO FUBM #SBODIZ/FU'BTU*OGFSFODFWJB&BSMZ&YJUJOHGSPN%FFQ/FVSBM/FUXPSLT JO1SPDPG*$13 <>.'JHVSOPW FUBM 4QBUJBMMZ"EBQUJWF$PNQVUBUJPO5JNFGPS3FTJEVBM/FUXPSLT JO1SPDPG$713 <>5#PMVLCBTJ +8BOH 0%FLFM BOE74BMJHSBNB "EBQUJWF/FVSBM/FUXPSLTGPS&⒏DJFOU*OGFSFODF JO1SPDPG*$.- <>+-JO FUBM 3VOUJNF/FVSBM1SVOJOH JO1SPDPG/*14 <>()VBOH %$IFO 5-J '8V -.BBUFO BOE,8FJOCFSHFS .VMUJ4DBMF%FOTF/FUXPSLTGPS3FTPVSDF&⒏DJFOU*NBHF$MBTTJpDBUJPO JO1SPDPG *$-3 <>98BOH ':V ;%PV 5%BSSFMM BOE+(PO[BMF[ 4LJQ/FU-FBSOJOH%ZOBNJD3PVUJOHJO$POWPMVUJPOBM/FUXPSLT JO1SPDPG&$$7 <>"7FJUBOE 4#FMPOHJF $POWPMVUJPOBM/FUXPSLTXJUI"EBQUJWF*OGFSFODF(SBQIT JO1SPDPG&$$7 <>--JVBOE+%FOH %ZOBNJD%FFQ/FVSBM/FUXPSLT0QUJNJ[JOH"DDVSBDZ&⒏DJFODZ5SBEF0⒎TCZ4FMFDUJWF&YFDVUJPO JO1SPDPG"""* <>;8V FUBM #MPDL%SPQ%ZOBNJD*OGFSFODF1BUITJO3FTJEVBM/FUXPSLT JO1SPDPG$713 <>3 :V FUBM /*411SVOJOH/FUXPSLTVTJOH/FVSPO*NQPSUBODF4DPSF1SPQBHBUJPO JO1SPDPG$713 <>+,VFO 9,POH ;-JO (8BOH +:JO 44FF BOE:5BO 4UPDIBTUJD%PXOTBNQMJOHGPS$PTU"EKVTUBCMF*OGFSFODFBOE*NQSPWFE3FHVMBSJ[BUJPOJO $POWPMVUJPOBM/FUXPSLT JO1SPDPG$713 <>9(BP :;IBP -%VE[JBL 3.VMMJOT BOE$9V %ZOBNJD$IBOOFM1SVOJOH'FBUVSF#PPTUJOHBOE4VQQSFTTJPO JO1SPDPG*$-3

ৠཹ <>()JOUPO FUBM %JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL JO1SPDPG/*148PSLTIPQ <>+#BBOE3$BSVBOB %P%FFQ/FUT3FBMMZ/FFEUPCF%FFQ JO1SPDPG/*14 <>"3PNFSP FUBM 'JU/FUT)JOUTGPS5IJO%FFQ/FUT JO1SPDPG*$-3 <>5$IFO *(PPEGFMMPX BOE+4IMFOT /FU/FU"DDFMFSBUJOH-FBSOJOHWJB,OPXMFEHF5SBOTGFS JO1SPDPG*$-3 <>(6SCBO FUBM %P%FFQ$POWPMVUJPOBM/FUT3FBMMZ/FFEUPCF%FFQBOE$POWPMVUJPOBM JO1SPDPG*$-3 <>+:JN %+PP +#BF BOE+,JN "(JGUGSPN,OPXMFEHF%JTUJMMBUJPO'BTU0QUJNJ[BUJPO /FUXPSL.JOJNJ[BUJPOBOE5SBOTGFS-FBSOJOH JO1SPDPG $713 <>".JTISBBOE%.BSS "QQSFOUJDF6TJOH,OPXMFEHF%JTUJMMBUJPO5FDIOJRVFT5P*NQSPWF-PX1SFDJTJPO/FUXPSL"DDVSBDZ JO1SPDPG*$-3 <>5'VSMBOFMMP ;-JQUPO .5TDIBOOFO -*UUJ BOE""OBOELVNBS #PSO"HBJO/FVSBM/FUXPSLT JO1SPDPG*$.- <>:;IBOH 59JBOH 5)PTQFEBMFT BOE)-V %FFQ.VUVBM-FBSOJOH JO1SPDPG$713 <>9-BO 9;IV BOE4(POH ,OPXMFEHF%JTUJMMBUJPOCZ0OUIF'MZ/BUJWF&OTFNCMF JO1SPDPG/*14 <>81BSL %,JN :-V BOE.$IP 3FMBUJPOBM,OPXMFEHF%JTUJMMBUJPO JO1SPDPG$713

ྔࢠԽ <>.$PVSCBSJBVY :#FOHJP BOE+%BWJE #JOBSZ$POOFDU5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJOBSZXFJHIUTEVSJOHQSPQBHBUJPOT JO1SPDPG/*14 <>*)VCBSB .$PVSCBSJBVY %4PVESZ 3&M:BOJW BOE:#FOHJP #JOBSJ[FE/FVSBM/FUXPSLT JO1SPDPG/*14 <>.3BTUFHBSJ 70SEPOF[+PTFQI BOE3'BSIBEJ 9/03/FU*NBHF/FU$MBTTJpDBUJPO6TJOH#JOBSZ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG&$$7 <>+8V $-FOH :8BOH 2)V BOE+$IFOH 2VBOUJ[FE$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF%FWJDFT JO1SPDPG$713 <>'-J #;IBOH BOE#-JV 5FSOBSZ8FJHIU/FUXPSLT JOBS9JW <>4;IPV :8V ;/J 9;IPV )8FO BOE:;PV %P3F'B/FU5SBJOJOH-PX#JUXJEUI$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI-PX#JUXJEUI(SBEJFOUT JO BS9JW <>$;IV 4)BO ).BP BOE8%BMMZ 5SBJOFE5FSOBSZ2VBOUJ[BUJPO JO1SPDPG*$-3 <>";IPV ":BP :(VP -9V BOE:$IFO *ODSFNFOUBM/FUXPSL2VBOUJ[BUJPO5PXBSET-PTTMFTT$//TXJUI-PX1SFDJTJPO8FJHIUT JO1SPDPG*$-3 <>48V (-J '$IFO BOE-4IJ 5SBJOJOHBOE*OGFSFODFXJUI*OUFHFSTJO%FFQ/FVSBM/FUXPSLT JO1SPDPG*$-3 <>#+BDPC FUBM 2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ*OGFSFODF JO1SPDPG$713 <>;-JV #8V 8-VP 9:BOH 8-JV BOE,$IFOH #J3FBM/FU&OIBODJOHUIF1FSGPSNBODFPGCJU$//T8JUI*NQSPWFE3FQSFTFOUBUJPOBM$BQBCJMJUZ BOE"EWBODFE5SBJOJOH"MHPSJUIN JO1SPDPG&$$7 <>/8BOH +$IPJ %#SBOE $$IFO BOE,(PQBMBLSJTIOBO 5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJU'MPBUJOH1PJOU/VNCFST JO1SPDPG/*14 <>(:BOH FUBM 48"-14UPDIBTUJD8FJHIU"WFSBHJOHJO-PX1SFDJTJPO5SBJOJOH JO1SPDPG*$.-