Research in Production Environment

Slide 1

Slide 1 text

೐۽؋࣌ ജ҃ীࢲ োҳೞӝ Sungjoo Ha Hyperconnect TensorFlow-KR ࣁ ߣ૩ য়೐ۄੋ ݽ੐ October 20th, 2019 Sungjoo Ha (shurain.net) 1

Slide 2

Slide 2 text

য়ט੄ ੉ঠӝ • ೐۽؋࣌ࢲ࠺झ ઺ੋ/ؼ ઁಿҗ োҳࢿҕ੉ ࠛഛपೠ ӝࣿ ѐߊ੄ ੉ঠӝ • ౱ীࢲ ࣻ೯ೠ Keyword Spo-ng োҳ੄ ৘ઁܳ ઺बਵ۽ • ೞ੉ಌழ֏౟ AI lab ੉ ੌೞח ߑध੄ ੉ঠӝ • ഥࢎ৬ ౱җ ౱ਗ ࢎ੉੄ ೤ਸ ݏ୶ח ੉ঠӝ Sungjoo Ha (shurain.net) 2

Slide 3

Slide 3 text

Hyperconnect Sungjoo Ha (shurain.net) 3

Slide 4

Slide 4 text

Hyperconnect Sungjoo Ha (shurain.net) 4

Slide 5

Slide 5 text

Hyperconnect AI Lab • ӝ҅೟ण ҙ۲ػ সޖ ੹߈੄ ׸׼ • ೐۽ં౟ ࢶ੿ • ؘ੉ఠ ࣻ૘ • ݽ؛ ѐߊ ߂ प೷ • ֤ޙച • ӝദ ଵৈ • ؘ੉ఠ QA • ߓನ Sungjoo Ha (shurain.net) 5

Slide 6

Slide 6 text

2019֙ ୡ • ӝઓ ౱੄ ನழझח ݽ߄ੌ ജ҃ীࢲ पदрਵ۽ ੉޷૑ ׮ܖӝ1 1 h$ps:/ /github.com/hyperconnect/MMNet/ Sungjoo Ha (shurain.net) 6

Slide 7

Slide 7 text

Workshop • 10ѐ ೟ഥ • 3700ಞ ֤ޙ • ਬ੷ פૉ/࠺ૉפझ ౟۪٘ ӝ߈ ই੉٣য ࠳ ۨੋझష߁ • 300о૑ ੿ب੄ ਫ਼੤੸ੋ ഝਊ୊ Ҋ޹ • 1֙ р ۽٘ݗ ҳࢿ Sungjoo Ha (shurain.net) 7

Slide 8

Slide 8 text

Project Selec+on • ׮নೠ ਃࣗܳ Ҋ۰ೞৈ ೐۽ં౟ ࢶ੿ • पഅ оמࢿ • ੐ಂ౟ • ӝࣿ੸ ઺ਃب • ౟۪٘ • Ӓ ઺ ೞաੋ ఃਕ٘ Ѩ୹(keyword spo-ng) Sungjoo Ha (shurain.net) 8

Slide 9

Slide 9 text

Keyword Spo+ng • ౠ੿ ೨बযо ߊࢿغ঻ח૑ Ѩ୹ೞח ޙઁ • ೐۽ં౟ ࢶఖ द Ҋ۰೮؍ ࢎ೦ • بݫੋ ഛ੢ • CV ৻੄ بݫੋਵ۽੄ ഛ੢ • դ੉ب • ࠙ܨ ޙઁח ࢚؀੸ਵ۽ ए਋פ low hanging fruit ੉ۄ ౸ױ • ӝઓ ੹ޙࢿ • ҃۝ ݽ؛ਸ ٜ݅য ݽ߄ੌ ߓನೞח Ѫ਷ ੉޷ ੜ ೞӝ ٸޙী જ਷ ࢿҗܳ Әߑ Ѣل ࣻ ੓ਵܻۄ ౸ ױ Sungjoo Ha (shurain.net) 9

Slide 10

Slide 10 text

Literature Survey Sungjoo Ha (shurain.net) 10

Slide 11

Slide 11 text

Baseline • ࠺Ү ؀࢚੉ غח ݽ؛੉ ੓যঠ ѐࢶ੉ ੄޷ о ੓਺ • ੼૓੸ਵ۽ ࠺Үೡ ࣻ ੓ח ݽ؛ਸ ט۰о ݴ ׮নೠ ஹನք౟ܳ ഛࠁ • ೐۽؋࣌ীࢲח ੉޷ োҳػ ݽ؛ਸ ҳഅೞח Ѫ੉ ୽࠙ೡ ࣻب ੓਺ Sungjoo Ha (shurain.net) 11

Slide 12

Slide 12 text

Baseline Selec+on • ೤ܻ੸ੋ ࢿמ੉ աয়ח૑ (SotA৬੄ ࠺Ү) • ҳഅ դ੉ب ߂ ҕध ௏٘ ҕѐ ৈࠗ • ੤അ੉ ө׮۽਍ ҃਋о ੗઱ ੓਺ • ೐۽؋࣌ীࢲ ੸ਊؼ ઁড ઑѤਸ ঴݃ա ݅ ઒ೞח૑ • ݽ߄ੌ CPUীࢲ पदр ࣻ೯ оמ?2 2 Convolu)onal Neural Networks for Small-Footprint Keyword Spo

Slide 13

Slide 13 text

Data • ҕѐ ؘ੉ఠࣇ • ֤ޙٜ੉ ҕా੸ਵ۽ ࢎਊೞח ؘ੉ఠࣇਸ ୭؀ೠ ഛࠁ • ҕ੿ೠ ࠺Үܳ ೧ঠೣ • ࠛ೙ਃೞѱ ӝઓ ݽ؛ࠁ׮ উ જ਷ ݽ؛ਸ ݅٘חؘী ௾ ֢۱ਸ ӝ਎੉૑ ঋب۾ • ইऔѱب ೟৻ ࣗࣘীѱח ؘ੉ఠܳ ҕѐೞ૑ ঋח ҃਋о ޖୋ ݆਺3 • ࠺ҕѐ ؘ੉ఠࣇ • ղо ҙब੓ח بݫੋীࢲ੄ ݽ؛ ࢿמ਷ ׮ܳ ࣻ ੓਺ • ؘ੉ఠ ࣻ૘ী ؀ೠ Ҋ޹ • য֢ప੉࣌ • ੿೤ࢿ ഛੋ • ؘ੉ఠ ఐ࢝੉ ೙ࣻ੸ 3 য়೑ػ ؘ੉ఠ੉Ҋ ֤ޙ ੘ࢿਸ ਤೠ ਊبۄ ೞ؊ۄب Ѣ੺ೞח ҃਋о ݆਺ Sungjoo Ha (shurain.net) 13

Slide 14

Slide 14 text

PoC • ࢜۽਍ ই੉٣য పझ౟ • Baseline ݽ؛ ѐࢶ • ׳ࢿೞҊ र਷ ݾ಴ܳ ࢸ੿ೞҊ • ୽࠙ೠ ੿ഛب + ݽ߄ੌ CPU पदр • ױ҅੸ਵ۽ ب଱ೡ ࣻ ੓ח ߑߨਸ ࢸ੿ • ୽࠙ೠ ੿ഛب ݢ੷ • ݽ߄ੌ CPU ࣘبח Ӓ ׮਺ • ઺р ࢑୹ޛ • ೐۽؋࣌ী ഝਊؼ ࣻ ੓ח૑ ஖ৌೞѱ Ҋ޹೧ঠ ೣ • ੉ܳ Ҋ۰ೠ ݃ੌझహਸ ੟ইঠ ೣ Sungjoo Ha (shurain.net) 14

Slide 15

Slide 15 text

Process • ݽ؛੉ ઁಿী ੸ਊغӝ ਤ೧ ೙ਃೠ ࠗ࠙ਸ ݽف ٜ݅য ೠ ߄௰ ࢎ੉௿ جܻӝ • بݫੋী ੸೤ೠ ੹୊ܻ ӝߨ ҳഅ • ೟ण ߂ Ѩૐ ౵੉೐ۄੋ ٜ݅ӝ • ٣೒۽੉ܳ ਤೠ Ҋ۰ • ױࣽ োҳ৬ ׳ܻ पઁ ࢲ࠺झ ജ҃ীࢲ੄ ݽ؛ ೯ز নधী ؀ೠ Ҋ޹੉ ೙ ਃ • TF-Lite ഝਊ оמࢿ ١ Sungjoo Ha (shurain.net) 15

Slide 16

Slide 16 text

Evalua&on • ৈ۞ ݽ؛ਸ ҳഅ/࠺Үೡ ٸীח ҕ੿ೞѱ • э਷ ؘ੉ఠ, э਷ ઙܨ੄ যӒݭప੉࣌ ١ਸ ഝਊ೧ঠ • ݽ؛ ୭੸ചೞח ࢎۈ੄ ৉۝ী ٮۄ ׮ܳ ࣻ ੓਺ • ֤ޙ ੤അ द ܻನ౴ػ Ѿҗࠁ׮ જ਷ Ѿҗо ઙઙ ա ১ • ֤ޙীࢲח ஏ੿ೞӝ ਊ੉ೠ ݫ౟ܼਸ ઱۽ ࠁ૑݅ ೐۽ ؋࣌ ജ҃ীࢲח ׮নೠ ݫ౟ܼਸ ࠁইঠ ೡ ࣻب ੓਺ • ੿ഛب vs. दр׼ য়ఐܫ • Flops vs. पઁ latency • ݽ؛੄ ഛनী ٮܲ ੿޻ب৬ ੤അਯ Sungjoo Ha (shurain.net) 16

Slide 17

Slide 17 text

Research • ਗೞח ݾ಴ী ب׳ೞӝө૑ ҅ࣘ೧ࢲ ׮নೠ दب • ࠁా ࢤп؀۽ ੜ غ૑ ঋҊ द೯଱য়ܳ ݆੉ ѻѱ ؽ • ݽف੄ ੋղ৬ ੉೧о ೙ਃೠ दр • ݻ о૑ ੽Ӕ ߑߨ • ܻఠ۞୛ ࢲ߬੉ীࢲ ਬݎ೧ࠁ৓؍ ݽ؛ ੤അೞݶࢲ ই੉٣য ঳ӝ • ׮ܲ بݫੋ੄ ই੉٣য ശ୛য়ӝ • ౱ਗٜҗ੄ షۿ Sungjoo Ha (shurain.net) 17

Slide 18

Slide 18 text

KWS Research Progress • ӝઓ োҳܳ ࠁ׮ࠁפ ResNetਸ ഝਊೠ ҳഅٜ੉ ޷ޑೞѱ য়ܻ૑օ ResNetҗ ׮ܲ ҳઑ • ੉ܳ ࢚ടী ݏѱ ୭؀ೠ ӝઓ ݽ؛җ ࠺तೞѱ ݏ୾ࠁפ Ѿҗо ޷ޑೞѱ જই૗ • ׮݅ ࣘبо ୽࠙൤ ࡅܰ૑ ঋইࢲ ੉ܳ оࣘೡ ߑߨਸ Ҋ޹ • ӝઓী ݽ߄ੌ ஹೊఠ ࠺੹ ৔৉ীࢲ੄ ੹ޙࢿਸ ୭؀ೠ ૑ۮ؀۽ ഝਊ Sungjoo Ha (shurain.net) 18

Slide 19

Slide 19 text

Audio Processing • ઱۽ MFCCܳ ഝਊ • ࢎۈ੉ ੋ૑ೞח ߑधਸ Ҋ۰ೠ ಹܻী ߸ ജ ੿ب۽ ੉೧೧ب ޖߑ • दр-҅ࣻ੄ Ԝ۽ ؘ੉ఠо ߸ജؽ Sungjoo Ha (shurain.net) 19

Slide 20

Slide 20 text

CNN-Based KWS • MFCCܳ 1-଻օ 2D-੉޷૑ ஂә • ӝઓ੄ ࠺੹ ৔৉੄ CNNਸ Ӓ؀۽ ੸ਊ • ޙઁח ୽࠙൤ և਷ recep/ve ﬁeldܳ ы୶ ӝ ਤ೧ࢲח Ө਷ ֎౟ਕ௼ܳ ऺইঠ ೣ Sungjoo Ha (shurain.net) 20

Slide 21

Slide 21 text

Temporal Convolu.on • ੄޷ܳ ࢤп೧ࠁݶ زदী ծ਷ ઱౵ࣻ৬ ֫ ਷ ઱౵ࣻܳ ೣԋ ࠊঠೡ Ѫ э਺ • MFCCܳ ׮઺଻օ 1D-੉޷૑ ஂә • ೠ ߣ੄ ஶࠅܖ࣌ਵ۽ب ݽٚ ೖ୛о recep-ve ﬁeldী ٜয১ • ো࢑۝੄ ஏݶীࢲب ੉ٙ • நद ஘ച੸ੋ ҳઑ Sungjoo Ha (shurain.net) 21

Slide 22

Slide 22 text

TC-ResNet • о߶਍ ResNet ݽ؛ী • Temporal convolu2on ੸ਊ • ই઱ рױೠ ই੉٣য • ౱ীࢲ ݽ߄ੌ CPU ౵੉೐ۄੋਸ ڳযࠁ Ҋ ARM যࣅ࠶ܻ ೐۽Ӓې߁ਸ ೧ࠁݴ ݽ ߄ੌীࢲ ࡅܲ ௏٘ী ؀ೠ ੉೧о ୽࠙൤ Ө঻ӝী औѱ ځৢܾ ࣻ ੓঻਺ Sungjoo Ha (shurain.net) 22

Slide 23

Slide 23 text

Result • ࣘب ࠛޙ ੿ഛب ѐࢶ • ӝઓ ੿ഛب SotA ݽ؛ ؀࠺ 385x ࡅܴ • ӝઓ ࣘب SotA ݽ؛ ؀࠺ 11.5%p ੿ഛب ѐ ࢶ • ߮஖݃௼ ജ҃ ߂ ௏٘ ҕѐ • h(ps:/ /github.com/hyperconnect/TC- ResNet/ • h(ps:/ /arxiv.org/abs/1904.03814 Sungjoo Ha (shurain.net) 23

Slide 24

Slide 24 text

Publishing • ੄޷੓ח Ѿҗח ୭؀ೠ ֤ޙച • ֤ޙਸ ॳݶࢲ ঳ਸ ࣻ ੓ח Ѫ • ਋ܻо ಽҊ੗ ೞח ޙઁо ޖ঺ੋ૑ ݺഛೞѱ ੿੄ೞח Ѫ • খਵ۽ ૓೯೧ঠ ೞח प೷੉ ޖ঺ੋ૑ ঌѱ غח Ѫ • Abla&on పझ౟ ١ਸ ా೧ ࠛ೙ਃೠ ஹನք౟ܳ ੉೧ೞח Ѫ • যରೖ ऀӝҊ ੓যࠊঠ ݻ ׳ ղ۽ ؊ જ਷ ӝࣿ੉ ա১ Sungjoo Ha (shurain.net) 24

Slide 25

Slide 25 text

Abla%on Test • ೐۽؋࣌ োҳ ѐߊ җ੿ীࢲח ݆਷ Ѫٜ੉ ੼૓੸ਵ۽ ੉ܖয૗ • ୭ઙ੸ੋ ݽ؛ীࢲ ৘੹ীח ੄޷о ੓঻ਵա ؊ ੉࢚ ੄޷ হח ࠗ࠙੉ ੓ਸ ࣻ ੓਺ • Abla'on పझ౟۽ ઁѢ • ৌब൤ ٜ݅঻؍ ஹನք౟о ࢎप ߹۽ ॶݽ হ঻׮ח Ѿҗח ޖୋ ൔೞѱ ա১ • Ѿҗ੸ਵ۽ח ೐۽؋࣌ ജ҃ীࢲ ࠛ೙ਃೠ ࠗ࠙ਸ ઁѢೞ޲۽ ੉ٙ Sungjoo Ha (shurain.net) 25

Slide 26

Slide 26 text

Retrospec)on • ೐۽ં౟ द੘ 3ѐਘ റ SotA ӝࣿ ѐߊ ৮ܐ ߂ ೐۽؋࣌ ૓೯ • ӝ҅೟ण ੹ޙࢿ > بݫੋ ੹ޙࢿ੉঻؍ ৘ • ӝઓ੄ ੹ޙࢿਸ ੜ ഝਊೠ ৘द • ݽ߄ੌ CPU पदр-ݽ؛ ୭੸ച Sungjoo Ha (shurain.net) 26

Slide 27

Slide 27 text

Produc'on + Research • ઁ੘ਸ ઺बਵ۽ ೞח ഥࢎ/౱ীࢲ ࢿҕ੸ੋ ӝ҅೟ण ઑ૒ ਍৔ • ࢲ۽੄ ӝ؀ܳ ݏ୾ঠ ೞҊ • ࢲ۽ ਦ-ਦೡ ࣻ ੓ח posi&ve-sum ѱ੐ਸ ٜ݅যঠ ೣ Sungjoo Ha (shurain.net) 27

Slide 28

Slide 28 text

Expecta(on Management • ML Magic • दبೞҊ पಁೡ ࣻ ੓਺ਸ ഥࢎীࢲח ੋ૑೧ঠ ೣ • ౱ীࢲח ܻझ௼ܳ झझ۽ ౸ױೞҊ ਑૒ੌ ࣻ ੓যঠ ೣ • ೞ૑݅ ੹ޙࢿ੉ ੜ ݏח ࠙ঠীࢲח ֥ۄ਍ Ѿҗܳ ױदрী յ ࣻب ੓਺ • ౱਷ ઁಿী ӝৈ ೧ঠೣ • ౱ীࢲ ೧׼ Ҋ޹ਸ ԙ ೧઻ঠ ೣ • ӝ҅೟ण ӝࣿ਷ ডр੄ ߸ഋਸ ా೧ ׮ߑݶਵ۽ ࢎਊؼ оמࢿ੉ ੓਺ • מز੸ਵ۽ ׮ܲ ౱җ ӝࣿ੄ ഝਊী ؀ೠ ੉ঠӝܳ ೧ঠೣ • ࣗ೐౟ਝয ѐߊ۱ + ӝ҅೟ण োҳ۱ Sungjoo Ha (shurain.net) 28

Slide 29

Slide 29 text

Posi%ve-Sum Game • ઁ۽ࢻ ѱ੐੉ա ֎Ѣ౭࠳ࢻ ѱ੐ࠁ׮ח ನ૑౭࠳ࢻ ѱ੐੉ ի׮ • ഥࢎ৬ ౱ ܻ؊о ౠ൤ Ҋ޹೧઻ঠ ೣ • ഥࢎب ౱ب ౱ਗب োҳ੄ ࢿҕ पಁ৬ ޖҙೞѱ ٙਸ ࠅ ࣻ ੓ח ߑߨਸ Ҋ޹೧ঠ ೣ • ઁಿী ٜযт ࣻ ੓ח োҳ • ౱ਗ੄ ࢿ੢җ ழܻয ٣߰܂ݢ౟ী ؀ೠ Ҋ޹ Sungjoo Ha (shurain.net) 29

Slide 30

Slide 30 text

Ownership • ೐۽ં౟੄ Ѿ੿ ߂ ߑೱ ࢸ੿ী ౱ਗٜ੉ ೣԋ ੿ ೣ • ੉ ӝࣿ੉ ഥࢎী ॶݽ ੓ਸө? • োҳܳ ਤೠ োҳח ؀ࠗ࠙੄ ഥࢎীࢲח ࡄ ਸ ߊೞӝ ൨ٝ • ղо ੉ োҳܳ ೞݶ ੤޷੓ਸө? • োҳীࢲ ݄൤ח ҃਋ ੋղबਸ ߊൃೡ ࣻ ੓ ח ੉ਬ • ӝࣿਸ о੢ ੜ ੉೧ೞҊ ੓ח Ѫ਷ োҳ੗ ࠄੋ • ࢎղ ׮ܲ ౱ٜҗ ૑ࣘ੸ਵ۽ ੉ঠӝ ೧ঠೣ Sungjoo Ha (shurain.net) 30

Slide 31

Slide 31 text

We Are Hiring!4 • Mobile Deep Learning • ML Pla1orm So4ware Engineering • Computer Vision • Speech Recogni?on • Natural Language • Genera?ve Modeling • Recommender Systems 4 h$ps:/ /hyperconnect.com/career/ [email protected] Sungjoo Ha (shurain.net) 31