メルカリの写真検索を支えるバックエンド CCSE 2019 version

Transcript

1. 1.

   Confidential - Do Not Share 1 ϝϧΧϦͷࣸਅݕࡧΛࢧ͑ΔόοΫΤϯυ גࣜձࣾϝϧΧϦ தՏ ޺จ

2. 2.

   Confidential - Do Not Share 2 • 2017೥7݄ೖࣾ • ॴଐ͸SRE→AI/MLνʔϜ

   • σόΠευϥΠό։ൃ͔ΒϑϩϯτΤ ϯυ։ൃ·Ͱ΍ΔԿͰ΋԰ Twitter: hnakagawa14 GitHub: hnakagawa தՏ ޺จ (Hirofumi Nakagawa) ࣗݾ঺հ 2
3. 3.

   Confidential - Do Not Share 3 Introduction

4. 4.

   Confidential - Do Not Share 4 What is ࣸਅݕࡧ •

   ࣸਅݕࡧͱ͸ɺ͍ΘΏΔը૾ ݕࡧػೳ • ΞϓϦ͔ΒࣸਅΛݩʹ঎඼Λ ݕࡧ͢Δ • ঎඼໊Λ஌Βͳͯ͘΋ը૾͔ Β঎඼ΛݕࡧͰ͖Δ ಈըϦϯΫ: https://youtu.be/kTni8EvOCgI
5. 5.

   Confidential - Do Not Share 5 جຊతͳࣸਅݕࡧͷ࢓૊Έ Deep Neural Networks

   (DNN)Λ࢖༻ͯ͠঎඼ը૾ ͔Βಛ௃ϕΫτϧΛऔಘ औಘͨ͠ಛ௃ϕΫτϧΛ Approximate Nearest Neighbor Index(ANN Index) ʹ௥Ճͯ͠ը૾indexΛߏங ݕࡧ࣌ʹ͸ಉ͘͡঎඼ը૾͔Β DNNΛհͯ͠ಛ௃ϕΫτϧΛऔ ಘ͠ɺANN Index͔Βݕࡧ 2 3 1
6. 6.

   Confidential - Do Not Share 6 What is Kubernetes •

   KubernetesʢҎԼk8sʣͱ͸ Φʔϓϯιʔεͷίϯςφɾ ΦʔέετϨʔγϣϯγες Ϝ • k8sʹ͸Custom Resource Definitionͱݺ͹ΕΔಠࣗͷ ϦιʔεΛఆٛͰ͖Δػೳ͕ ͋Γɺ։ൃऀ͸ͦͷػೳΛհ ͯ͠k8sͷػೳΛ֦ுͰ͖Δ
7. 7.

   Confidential - Do Not Share 7 ML Platform Lykeion •

   Training/Serving CRD & ΧελϜ ίϯτϩʔϥ • ίϯςφϕʔεɾύΠϓϥϯ • Training/Serving ίϯςφΠϝʔ δɾϏϧμʔ • ϞσϧɾϨϙδτϦ ࣸਅݕࡧ͸Lykeionͱݺ͹ΕΔ಺੡ͷ ML Platform্ʹߏங͞Ε͓ͯΓɺԼ هͷػೳ͸PlatformଆͷػೳΛ࢖༻͠ ͍ͯΔ
8. 8.

   Confidential - Do Not Share 8 Architectureɹ

9. 9.

   Confidential - Do Not Share 9 Architecture֓ཁਤ

10. 10.

    Confidential - Do Not Share 10 1.TrainingɾϦιʔεͷ࡞੒

11. 11.

    Confidential - Do Not Share 11 TrainingɾϦιʔεͷ࡞੒ • Training custom

    resourceΛ CronJob͕࡞੒ • ΧελϜɾίϯτϩʔϥ͕ CRDɾϦιʔεͰઃఆ͞Εͨ ίϯςφϕʔεɾύΠϓϥΠ ϯΛ࣮ߦ • ࣮ߦ͢Δόον୯Ґͱͯ͠͸ Hourly, Daily, Monthly͕ଘࡏ
12. 12.

    Confidential - Do Not Share 12 • ֤޻ఔΛݸผͷίϯςφɾΠϝʔδͰ࣮ߦ • ϥΠϒϥϦͷґଘؔ܎ͳͲ؀ڥφΠʔϒͳMLύΠϓϥΠϯͷ໰୊Λղܾ

    • ύΠϓϥΠϯDAG͸YAMLͰهड़ ֤޻ఔͷೖग़ྗ͸Persistent VolumeʢҎ ԼPVʣΛհ͢ ίϯςφϕʔεɾύΠϓϥΠϯ
13. 13.

    Confidential - Do Not Share 13 TrainingɾϦιʔεͷ࡞੒ • શͯͷόον࣮ߦ৘ใ͕ CRDɾϦιʔεͱͯ͠k8s্ʹ

    ࢒Δ • ಉ͡ॲཧΛ࠶࣮ߦग़དྷΔͨ Ίɺόονͷ࠶࣮ߦΛ൐͏ো ֐෮چ࡞ۀ͕༰қ
14. 14.

    Confidential - Do Not Share 14 2.ը૾ͷμ΢ϯϩʔυ

15. 15.

    Confidential - Do Not Share 15 • S3্ʹଘࡏ͢ΔϝϧΧϦɾΠϝʔδετΞ͔Β঎඼ը૾Λμ΢ϯϩʔυ • ύΠϓϥΠϯ্΋ͬͱ΋͕͔͔࣌ؒΔ޻ఔʢը૾਺͕๲େͳͨΊ)

    • ͦͷͨΊPVʹҰఆظؒΩϟογϡ͢ΔࣄʹΑͬͯ࠶ΠϯσοΫε͕ඞཁ ͳ࣌ʹ͸ૉૣ͘ύΠϓϥΠϯΛճͤΔΑ͏ʹ͍ͯ͠Δ ը૾ͷμ΢ϯϩʔυ
16. 16.

    Confidential - Do Not Share 16 3. ΞηοτͷΞοϓϩʔυ

17. 17.

    Confidential - Do Not Share 17 • ύΠϓϥΠϯͷ੒Ռ෺Ͱ͋Δಛ௃ϕΫτϧͱANN IndexΛϞσϧɾϨϙδτ Ϧʹอଘ

    • શͯͷ੒Ռ෺͸όʔδϣϯ؅ཧ͞Εͨঢ়ଶͰอଘ͞ΕΔ • ϞσϧɾϨϙδτϦ͸GCS্ʹߏங ΞηοτͷΞοϓϩʔυ
18. 18.

    Confidential - Do Not Share 18 4.ServingΠϝʔδͷϏϧυ

19. 19.

    Confidential - Do Not Share 19 1. ϞσϧɾϨϙδτϦΛImage Builderͱݺ͹ΕΔdaemon͕؂ࢹ 2.

    ৽͍͠Serving͢΂͖Ϧιʔε͕௥Ճ͞ΕΔͱࣗಈͰServingίϯςφɾΠϝ ʔδΛϏϧυ 3. ίϯςφɾΠϝʔδ͸શͯͷANN Index౳ͷαʔϏϯάʹඞཁͳϦιʔεΛ શؚͯΜͰ͍Δ 4. Ϗϧυ͞ΕͨίϯςφɾΠϝʔδΛίϯςφɾϨδετϦʹϓογϡ ServingΠϝʔδͷϏϧυ
20. 20.

    Confidential - Do Not Share 20 5.ServingϦιʔεͷ࡞੒

21. 21.

    Confidential - Do Not Share 21 ServingɾϦιʔεͷ࡞੒ • Image Builder͸ίϯςφɾΠϝʔ

    δΛϏϧυͨ͋͠ͱɺServingΧε λϜɾϦιʔεΛ࡞੒ • ServingΧελϜɾίϯτϩʔϥ͸ CRDɾϦιʔεͷઃఆΛݩʹඞཁ ͳDeploymentɺService౳Λ࡞੒ • ຊγεςϜͰ͸ߏங͞ΕͨANN IndexΛݸผͷIndexαʔϏεͱ͠ ͯσϓϩΠ
22. 22.

    Confidential - Do Not Share 22 6.ServiceσΟεΧόϦ

23. 23.

    Confidential - Do Not Share 23 αʔϏεɾσΟεΧόϦ • Ϋϥελ্ʹଘࡏ͢ΔIndexαʔϏ εΛk8sΛհͯࣗ͠ಈతʹऔಘ͢

    Δ • ͳΔ΂͘େ͖ͳཻ౓ͷIndexΛ࢖ ༻͢ΔΑ͏ɺҟͳΔظؒɾཻ౓ͷ IndexαʔϏε(Hourly, Daily, Monthly) Λࣗಈతʹ૊Έ߹ΘͤΔ • REST <-> IndexαʔϏεؒͷϓϩ τίϧ͸gRPCΛ࢖༻
24. 24.

    Confidential - Do Not Share 24 7.֓ཁਤৼΓฦΓ

25. 25.

    Confidential - Do Not Share 25 Conclusionɹ

26. 26.

    Confidential - Do Not Share 26 1. ίϯςφɾϕʔεͷ࠶ݱੑͷߴ͍γεςϜ 2. k8sͷCRD/ΧελϜɾίϯτϩʔϥ΍αʔϏεɾσΟεΧόϦ౳ͷػೳΛ׆༻

    3. Batch Execution as Custom Resource౳ɺML PlatformͰ࣮ݱ͞Ε͍ͯΔػ ೳΛ࢖༻͠ɺϩόετͳγεςϜΛߏங 4. Ϋϥ΢υɾΠϯϑϥΛk8sͰந৅Խ͢ΔࣄʹΑͬͯɺ֤Ϋϥ΢υɾϕϯμͷྑ ͍ͱ͜औΓΛ͍ͯ͠Δ ࣸਅݕࡧͷόοΫΤϯυɾΠϯϑϥ
27. 27.

    Confidential - Do Not Share 27 Next Futureɹ

28. 28.

    Confidential - Do Not Share 28 Realtime image search •

    EdgeଆͰobject detection & feature extractionΛߦ͏ • feature vectorΛΫΤϦʹANN Index service͔Βྨࣅ঎඼Λऔಘ • Graph Runtime͸TF Lite + custom operatorΛ࢖༻
29. 29.

    Confidential - Do Not Share 29 Realtime image search demo

    • ݕࡧʹඞཁͳਪ࿦ॲཧͷେ෦෼ΛEdgeଆͰߦ ͍ͬͯΔ • ϦΞϧλΠϜͳΠϯλϥΫγϣϯΛ࣮ݱ • UX্େ͖ͳϝϦοτ͕༗Δ
30. 30.

    Confidential - Do Not Share 30 Thank you all for

    coming todayɹ