メルカリの写真検索を支えるバックエンド CCSE 2019 version

Transcript

1. Confidential - Do Not Share 1 ϝϧΧϦͷࣸਅݕࡧΛࢧ͑ΔόοΫΤϯυ גࣜձࣾϝϧΧϦ தՏ ޺จ

2. Confidential - Do Not Share 2 • 2017೥7݄ೖࣾ • ॴଐ͸SRE→AI/MLνʔϜ

   • σόΠευϥΠό։ൃ͔ΒϑϩϯτΤ ϯυ։ൃ·Ͱ΍ΔԿͰ΋԰ Twitter: hnakagawa14 GitHub: hnakagawa தՏ ޺จ (Hirofumi Nakagawa) ࣗݾ঺հ 2

3. Confidential - Do Not Share 3 Introduction

4. Confidential - Do Not Share 4 What is ࣸਅݕࡧ •

   ࣸਅݕࡧͱ͸ɺ͍ΘΏΔը૾ ݕࡧػೳ • ΞϓϦ͔ΒࣸਅΛݩʹ঎඼Λ ݕࡧ͢Δ • ঎඼໊Λ஌Βͳͯ͘΋ը૾͔ Β঎඼ΛݕࡧͰ͖Δ ಈըϦϯΫ: https://youtu.be/kTni8EvOCgI

5. Confidential - Do Not Share 5 جຊతͳࣸਅݕࡧͷ࢓૊Έ Deep Neural Networks

   (DNN)Λ࢖༻ͯ͠঎඼ը૾ ͔Βಛ௃ϕΫτϧΛऔಘ औಘͨ͠ಛ௃ϕΫτϧΛ Approximate Nearest Neighbor Index(ANN Index) ʹ௥Ճͯ͠ը૾indexΛߏங ݕࡧ࣌ʹ͸ಉ͘͡঎඼ը૾͔Β DNNΛհͯ͠ಛ௃ϕΫτϧΛऔ ಘ͠ɺANN Index͔Βݕࡧ 2 3 1

6. Confidential - Do Not Share 6 What is Kubernetes •

   KubernetesʢҎԼk8sʣͱ͸ Φʔϓϯιʔεͷίϯςφɾ ΦʔέετϨʔγϣϯγες Ϝ • k8sʹ͸Custom Resource Definitionͱݺ͹ΕΔಠࣗͷ ϦιʔεΛఆٛͰ͖Δػೳ͕ ͋Γɺ։ൃऀ͸ͦͷػೳΛհ ͯ͠k8sͷػೳΛ֦ுͰ͖Δ

7. Confidential - Do Not Share 7 ML Platform Lykeion •

   Training/Serving CRD & ΧελϜ ίϯτϩʔϥ • ίϯςφϕʔεɾύΠϓϥϯ • Training/Serving ίϯςφΠϝʔ δɾϏϧμʔ • ϞσϧɾϨϙδτϦ ࣸਅݕࡧ͸Lykeionͱݺ͹ΕΔ಺੡ͷ ML Platform্ʹߏங͞Ε͓ͯΓɺԼ هͷػೳ͸PlatformଆͷػೳΛ࢖༻͠ ͍ͯΔ

8. Confidential - Do Not Share 8 Architectureɹ

9. Confidential - Do Not Share 9 Architecture֓ཁਤ

10. Confidential - Do Not Share 10 1.TrainingɾϦιʔεͷ࡞੒

11. Confidential - Do Not Share 11 TrainingɾϦιʔεͷ࡞੒ • Training custom

    resourceΛ CronJob͕࡞੒ • ΧελϜɾίϯτϩʔϥ͕ CRDɾϦιʔεͰઃఆ͞Εͨ ίϯςφϕʔεɾύΠϓϥΠ ϯΛ࣮ߦ • ࣮ߦ͢Δόον୯Ґͱͯ͠͸ Hourly, Daily, Monthly͕ଘࡏ

12. Confidential - Do Not Share 12 • ֤޻ఔΛݸผͷίϯςφɾΠϝʔδͰ࣮ߦ • ϥΠϒϥϦͷґଘؔ܎ͳͲ؀ڥφΠʔϒͳMLύΠϓϥΠϯͷ໰୊Λղܾ

    • ύΠϓϥΠϯDAG͸YAMLͰهड़ ֤޻ఔͷೖग़ྗ͸Persistent VolumeʢҎ ԼPVʣΛհ͢ ίϯςφϕʔεɾύΠϓϥΠϯ

13. Confidential - Do Not Share 13 TrainingɾϦιʔεͷ࡞੒ • શͯͷόον࣮ߦ৘ใ͕ CRDɾϦιʔεͱͯ͠k8s্ʹ

    ࢒Δ • ಉ͡ॲཧΛ࠶࣮ߦग़དྷΔͨ Ίɺόονͷ࠶࣮ߦΛ൐͏ো ֐෮چ࡞ۀ͕༰қ

14. Confidential - Do Not Share 14 2.ը૾ͷμ΢ϯϩʔυ

15. Confidential - Do Not Share 15 • S3্ʹଘࡏ͢ΔϝϧΧϦɾΠϝʔδετΞ͔Β঎඼ը૾Λμ΢ϯϩʔυ • ύΠϓϥΠϯ্΋ͬͱ΋͕͔͔࣌ؒΔ޻ఔʢը૾਺͕๲େͳͨΊ)

    • ͦͷͨΊPVʹҰఆظؒΩϟογϡ͢ΔࣄʹΑͬͯ࠶ΠϯσοΫε͕ඞཁ ͳ࣌ʹ͸ૉૣ͘ύΠϓϥΠϯΛճͤΔΑ͏ʹ͍ͯ͠Δ ը૾ͷμ΢ϯϩʔυ

16. Confidential - Do Not Share 16 3. ΞηοτͷΞοϓϩʔυ

17. Confidential - Do Not Share 17 • ύΠϓϥΠϯͷ੒Ռ෺Ͱ͋Δಛ௃ϕΫτϧͱANN IndexΛϞσϧɾϨϙδτ Ϧʹอଘ

    • શͯͷ੒Ռ෺͸όʔδϣϯ؅ཧ͞Εͨঢ়ଶͰอଘ͞ΕΔ • ϞσϧɾϨϙδτϦ͸GCS্ʹߏங ΞηοτͷΞοϓϩʔυ

18. Confidential - Do Not Share 18 4.ServingΠϝʔδͷϏϧυ

19. Confidential - Do Not Share 19 1. ϞσϧɾϨϙδτϦΛImage Builderͱݺ͹ΕΔdaemon͕؂ࢹ 2.

    ৽͍͠Serving͢΂͖Ϧιʔε͕௥Ճ͞ΕΔͱࣗಈͰServingίϯςφɾΠϝ ʔδΛϏϧυ 3. ίϯςφɾΠϝʔδ͸શͯͷANN Index౳ͷαʔϏϯάʹඞཁͳϦιʔεΛ શؚͯΜͰ͍Δ 4. Ϗϧυ͞ΕͨίϯςφɾΠϝʔδΛίϯςφɾϨδετϦʹϓογϡ ServingΠϝʔδͷϏϧυ

20. Confidential - Do Not Share 20 5.ServingϦιʔεͷ࡞੒

21. Confidential - Do Not Share 21 ServingɾϦιʔεͷ࡞੒ • Image Builder͸ίϯςφɾΠϝʔ

    δΛϏϧυͨ͋͠ͱɺServingΧε λϜɾϦιʔεΛ࡞੒ • ServingΧελϜɾίϯτϩʔϥ͸ CRDɾϦιʔεͷઃఆΛݩʹඞཁ ͳDeploymentɺService౳Λ࡞੒ • ຊγεςϜͰ͸ߏங͞ΕͨANN IndexΛݸผͷIndexαʔϏεͱ͠ ͯσϓϩΠ

22. Confidential - Do Not Share 22 6.ServiceσΟεΧόϦ

23. Confidential - Do Not Share 23 αʔϏεɾσΟεΧόϦ • Ϋϥελ্ʹଘࡏ͢ΔIndexαʔϏ εΛk8sΛհͯࣗ͠ಈతʹऔಘ͢

    Δ • ͳΔ΂͘େ͖ͳཻ౓ͷIndexΛ࢖ ༻͢ΔΑ͏ɺҟͳΔظؒɾཻ౓ͷ IndexαʔϏε(Hourly, Daily, Monthly) Λࣗಈతʹ૊Έ߹ΘͤΔ • REST <-> IndexαʔϏεؒͷϓϩ τίϧ͸gRPCΛ࢖༻

24. Confidential - Do Not Share 24 7.֓ཁਤৼΓฦΓ

25. Confidential - Do Not Share 25 Conclusionɹ

26. Confidential - Do Not Share 26 1. ίϯςφɾϕʔεͷ࠶ݱੑͷߴ͍γεςϜ 2. k8sͷCRD/ΧελϜɾίϯτϩʔϥ΍αʔϏεɾσΟεΧόϦ౳ͷػೳΛ׆༻

    3. Batch Execution as Custom Resource౳ɺML PlatformͰ࣮ݱ͞Ε͍ͯΔػ ೳΛ࢖༻͠ɺϩόετͳγεςϜΛߏங 4. Ϋϥ΢υɾΠϯϑϥΛk8sͰந৅Խ͢ΔࣄʹΑͬͯɺ֤Ϋϥ΢υɾϕϯμͷྑ ͍ͱ͜औΓΛ͍ͯ͠Δ ࣸਅݕࡧͷόοΫΤϯυɾΠϯϑϥ

27. Confidential - Do Not Share 27 Next Futureɹ

28. Confidential - Do Not Share 28 Realtime image search •

    EdgeଆͰobject detection & feature extractionΛߦ͏ • feature vectorΛΫΤϦʹANN Index service͔Βྨࣅ঎඼Λऔಘ • Graph Runtime͸TF Lite + custom operatorΛ࢖༻

29. Confidential - Do Not Share 29 Realtime image search demo

    • ݕࡧʹඞཁͳਪ࿦ॲཧͷେ෦෼ΛEdgeଆͰߦ ͍ͬͯΔ • ϦΞϧλΠϜͳΠϯλϥΫγϣϯΛ࣮ݱ • UX্େ͖ͳϝϦοτ͕༗Δ

30. Confidential - Do Not Share 30 Thank you all for

    coming todayɹ