メルカリの写真検索を支えるバックエンド CCSE 2019 version

メルカリの写真検索を支えるバックエンド CCSE 2019 version

過去発表分との差分としてEdge関連のデモが追加されています。
デモ動画: https://youtu.be/UsNMDuZyaJU

Transcript

  1. 2.

    Confidential - Do Not Share 2 • 2017೥7݄ೖࣾ • ॴଐ͸SRE→AI/MLνʔϜ

    • σόΠευϥΠό։ൃ͔ΒϑϩϯτΤ ϯυ։ൃ·Ͱ΍ΔԿͰ΋԰ Twitter: hnakagawa14 GitHub: hnakagawa தՏ ޺จ (Hirofumi Nakagawa) ࣗݾ঺հ 2
  2. 4.

    Confidential - Do Not Share 4 What is ࣸਅݕࡧ •

    ࣸਅݕࡧͱ͸ɺ͍ΘΏΔը૾ ݕࡧػೳ • ΞϓϦ͔ΒࣸਅΛݩʹ঎඼Λ ݕࡧ͢Δ • ঎඼໊Λ஌Βͳͯ͘΋ը૾͔ Β঎඼ΛݕࡧͰ͖Δ ಈըϦϯΫ: https://youtu.be/kTni8EvOCgI
  3. 5.

    Confidential - Do Not Share 5 جຊతͳࣸਅݕࡧͷ࢓૊Έ Deep Neural Networks

    (DNN)Λ࢖༻ͯ͠঎඼ը૾ ͔Βಛ௃ϕΫτϧΛऔಘ औಘͨ͠ಛ௃ϕΫτϧΛ Approximate Nearest Neighbor Index(ANN Index) ʹ௥Ճͯ͠ը૾indexΛߏங ݕࡧ࣌ʹ͸ಉ͘͡঎඼ը૾͔Β DNNΛհͯ͠ಛ௃ϕΫτϧΛऔ ಘ͠ɺANN Index͔Βݕࡧ 2 3 1
  4. 6.

    Confidential - Do Not Share 6 What is Kubernetes •

    KubernetesʢҎԼk8sʣͱ͸ Φʔϓϯιʔεͷίϯςφɾ ΦʔέετϨʔγϣϯγες Ϝ • k8sʹ͸Custom Resource Definitionͱݺ͹ΕΔಠࣗͷ ϦιʔεΛఆٛͰ͖Δػೳ͕ ͋Γɺ։ൃऀ͸ͦͷػೳΛհ ͯ͠k8sͷػೳΛ֦ுͰ͖Δ
  5. 7.

    Confidential - Do Not Share 7 ML Platform Lykeion •

    Training/Serving CRD & ΧελϜ ίϯτϩʔϥ • ίϯςφϕʔεɾύΠϓϥϯ • Training/Serving ίϯςφΠϝʔ δɾϏϧμʔ • ϞσϧɾϨϙδτϦ ࣸਅݕࡧ͸Lykeionͱݺ͹ΕΔ಺੡ͷ ML Platform্ʹߏங͞Ε͓ͯΓɺԼ هͷػೳ͸PlatformଆͷػೳΛ࢖༻͠ ͍ͯΔ
  6. 11.

    Confidential - Do Not Share 11 TrainingɾϦιʔεͷ࡞੒ • Training custom

    resourceΛ CronJob͕࡞੒ • ΧελϜɾίϯτϩʔϥ͕ CRDɾϦιʔεͰઃఆ͞Εͨ ίϯςφϕʔεɾύΠϓϥΠ ϯΛ࣮ߦ • ࣮ߦ͢Δόον୯Ґͱͯ͠͸ Hourly, Daily, Monthly͕ଘࡏ
  7. 12.

    Confidential - Do Not Share 12 • ֤޻ఔΛݸผͷίϯςφɾΠϝʔδͰ࣮ߦ • ϥΠϒϥϦͷґଘؔ܎ͳͲ؀ڥφΠʔϒͳMLύΠϓϥΠϯͷ໰୊Λղܾ

    • ύΠϓϥΠϯDAG͸YAMLͰهड़ ֤޻ఔͷೖग़ྗ͸Persistent VolumeʢҎ ԼPVʣΛհ͢ ίϯςφϕʔεɾύΠϓϥΠϯ
  8. 13.

    Confidential - Do Not Share 13 TrainingɾϦιʔεͷ࡞੒ • શͯͷόον࣮ߦ৘ใ͕ CRDɾϦιʔεͱͯ͠k8s্ʹ

    ࢒Δ • ಉ͡ॲཧΛ࠶࣮ߦग़དྷΔͨ Ίɺόονͷ࠶࣮ߦΛ൐͏ো ֐෮چ࡞ۀ͕༰қ
  9. 15.

    Confidential - Do Not Share 15 • S3্ʹଘࡏ͢ΔϝϧΧϦɾΠϝʔδετΞ͔Β঎඼ը૾Λμ΢ϯϩʔυ • ύΠϓϥΠϯ্΋ͬͱ΋͕͔͔࣌ؒΔ޻ఔʢը૾਺͕๲େͳͨΊ)

    • ͦͷͨΊPVʹҰఆظؒΩϟογϡ͢ΔࣄʹΑͬͯ࠶ΠϯσοΫε͕ඞཁ ͳ࣌ʹ͸ૉૣ͘ύΠϓϥΠϯΛճͤΔΑ͏ʹ͍ͯ͠Δ ը૾ͷμ΢ϯϩʔυ
  10. 17.

    Confidential - Do Not Share 17 • ύΠϓϥΠϯͷ੒Ռ෺Ͱ͋Δಛ௃ϕΫτϧͱANN IndexΛϞσϧɾϨϙδτ Ϧʹอଘ

    • શͯͷ੒Ռ෺͸όʔδϣϯ؅ཧ͞Εͨঢ়ଶͰอଘ͞ΕΔ • ϞσϧɾϨϙδτϦ͸GCS্ʹߏங ΞηοτͷΞοϓϩʔυ
  11. 19.

    Confidential - Do Not Share 19 1. ϞσϧɾϨϙδτϦΛImage Builderͱݺ͹ΕΔdaemon͕؂ࢹ 2.

    ৽͍͠Serving͢΂͖Ϧιʔε͕௥Ճ͞ΕΔͱࣗಈͰServingίϯςφɾΠϝ ʔδΛϏϧυ 3. ίϯςφɾΠϝʔδ͸શͯͷANN Index౳ͷαʔϏϯάʹඞཁͳϦιʔεΛ શؚͯΜͰ͍Δ 4. Ϗϧυ͞ΕͨίϯςφɾΠϝʔδΛίϯςφɾϨδετϦʹϓογϡ ServingΠϝʔδͷϏϧυ
  12. 21.

    Confidential - Do Not Share 21 ServingɾϦιʔεͷ࡞੒ • Image Builder͸ίϯςφɾΠϝʔ

    δΛϏϧυͨ͋͠ͱɺServingΧε λϜɾϦιʔεΛ࡞੒ • ServingΧελϜɾίϯτϩʔϥ͸ CRDɾϦιʔεͷઃఆΛݩʹඞཁ ͳDeploymentɺService౳Λ࡞੒ • ຊγεςϜͰ͸ߏங͞ΕͨANN IndexΛݸผͷIndexαʔϏεͱ͠ ͯσϓϩΠ
  13. 23.

    Confidential - Do Not Share 23 αʔϏεɾσΟεΧόϦ • Ϋϥελ্ʹଘࡏ͢ΔIndexαʔϏ εΛk8sΛհͯࣗ͠ಈతʹऔಘ͢

    Δ • ͳΔ΂͘େ͖ͳཻ౓ͷIndexΛ࢖ ༻͢ΔΑ͏ɺҟͳΔظؒɾཻ౓ͷ IndexαʔϏε(Hourly, Daily, Monthly) Λࣗಈతʹ૊Έ߹ΘͤΔ • REST <-> IndexαʔϏεؒͷϓϩ τίϧ͸gRPCΛ࢖༻
  14. 26.

    Confidential - Do Not Share 26 1. ίϯςφɾϕʔεͷ࠶ݱੑͷߴ͍γεςϜ 2. k8sͷCRD/ΧελϜɾίϯτϩʔϥ΍αʔϏεɾσΟεΧόϦ౳ͷػೳΛ׆༻

    3. Batch Execution as Custom Resource౳ɺML PlatformͰ࣮ݱ͞Ε͍ͯΔػ ೳΛ࢖༻͠ɺϩόετͳγεςϜΛߏங 4. Ϋϥ΢υɾΠϯϑϥΛk8sͰந৅Խ͢ΔࣄʹΑͬͯɺ֤Ϋϥ΢υɾϕϯμͷྑ ͍ͱ͜औΓΛ͍ͯ͠Δ ࣸਅݕࡧͷόοΫΤϯυɾΠϯϑϥ
  15. 28.

    Confidential - Do Not Share 28 Realtime image search •

    EdgeଆͰobject detection & feature extractionΛߦ͏ • feature vectorΛΫΤϦʹANN Index service͔Βྨࣅ঎඼Λऔಘ • Graph Runtime͸TF Lite + custom operatorΛ࢖༻
  16. 29.

    Confidential - Do Not Share 29 Realtime image search demo

    • ݕࡧʹඞཁͳਪ࿦ॲཧͷେ෦෼ΛEdgeଆͰߦ ͍ͬͯΔ • ϦΞϧλΠϜͳΠϯλϥΫγϣϯΛ࣮ݱ • UX্େ͖ͳϝϦοτ͕༗Δ