Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
メルカリ写真検索における Amazon EKS の活用事例
Search
Hirofumi Nakagawa/中河 宏文
June 14, 2019
Technology
6
29k
メルカリ写真検索における Amazon EKS の活用事例
AWS Summit Tokyo 2019
Hirofumi Nakagawa/中河 宏文
June 14, 2019
Tweet
Share
More Decks by Hirofumi Nakagawa/中河 宏文
See All by Hirofumi Nakagawa/中河 宏文
IoTデバイスでMLモデルを動かす技術
hnakagawa
0
180
Kanazawa_AI.pdf
hnakagawa
0
190
メルカリ写真検索における Amazon EKS の活用事例と プロダクトにおけるEdgeAI technologyの展望
hnakagawa
5
9k
メルカリの写真検索を支えるバックエンド CCSE 2019 version
hnakagawa
0
320
メルカリの写真検索を支えるバックエンド
hnakagawa
1
1.2k
Mercari ML Platform
hnakagawa
1
17k
mlct.pdf
hnakagawa
2
2.1k
機械学習によるマーケット健全化施策を支える技術
hnakagawa
0
240
メルカリのマーケット健全化施策を支えるML基盤
hnakagawa
10
9.1k
Other Decks in Technology
See All in Technology
o11yで育てる、強い内製開発組織
_awache
3
100
それでも私はContextに値を詰めたい | Go Conference 2025 / go conference 2025 fill context
budougumi0617
4
1.1k
職種別ミートアップで社内から盛り上げる アウトプット文化の醸成と関係強化/ #DevRelKaigi
nishiuma
2
120
Windows で省エネ
murachiakira
0
150
C# 14 / .NET 10 の新機能 (RC 1 時点)
nenonaninu
1
1.4k
"複雑なデータ処理 × 静的サイト" を両立させる、楽をするRails運用 / A low-effort Rails workflow that combines “Complex Data Processing × Static Sites”
hogelog
3
1.7k
組織観点からIAM Identity CenterとIAMの設計を考える
nrinetcom
PRO
1
150
AI時代だからこそ考える、僕らが本当につくりたいスクラムチーム / A Scrum Team we really want to create in this AI era
takaking22
6
2.8k
Modern_Data_Stack最新動向クイズ_買収_AI_激動の2025年_.pdf
sagara
0
180
いま注目しているデータエンジニアリングの論点
ikkimiyazaki
0
570
20250929_QaaS_vol20
mura_shin
0
110
Green Tea Garbage Collector の今
zchee
PRO
2
380
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Agile that works and the tools we love
rasmusluckow
331
21k
Making Projects Easy
brettharned
118
6.4k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Designing Experiences People Love
moore
142
24k
Balancing Empowerment & Direction
lara
4
670
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Six Lessons from altMBA
skipperchong
28
4k
A better future with KSS
kneath
239
17k
Transcript
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ϝϧΧϦࣸਅݕࡧʹ͓͚Δ Amazon EKS ͷ׆༻ࣄྫ ࣸਅݕࡧΛࢧ͑ΔόοΫΤϯυ !1 גࣜձࣾϝϧΧϦ தՏ จ
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. தՏ จ • 20177݄ೖࣾ • ॴଐSRE→AI/MLνʔϜ • σόΠευϥΠό։ൃ͔ΒϑϩϯτΤϯυ ։ൃ·ͰΔԿͰ Twitter: hnakagawa14 GitHub: hnakagawa !2 ࣗݾհ
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Introduction !3
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. What is ࣸਅݕࡧ • ࣸਅݕࡧͱɺ͍ΘΏΔը૾ݕࡧػೳ • ΞϓϦ͔ΒࣸਅΛݩʹΛݕࡧ͢Δ • ໊ΛΒͳͯ͘ը૾͔ΒΛݕࡧ Ͱ͖Δ !4 ಈըϦϯΫ: https://youtu.be/kTni8EvOCgI
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. جຊతͳࣸਅݕࡧͷΈ !5 Deep Neural Networks (DNN)Λ༻ͯ͠ը૾ ͔ΒಛϕΫτϧΛऔಘ औಘͨ͠ಛϕΫτϧΛ Approximate Nearest Neighbor Index(ANN Index) ʹՃͯ͠ը૾indexΛߏங ݕࡧ࣌ʹಉ͘͡ը૾͔Β DNNΛհͯ͠ಛϕΫτϧΛऔ ಘ͠ɺANN Index͔Βݕࡧ 2 3 1
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. What is Kubernetes • KubernetesʢҎԼk8sʣͱΦʔϓϯιʔε ͷίϯςφɾΦʔέετϨʔγϣϯγες Ϝ • k8sʹCustom Resource Definitionͱݺ ΕΔಠࣗͷϦιʔεΛఆٛͰ͖Δػೳ͕͋ Γɺ։ൃऀͦͷػೳΛհͯ͠k8sͷػೳΛ ֦ுͰ͖Δ • Amazon Elastic Container Service for Kubernetes (Amazon EKS) ͱk8sͷϚω ʔδυɾαʔϏεɺίϯτϩʔϧϓϨʔϯ ͷཧΛߦͬͯ͘ΕΔ !6
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. What is Custom Resource Definition • Custom Resource DefinitionʢҎԼCRDʣͱ ಠࣗʹϦιʔεΛఆٛͰ͖Δk8sͷػೳ • CRDɾϦιʔεͱɺΧελϜɾίϯτϩʔ ϥͰߏ͞ΕΔ • ΧελϜɾίϯτϩʔϥ͕CRDɾϦιʔε ͷϥΠϑαΠΫϧ/ঢ়ଶʹԠͯ͡Ϋϥελͷ ঢ়ଶΛίϯτϩʔϧ͢Δ !7
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ML Platform Lykeion ࣸਅݕࡧLykeionͱݺΕΔͷML Platform্ ʹߏங͞Ε͓ͯΓɺԼهͷػೳPlatformଆͷػೳ Λ༻͍ͯ͠Δ !8 • Training/Serving CRD & ΧελϜίϯτϩʔϥ • ίϯςφϕʔεɾύΠϓϥΠϯ • Training/Serving ίϯςφΠϝʔδɾϏϧμʔ • ϞσϧɾϨϙδτϦ
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Architecture !9
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Architecture֓ཁਤ !10 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 1.TrainingɾϦιʔεͷ࡞ !11 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. TrainingɾϦιʔεͷ࡞ • Training custom resourceΛCronJob͕࡞ • ΧελϜɾίϯτϩʔϥ͕CRDɾϦιʔε Ͱઃఆ͞ΕͨίϯςφϕʔεɾύΠϓϥΠ ϯΛ࣮ߦ • ࣮ߦ͢Δόον୯Ґͱͯ͠Hourly, Daily, Monthly͕ଘࡏ !12
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ίϯςφϕʔεɾύΠϓϥΠϯ • ֤ఔΛݸผͷίϯςφɾΠϝʔδͰ࣮ߦ • ϥΠϒϥϦͷґଘؔͳͲڥφΠʔϒͳMLύΠϓϥΠϯͷΛղܾ • ύΠϓϥΠϯDAGYAMLͰهड़ • ֤ఔͷೖग़ྗPersistent VolumeʢҎԼPVʣΛհ͢ !13
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Batch Execution as Custom Resource • શͯͷόον࣮ߦใ͕CRDɾϦιʔεͱͯ͠ k8s্ʹΔ • ಉ͡ॲཧΛ࠶࣮ߦग़དྷΔͨΊɺόονͷ࠶࣮ߦ Λ͏ো෮چ࡞ۀ͕༰қ !14
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 2.ը૾ͷμϯϩʔυ !15 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ը૾ͷμϯϩʔυ • Amazon S3্ʹଘࡏ͢ΔϝϧΧϦɾΠϝʔδετΞ͔Βը૾Λμ ϯϩʔυ • ύΠϓϥΠϯ্ͬͱ͕͔͔࣌ؒΔఔʢը૾͕େͳͨΊ) • ͦͷͨΊPVʹҰఆظؒΩϟογϡ͢ΔࣄʹΑͬͯ࠶ΠϯσοΫε͕ ඞཁͳ࣌ʹૉૣ͘ύΠϓϥΠϯΛճͤΔΑ͏ʹ͍ͯ͠Δ • PVͷ࣮ମ Amazon EBS !16
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 3.ΞηοτͷΞοϓϩʔυ !17 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ΞηοτͷΞοϓϩʔυ • ύΠϓϥΠϯͷՌͰ͋ΔಛϕΫτϧͱANN IndexΛϞσϧɾϨϙδτϦʹอଘ • શͯͷՌόʔδϣϯཧ͞Εͨঢ়ଶͰอଘ͞ΕΔ • ϞσϧɾϨϙδτϦGCS্ʹߏங !18
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 4.ServingΠϝʔδͷϏϧυ !19 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ServingΠϝʔδͷϏϧυ 1. ϞσϧɾϨϙδτϦΛImage BuilderͱݺΕΔdaemon͕ࢹ 2. ৽͍͠Serving͖͢Ϧιʔε͕Ճ͞ΕΔͱࣗಈͰServingίϯςφɾΠϝʔδΛϏϧυ • ίϯςφɾΠϝʔδશͯͷANN IndexͷαʔϏϯάʹඞཁͳϦιʔεΛશؚͯΜͰ͍Δ 3. Ϗϧυ͞ΕͨίϯςφɾΠϝʔδΛίϯςφɾϨδετϦʹϓογϡ !20
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 5.ServingɾϦιʔεͷ࡞ !21 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ServingɾϦιʔεͷ࡞ • Image BuilderίϯςφɾΠϝʔδΛϏ ϧυͨ͋͠ͱɺServingΧελϜɾϦιʔ εΛ࡞ • ServingΧελϜɾίϯτϩʔϥCRDɾ ϦιʔεͷઃఆΛݩʹඞཁͳ DeploymentɺServiceΛ࡞ • ຊγεςϜͰߏங͞ΕͨANN IndexΛ ݸผͷIndexαʔϏεͱͯ͠σϓϩΠ !22
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. 6.αʔϏεɾσΟεΧόϦ !23 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. αʔϏεɾσΟεΧόϦ • Ϋϥελ্ʹଘࡏ͢ΔIndexαʔϏεΛ k8sΛհͯࣗ͠ಈతʹऔಘ͢Δ • ͳΔ͘େ͖ͳཻͷIndexΛ༻͢ΔΑ ͏ɺҟͳΔظؒɾཻͷIndexαʔϏε (Hourly, Daily, Monthly) ΛࣗಈతʹΈ߹ ΘͤΔ • REST <-> IndexαʔϏεؒͷϓϩτίϧ gRPCΛ༻ !24
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ֓ཁਤͷৼΓฦΓ !25 S3 EKS
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Conclusion !26
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. ࣸਅݕࡧͷόοΫΤϯυɾΠϯϑϥ 1. ίϯςφɾϕʔεͷ࠶ݱੑͷߴ͍γεςϜ 2. k8sͷCRD/ΧελϜɾίϯτϩʔϥαʔϏεɾσΟεΧόϦͷػೳΛ׆༻ 3. Batch Execution as Custom ResourceɺML PlatformͰ࣮ݱ͞Ε͍ͯΔػೳΛ༻ ͠ɺϩόετͳγεςϜΛߏங 4. ΫϥυɾΠϯϑϥΛk8sͰநԽ͢ΔࣄʹΑͬͯɺ֤Ϋϥυɾϕϯμͷྑ͍ͱ͜औΓ Λ͍ͯ͠Δ !27
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Amazon EKSʹ͍ͭͯ • k8sͱ͍͑GCPͱ͍͏Πϝʔδ͕͋Δ͕(։ൃݩ͔ͩΒવ)ɺEKS͜ͳΕ͖ͯͯ҆ఆͨ͠ӡ ༻͕ग़དྷΔΑ͏ʹͳͬͨɻ͔͠͠Pros/Consવଘࡏ͢Δɻ • Pros • ૉͷk8sʹۙ͘ɺͦ͏͍ͬͨҙຯͰॊೈͳߏΛΈ͍͢ • Cons • ଞࣾͷϚωʔδυk8sαʔϏεʹൺɺk8sʹ͍ͭͯඞཁͳࣄલ͕ࣝଟ͍ • ͔͍͠͠উखͷྑ͍AWSͷS3RDSͷଞαʔϏεΛɺk8sͱڞʹ͍͍ͨϢʔβ͕᪳͢ Δཧ༝ͳ͍ !28
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. AWSͷྑ͍ͱ͜Ζ • AWSͷྑ͍ॴԿͱݴͬͯRDSɺAuroraʹද͞ΕΔɺϚωʔδυɾRDBMSαʔϏε • ࠷ॳRDSͰ࡞ͬͯɺύϑΥʔϚϯεʹࠔͬͨΒAuroraΛ͏ • Persistent͕ॏཁͳɺಛʹRDBΛ༻͢ΔγεςϜͰɺ࠷ॳͷબࢶʹͳΔ • ࠓճͷߏྫͱগ͠ҧ͏͕ɺk8s(EKS)ͰϏδωεɾϩδοΫ͕ͷΔΠϯϑϥ෦Λ நԽ͠PersistentʹRDSͷαʔϏεΛ༻͢Δͷ͓͢͢Ίߏͷ̍ͭ !29
© 2019, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Thank you all for coming today !30