Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Microservices on Multi-Cloud
Search
kazeburo
March 23, 2018
Technology
21
6k
Microservices on Multi-Cloud
MANABIYA TERATAIL DEVELOPER DAYS 2018-03-23
kazeburo
March 23, 2018
Tweet
Share
More Decks by kazeburo
See All by kazeburo
クラウド開発の舞台裏とSRE文化の醸成 / SRE NEXT 2025 Lunch Session
kazeburo
1
1.3k
さくらのクラウド 開発の挑戦とその舞台裏
kazeburo
1
1k
[SRE kaigi 2025] ガバメントクラウドに向けた開発と変化するSRE組織のあり方 / Development for Government Cloud and the Evolving Role of SRE Teams
kazeburo
4
3.6k
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
7.4k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
3.3k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.4k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
14k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.5k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
750
Other Decks in Technology
See All in Technology
OAuthからOIDCへ ― 認可の仕組みが認証に拡張されるまで
yamatai1212
0
140
Introdução a Service Mesh usando o Istio
aeciopires
1
250
コンテキストエンジニアリング入門〜AI Coding Agent作りで学ぶ文脈設計〜
kworkdev
PRO
3
2k
Copilot Studio ハンズオン - 生成オーケストレーションモード
tomoyasasakimskk
0
160
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
AIフル活用で挑む!空間アプリ開発のリアル
taat
0
110
私のMCPの使い方
tsubakimoto_s
0
110
Claude Codeを駆使した初めてのiOSアプリ開発 ~ゼロから3週間でグローバルハッカソンで入賞するまで~
oikon48
10
5.2k
CREが作る自己解決サイクルSlackワークフローに組み込んだAIによる社内ヘルプデスク改革 #cre_meetup
bengo4com
0
100
名刺メーカーDevグループ 紹介資料
sansan33
PRO
0
940
Oracle Autonomous AI Database:サービス概要のご紹介
oracle4engineer
PRO
2
15k
新規事業におけるGORM+SQLx併用アーキテクチャ
hacomono
PRO
0
440
Featured
See All Featured
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Building Applications with DynamoDB
mza
96
6.7k
Thoughts on Productivity
jonyablonski
70
4.9k
Designing for humans not robots
tammielis
254
26k
Learning to Love Humans: Emotional Interface Design
aarron
274
41k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
31
2.7k
Building an army of robots
kneath
306
46k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Statistics for Hackers
jakevdp
799
220k
The Pragmatic Product Professional
lauravandoore
36
7k
Music & Morning Musume
bryan
46
6.9k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.7k
Transcript
Microservices on Multi-Cloud Masahiro @kazeburo Nagano MANABIYA Teratail developer days
2018/03/23
Me • խ • @kazeburo • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability
Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ • झຯDBͷ Restore
Agenda • ϝϧΧϦʹ͍ͭͯ • ϝϧΧϦͷ Infrastructure History #1 - Multi-Cloud
• ϝϧΧϦͷ Infrastructure History #2 - Microservices on Multi-Cloud • Microservices on Multi-Cloud ͷ՝
None
ϝϧΧϦ • ຊ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ
ถࠃ/ӳࠃ ͷల։ JP UK US
KPI μϯϩʔυ GMV(૯औҾֹ) 1ԯDLҎ্(JP+US+UK) ݄ؒ100ԯԁҎ্ ग़ 1100ສҎ্
γεςϜ֓ཁ ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services
LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ը૾ ܾࡁ AI ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏
Infrastructure
Infrastructure in 2017 DNS: Amazon Route53 CDN: Akamai, CloudFront Storage:
Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel JP UK US
Infrastructure in 2018 DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux(JP)
Storage: Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog JP UK US + +
Infrastructure History #1 2013 - 2017 / Multi-Cloud
Infrastructure History (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷʮ͘͞ΒͷVPSʯ1ʹWebDBࡌͤͨߏͰ։࢝ •
Infrastructure ઐऀ͕͍ͳ͍தͰɺ։ൃऀʹۙͳج൫Λબ • ϦϦʔεޙ2ϲ݄Ͱʮ͘͞ΒͷΫϥυʯʮઐ༻αʔόʯҠߦ
ʮ͘͞Βͷઐ༻αʔόʯ • Metal as a Service • ཧαʔόΛΫϥυͷΑ͏ʹѻ͑Δ • ཧαʔόͳΒͰͷύϑΥʔϚϯε
• ωοτϫʔΫͱϋʔυΣΞͷอक ͘͞ΒΠϯλʔωοτ༷͕୲ • ʮ͘͞ΒͷΫϥυʯͱଓ͕Մೳ • ίετύϑΥʔϚϯεʹ༏ΕΔ
Infrastructure History (2) • 2014/09 US ϦϦʔε • AWS (Oregon)
ʹͯαʔϏεߏங • JPϦϦʔε͔Β͠Β͘ܦͪɺ։ൃऀʹAWSܦݧऀ͕૿Ճ • ͦΕͰ Infrastructure ઐऀগͳ͘ɺRDSElastiCacheϚωʔδυαʔϏεΛ ར༻ͯ͠αʔϏεΛߏங • USࠃͷ MaaS Λݕ౼͕ͨ͠ɺUSͰͷαʔϏεͷ༧͕͘͠ɺΫϥυͷॊ ೈ͞Λ JP ΑΓॏཁࢹ
Infrastructure History (3) • 2015/11 SREνʔϜൃ • JP/US ͷΞʔΩςΫνϟΛվળ͠ɺαʔϏεͷ৴པੑͱεέʔϥϏϦςΟͷ ্ʹͱऔΓΉ
• 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़ͱͯ͠ʮGCPʯ্ͰαʔϏεΛߏங
Multi-Cloud in 2017/03 JP UK US ઐ༻αʔό EC2 GCE IaaS
Λத৺ͱͨ͠ Multi-Cloud (Hybrid Cloud) ͨͩ͠ɺͦΕͧΕͷαʔϏεΛΈΔͱ୯ಠͷCloudΛར༻
Multi-Cloud Operations • ՄೳͳݶΓڞ௨ͷΞʔΩςΫνϟΛ࠾༻ • ଞͷΫϥυʹଘࡏ͠ͳ͍ϚωʔδυαʔϏεͷϦϓϨΠε • Consul/Local DNSͷಋೖ •
ΦϖϨʔγϣϯͷڞ௨ԽɾগਓͰͷӡ༻ͷ࣮ݱ • JP ͷنͰ࣮ͷ͋ΔߏɻUS AppStoreͰ3Ґ࣌ͷτϥϑΟοΫΛ҆ఆͯ͠ॲཧ • Ansible playbookɺDBͷϚΠάϨʔγϣϯ࡞ۀͷڞ௨Խ
Architecture nginx nginx nginx DNS-RR App App App App App
App MySQL MySQL memcached memcached util util cloud cloud JP nginx nginx nginx App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK γϯϓϧͳ3ߏ ΫϥυͰEC2/GCE (αʔό) Λ த৺ʹߏ ɾ USಠࣗͷαʔϏε খنDBʹ RDSΛ͏͜ͱ UKͰCloud Load BalancerΛར༻
Internal DNS App App App App App App DNS DNS
unbound unbound unbound unbound unbound unbound DNS unbound Consul DNS *.consul *.local • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε্ • resolv.conf ΑΓোʹڧ͍ • αʔϏεͷՄ༻ੑͱॊೈੑΛ֬อ • ΞϓϦέʔγϣϯIPΞυϨεͰͳ͘ϗετ໊Λར༻ • ΞϓϦέʔγϣϯίʔυͷมߋͳ͠ʹߏมߋ͕Մೳ • Internal LBସͱͯ͠consul Λ͔ͭͬͨԽͱෛՙࢄΛଟ༻
Infrastructure History #2 2018 - / Microservices on Multi-Cloud
Microservices • αʔϏεͷ Resilience Λ্ͤ͞Δ • ࡉ͔͍୯ҐͰͷεέʔϦϯάɺোͷ • νʔϜɾ৫ͷ Scalability
ΛߴΊΔ • 1000໊Ҏ্ͷΤϯδχΞ৫Λࢤ • αʔϏε։ൃͷΛ͞Βʹ͍͋͛ͯͨ͘Ί
US Re-Architecture • US marketʹΑΓ࠷దԽ͘͢ Client ΛFull Renewal • MicroservicesͷroutingΛߦ͏API
GatewayΛGolangͰ࣮ • AWS্ͷMonolith APIΛWrap • ؇͔ͳҠߦΛ࣮ݱ API Gateway search personalization offer gRPC JSON over HTTPs Protocol Buffers over HTTPs gRPC gRPC Monolith API
API Fork • 3ͭͷRegionͰڞ༗͍ͯͨ͠Monolith APIͷίʔυΛ US,UK ͱ JP Ͱ •
ࣗregionͷมߋ͕ଞregionʹӨڹ͢Δ͜ͱΛ͑ΔɻௐɾQAίετݮ • ΑΓ֤ࠃͷࣄʹ͋ͬͨ։ൃΛ֤ࠃͰߦ͏ • US,UKͷݱ࠾༻ਐల
API Gateway in JP • Monolith API͔ΒݺΕΔ Microservices ͢Ͱʹӡ༻த •
JPͰMicroservicesΛ͞ΒʹਐΊΔͨΊ API GatewayΛಋೖ • Golang͕ͩɺUSͱҟͳΔ࣮ • Clientͷมߋͳ͘Protocolҡ࣋ • DNS cacheɺRequest bufferingͳͲͷՃ API Gateway JSON over HTTPs JSON over HTTPs ServiceA ServiceC ServiceB
Infrastructure in 2018 JP UK US + + ͦΕͧΕͷRegionʹ͋Θͤͨ Microservices
on Multi-Cloud
Microservices Tech Stack • Container / Docker • Kubernetes •
Spinnaker
Container / Docker • Container • Ϧιʔεͷɾ੍ޚ • VMΑΓܰྔͳOSڥΛ࣮ݱ •
Docker • ϙʔλϏςΟͷ࣮ݱ • DockerfileʹΑΔҰ؏ͨ͠Πϝʔδͷ࡞
Container use case Github PR Daily job BigQuery (app-log) index
Container Registory DEPLOY!! Application͚ͩͰͳ͘ MLRecommendͷσʔλΛؚΉContainerΛ࡞ ෳࡶͳMiddleware҆ఆͯ͠ఏڙ container for keyword suggest service
Kubernetes • Container ͷ Orchestration Platform • ࣗಈScalingɺࣗಈhealing • Container
ӡ༻ίετͷݮ • GKE(Google Kubernetes Engine) Λத৺ʹར༻ • k8s͕MicroservicesͷKey factor • AWS EKS/Fargateͷݕূ • ͘͞ΒͷΫϥυɺk8s on Metalͷݕ౼ɾݕূ
Spinnaker • Continuous Delivery Platform • Developed by Netflix •
googleͳͲͷڠྗɾOSSԽ • Deploy pipelineΛఆٛ͠ɺࣗಈ࣮ߦ͢Δ • Multi-Cloud ରԠ • k8s, ECS, OpenStack... • SpinnakerʹΑΔContinuous Delivery http://tech.mercari.com/entry/2017/08/21/092743
Microservices on Multi-Cloud ͷ՝
Microservices on Multi-Cloud Pros/Cons • Pros: Service ʹద࣮ͨ͠ߦڥͷબ • σʔλϕʔεɾMLܥαʔϏεͳͲ৽͍ٕ͠ज़Λૉૣ͘औΓࠐΉ
• ։ൃऀ͕ٕज़બݖΛͭ͜ͱͰɺΦʔφʔγοϓΛΑΓڧ͘ • Cons: Ϋϥυؒ࿈ܞͷޮੑ • ωοτϫʔΫίετ • Ϋϥυؒͷڑ • Cons: αʔϏεͷՄ༻ੑҡ࣋
Distance between clouds ੴङ DC Cloud Service Mircoservices Infrastructure ઐ༻αʔό
Monolith API Infrastructure 1,000 km
Distance between clouds $ ping -c 3 example.mercari.jp PING example.mercari.jp
(x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=50 time=18.6 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=50 time=18.4 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=50 time=20.6 ms ੴङ(ઐ༻αʔό) ▶︎ ౦ژ(Google Cloud Load Balancer) $ ping -c 3 example.mercari.jp PING example.mercari.jp (x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=56 time=1.09 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=56 time=1.08 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=56 time=1.14 ms ౦ژ(͘͞ΒͷΫϥυ) ▶︎ ౦ژ(Google Cloud Load Balancer) 18-20 ms 1 ms ಉ͡DCͰ͋Ε 0.1 ms
Distance between clouds by HTTPS $ ./httpstat.sh https://example.mercari.jp/hc HTTP/1.1 200
OK Server: nginx/1.13.3 Date: Wed, 11 Oct 2017 01:59:15 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Wed, 11 Oct 2017 02:59:15 GMT Cache-Control: max-age=3600 Cache-Control: public Via: 1.1 google Alt-Svc: clear DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 1ms | 19ms | 165ms | 20ms | 0ms ] | | | | | namelookup:1ms | | | | connect:20ms | | | pretransfer:185ms | | starttransfer:205ms | total:205ms
How to beyond the distance • 3 way handshakeΛආ͚ΔɻTLS ͷ
handshake ආ͚Δ • HTTP/1, HTTP/2 ͷKeepAlive Λ׆༻͢Δ • ChoconͰͷConnection Aggregation
chocon • GoͰ࣮ͨ͠γϯϓϧͳ Proxy Server • OSSͱͯ͠ެ։ • github.com/kazeburo/chocon •
1Ҏ্ͷՔಇ࣮
chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME
chocon.local. ෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
After Chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control:
max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms pingͱಉͷ
Durability, Availability • Multi-CloudͰՄ༻ੑԼ͕Δ • ͲͷΫϥυ͕མͪͯαʔϏεͷܧଓʹӨڹ • Քಇ 99.99% ͱ
99.95% ͷΫϥυΛ͍ͬͯΔ߹ɺՔಇ 99.95%ʹͳΔ • MicroservicesͰಛఆͷαʔϏε͕མͪͯશମʹӨڹ͠ͳ͍ͤ͞ͳ͍ • Өڹ͕͑ΒΕΔMicroservicesಛఆͷCloudͰӡ༻ • ߴ͍Մ༻ੑ͕ඞཁͱ͞ΕΔMicroservicesMulti-CloudͰల։
Massive Computing Resource Service Mesh Service Mesh J Infrastructure in
the near future? Security / DDoS mitigation API Gateway A B C D D E CloudA CloudB F CloudC (Monolith API) H K L M
ॊೈͰ৴པੑͷߴ͍ Infrastructure Λ Microservices ͱ Multi-Cloud Ͱ࣮ݱ
We’re Hiring! careers.mercari.com