Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Microservices on Multi-Cloud
Search
kazeburo
March 23, 2018
Technology
21
5.8k
Microservices on Multi-Cloud
MANABIYA TERATAIL DEVELOPER DAYS 2018-03-23
kazeburo
March 23, 2018
Tweet
Share
More Decks by kazeburo
See All by kazeburo
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
10
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
1.7k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
12k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.3k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
550
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.1k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
13k
sacloudns
kazeburo
2
320
Other Decks in Technology
See All in Technology
Datachain会社紹介資料(2024年11月) / Company Deck
datachain
4
17k
製造現場のデジタル化における課題とPLC Data to Cloudによる新しいアプローチ
hamadakoji
0
210
元旅行会社の情シス部員が教えるおすすめなre:Inventへの行き方 / What is the most efficient way to re:Invent
naospon
2
280
Microsoft Fabric OneLake の実体について
ryomaru0825
0
190
LINEヤフー株式会社における音声言語情報処理AI研究開発@SP/SLP研究会 2024.10.22
lycorptech_jp
PRO
2
280
Terraform未経験の御様に対してどの ように導⼊を進めていったか
tkikuchi
1
250
第23回Ques_タイミーにおけるQAチームの在り方 / QA Team in Timee
takeyaqa
0
190
What to do after `laravel new`
mattstauffer
0
140
Engineering at LY Corporation
lycorp_recruit_jp
0
320
利きプロセススケジューラ
sat
PRO
4
2.6k
Deno+JSRでパッケージを作って公開する
askua
0
120
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
2
130
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
45
6.7k
How To Stay Up To Date on Web Technology
chriscoyier
788
250k
How to Ace a Technical Interview
jacobian
276
23k
Become a Pro
speakerdeck
PRO
25
5k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Being A Developer After 40
akosma
86
590k
The Cult of Friendly URLs
andyhume
78
6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.2k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.8k
Into the Great Unknown - MozCon
thekraken
32
1.5k
BBQ
matthewcrist
85
9.3k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Transcript
Microservices on Multi-Cloud Masahiro @kazeburo Nagano MANABIYA Teratail developer days
2018/03/23
Me • խ • @kazeburo • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability
Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ • झຯDBͷ Restore
Agenda • ϝϧΧϦʹ͍ͭͯ • ϝϧΧϦͷ Infrastructure History #1 - Multi-Cloud
• ϝϧΧϦͷ Infrastructure History #2 - Microservices on Multi-Cloud • Microservices on Multi-Cloud ͷ՝
None
ϝϧΧϦ • ຊ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ
ถࠃ/ӳࠃ ͷల։ JP UK US
KPI μϯϩʔυ GMV(૯औҾֹ) 1ԯDLҎ্(JP+US+UK) ݄ؒ100ԯԁҎ্ ग़ 1100ສҎ্
γεςϜ֓ཁ ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services
LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ը૾ ܾࡁ AI ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏
Infrastructure
Infrastructure in 2017 DNS: Amazon Route53 CDN: Akamai, CloudFront Storage:
Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel JP UK US
Infrastructure in 2018 DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux(JP)
Storage: Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog JP UK US + +
Infrastructure History #1 2013 - 2017 / Multi-Cloud
Infrastructure History (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷʮ͘͞ΒͷVPSʯ1ʹWebDBࡌͤͨߏͰ։࢝ •
Infrastructure ઐऀ͕͍ͳ͍தͰɺ։ൃऀʹۙͳج൫Λબ • ϦϦʔεޙ2ϲ݄Ͱʮ͘͞ΒͷΫϥυʯʮઐ༻αʔόʯҠߦ
ʮ͘͞Βͷઐ༻αʔόʯ • Metal as a Service • ཧαʔόΛΫϥυͷΑ͏ʹѻ͑Δ • ཧαʔόͳΒͰͷύϑΥʔϚϯε
• ωοτϫʔΫͱϋʔυΣΞͷอक ͘͞ΒΠϯλʔωοτ༷͕୲ • ʮ͘͞ΒͷΫϥυʯͱଓ͕Մೳ • ίετύϑΥʔϚϯεʹ༏ΕΔ
Infrastructure History (2) • 2014/09 US ϦϦʔε • AWS (Oregon)
ʹͯαʔϏεߏங • JPϦϦʔε͔Β͠Β͘ܦͪɺ։ൃऀʹAWSܦݧऀ͕૿Ճ • ͦΕͰ Infrastructure ઐऀগͳ͘ɺRDSElastiCacheϚωʔδυαʔϏεΛ ར༻ͯ͠αʔϏεΛߏங • USࠃͷ MaaS Λݕ౼͕ͨ͠ɺUSͰͷαʔϏεͷ༧͕͘͠ɺΫϥυͷॊ ೈ͞Λ JP ΑΓॏཁࢹ
Infrastructure History (3) • 2015/11 SREνʔϜൃ • JP/US ͷΞʔΩςΫνϟΛվળ͠ɺαʔϏεͷ৴པੑͱεέʔϥϏϦςΟͷ ্ʹͱऔΓΉ
• 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़ͱͯ͠ʮGCPʯ্ͰαʔϏεΛߏங
Multi-Cloud in 2017/03 JP UK US ઐ༻αʔό EC2 GCE IaaS
Λத৺ͱͨ͠ Multi-Cloud (Hybrid Cloud) ͨͩ͠ɺͦΕͧΕͷαʔϏεΛΈΔͱ୯ಠͷCloudΛར༻
Multi-Cloud Operations • ՄೳͳݶΓڞ௨ͷΞʔΩςΫνϟΛ࠾༻ • ଞͷΫϥυʹଘࡏ͠ͳ͍ϚωʔδυαʔϏεͷϦϓϨΠε • Consul/Local DNSͷಋೖ •
ΦϖϨʔγϣϯͷڞ௨ԽɾগਓͰͷӡ༻ͷ࣮ݱ • JP ͷنͰ࣮ͷ͋ΔߏɻUS AppStoreͰ3Ґ࣌ͷτϥϑΟοΫΛ҆ఆͯ͠ॲཧ • Ansible playbookɺDBͷϚΠάϨʔγϣϯ࡞ۀͷڞ௨Խ
Architecture nginx nginx nginx DNS-RR App App App App App
App MySQL MySQL memcached memcached util util cloud cloud JP nginx nginx nginx App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK γϯϓϧͳ3ߏ ΫϥυͰEC2/GCE (αʔό) Λ த৺ʹߏ ɾ USಠࣗͷαʔϏε খنDBʹ RDSΛ͏͜ͱ UKͰCloud Load BalancerΛར༻
Internal DNS App App App App App App DNS DNS
unbound unbound unbound unbound unbound unbound DNS unbound Consul DNS *.consul *.local • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε্ • resolv.conf ΑΓোʹڧ͍ • αʔϏεͷՄ༻ੑͱॊೈੑΛ֬อ • ΞϓϦέʔγϣϯIPΞυϨεͰͳ͘ϗετ໊Λར༻ • ΞϓϦέʔγϣϯίʔυͷมߋͳ͠ʹߏมߋ͕Մೳ • Internal LBସͱͯ͠consul Λ͔ͭͬͨԽͱෛՙࢄΛଟ༻
Infrastructure History #2 2018 - / Microservices on Multi-Cloud
Microservices • αʔϏεͷ Resilience Λ্ͤ͞Δ • ࡉ͔͍୯ҐͰͷεέʔϦϯάɺোͷ • νʔϜɾ৫ͷ Scalability
ΛߴΊΔ • 1000໊Ҏ্ͷΤϯδχΞ৫Λࢤ • αʔϏε։ൃͷΛ͞Βʹ͍͋͛ͯͨ͘Ί
US Re-Architecture • US marketʹΑΓ࠷దԽ͘͢ Client ΛFull Renewal • MicroservicesͷroutingΛߦ͏API
GatewayΛGolangͰ࣮ • AWS্ͷMonolith APIΛWrap • ؇͔ͳҠߦΛ࣮ݱ API Gateway search personalization offer gRPC JSON over HTTPs Protocol Buffers over HTTPs gRPC gRPC Monolith API
API Fork • 3ͭͷRegionͰڞ༗͍ͯͨ͠Monolith APIͷίʔυΛ US,UK ͱ JP Ͱ •
ࣗregionͷมߋ͕ଞregionʹӨڹ͢Δ͜ͱΛ͑ΔɻௐɾQAίετݮ • ΑΓ֤ࠃͷࣄʹ͋ͬͨ։ൃΛ֤ࠃͰߦ͏ • US,UKͷݱ࠾༻ਐల
API Gateway in JP • Monolith API͔ΒݺΕΔ Microservices ͢Ͱʹӡ༻த •
JPͰMicroservicesΛ͞ΒʹਐΊΔͨΊ API GatewayΛಋೖ • Golang͕ͩɺUSͱҟͳΔ࣮ • Clientͷมߋͳ͘Protocolҡ࣋ • DNS cacheɺRequest bufferingͳͲͷՃ API Gateway JSON over HTTPs JSON over HTTPs ServiceA ServiceC ServiceB
Infrastructure in 2018 JP UK US + + ͦΕͧΕͷRegionʹ͋Θͤͨ Microservices
on Multi-Cloud
Microservices Tech Stack • Container / Docker • Kubernetes •
Spinnaker
Container / Docker • Container • Ϧιʔεͷɾ੍ޚ • VMΑΓܰྔͳOSڥΛ࣮ݱ •
Docker • ϙʔλϏςΟͷ࣮ݱ • DockerfileʹΑΔҰ؏ͨ͠Πϝʔδͷ࡞
Container use case Github PR Daily job BigQuery (app-log) index
Container Registory DEPLOY!! Application͚ͩͰͳ͘ MLRecommendͷσʔλΛؚΉContainerΛ࡞ ෳࡶͳMiddleware҆ఆͯ͠ఏڙ container for keyword suggest service
Kubernetes • Container ͷ Orchestration Platform • ࣗಈScalingɺࣗಈhealing • Container
ӡ༻ίετͷݮ • GKE(Google Kubernetes Engine) Λத৺ʹར༻ • k8s͕MicroservicesͷKey factor • AWS EKS/Fargateͷݕূ • ͘͞ΒͷΫϥυɺk8s on Metalͷݕ౼ɾݕূ
Spinnaker • Continuous Delivery Platform • Developed by Netflix •
googleͳͲͷڠྗɾOSSԽ • Deploy pipelineΛఆٛ͠ɺࣗಈ࣮ߦ͢Δ • Multi-Cloud ରԠ • k8s, ECS, OpenStack... • SpinnakerʹΑΔContinuous Delivery http://tech.mercari.com/entry/2017/08/21/092743
Microservices on Multi-Cloud ͷ՝
Microservices on Multi-Cloud Pros/Cons • Pros: Service ʹద࣮ͨ͠ߦڥͷબ • σʔλϕʔεɾMLܥαʔϏεͳͲ৽͍ٕ͠ज़Λૉૣ͘औΓࠐΉ
• ։ൃऀ͕ٕज़બݖΛͭ͜ͱͰɺΦʔφʔγοϓΛΑΓڧ͘ • Cons: Ϋϥυؒ࿈ܞͷޮੑ • ωοτϫʔΫίετ • Ϋϥυؒͷڑ • Cons: αʔϏεͷՄ༻ੑҡ࣋
Distance between clouds ੴङ DC Cloud Service Mircoservices Infrastructure ઐ༻αʔό
Monolith API Infrastructure 1,000 km
Distance between clouds $ ping -c 3 example.mercari.jp PING example.mercari.jp
(x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=50 time=18.6 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=50 time=18.4 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=50 time=20.6 ms ੴङ(ઐ༻αʔό) ▶︎ ౦ژ(Google Cloud Load Balancer) $ ping -c 3 example.mercari.jp PING example.mercari.jp (x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=56 time=1.09 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=56 time=1.08 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=56 time=1.14 ms ౦ژ(͘͞ΒͷΫϥυ) ▶︎ ౦ژ(Google Cloud Load Balancer) 18-20 ms 1 ms ಉ͡DCͰ͋Ε 0.1 ms
Distance between clouds by HTTPS $ ./httpstat.sh https://example.mercari.jp/hc HTTP/1.1 200
OK Server: nginx/1.13.3 Date: Wed, 11 Oct 2017 01:59:15 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Wed, 11 Oct 2017 02:59:15 GMT Cache-Control: max-age=3600 Cache-Control: public Via: 1.1 google Alt-Svc: clear DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 1ms | 19ms | 165ms | 20ms | 0ms ] | | | | | namelookup:1ms | | | | connect:20ms | | | pretransfer:185ms | | starttransfer:205ms | total:205ms
How to beyond the distance • 3 way handshakeΛආ͚ΔɻTLS ͷ
handshake ආ͚Δ • HTTP/1, HTTP/2 ͷKeepAlive Λ׆༻͢Δ • ChoconͰͷConnection Aggregation
chocon • GoͰ࣮ͨ͠γϯϓϧͳ Proxy Server • OSSͱͯ͠ެ։ • github.com/kazeburo/chocon •
1Ҏ্ͷՔಇ࣮
chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME
chocon.local. ෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
After Chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control:
max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms pingͱಉͷ
Durability, Availability • Multi-CloudͰՄ༻ੑԼ͕Δ • ͲͷΫϥυ͕མͪͯαʔϏεͷܧଓʹӨڹ • Քಇ 99.99% ͱ
99.95% ͷΫϥυΛ͍ͬͯΔ߹ɺՔಇ 99.95%ʹͳΔ • MicroservicesͰಛఆͷαʔϏε͕མͪͯશମʹӨڹ͠ͳ͍ͤ͞ͳ͍ • Өڹ͕͑ΒΕΔMicroservicesಛఆͷCloudͰӡ༻ • ߴ͍Մ༻ੑ͕ඞཁͱ͞ΕΔMicroservicesMulti-CloudͰల։
Massive Computing Resource Service Mesh Service Mesh J Infrastructure in
the near future? Security / DDoS mitigation API Gateway A B C D D E CloudA CloudB F CloudC (Monolith API) H K L M
ॊೈͰ৴པੑͷߴ͍ Infrastructure Λ Microservices ͱ Multi-Cloud Ͱ࣮ݱ
We’re Hiring! careers.mercari.com