Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Microservices on Multi-Cloud
kazeburo
March 23, 2018
Technology
21
5.4k
Microservices on Multi-Cloud
MANABIYA TERATAIL DEVELOPER DAYS 2018-03-23
kazeburo
March 23, 2018
Tweet
Share
More Decks by kazeburo
See All by kazeburo
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
5
1.1k
sacloudns
kazeburo
2
220
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
1k
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
3
3.8k
Mercari Item Search: Behind The Scenes (20min)
kazeburo
3
2.6k
成長し続けるインフラストラクチャとメルカリの挑戦/mercari infrastructure and software
kazeburo
23
4.6k
logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death
kazeburo
9
11k
Perl in Mercari YAPC::Okinawa 2018 ONNASON
kazeburo
8
5k
インフラチームからSREへ / SRE in Mercari Developers Summit 2018
kazeburo
26
9k
Other Decks in Technology
See All in Technology
S3とCloudWatch Logsの見直しから始めるコスト削減 / Cost saving S3 and CloudWatch Logs
shonansurvivors
0
110
CUEとKubernetesカスタムオペレータを用いた新しいネットワークコントローラをつくってみた
hrk091
0
230
マネーフォワードクラウドを支える事業者基盤
machisuke
0
240
PCI DSS に準拠したシステム開発
yutadayo
0
270
API連携に伴う規制と対応 / Regulations and responses to API linkage
moneyforward
0
110
OVN-Kubernetes-Introduction-ja-2023-01-27.pdf
orimanabu
1
220
ERC3525 Semi-Fungible token
sbtechnight
0
330
Hasuraの本番運用に向けて
nori3tsu
0
280
Multi-Cloud Gatewayでデータを統治せよ!/ Data Federation with MCG
tutsunom
1
140
立ち止まっても、寄り道しても / even if I stop, even if I take a detour
katoaz
0
150
ROS_Japan_UG_#49_LT
maeharakeisuke
0
200
plotlyで動くグラフを作る
kosshi
0
730
Featured
See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
657
120k
Designing the Hi-DPI Web
ddemaree
273
32k
Building an army of robots
kneath
301
40k
The Invisible Customer
myddelton
113
12k
Faster Mobile Websites
deanohume
295
29k
Code Reviewing Like a Champion
maltzj
508
38k
The Power of CSS Pseudo Elements
geoffreycrofte
52
4.3k
How STYLIGHT went responsive
nonsquared
89
4.2k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
10
1.3k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
29
7.8k
jQuery: Nuts, Bolts and Bling
dougneiner
57
6.6k
Design by the Numbers
sachag
271
18k
Transcript
Microservices on Multi-Cloud Masahiro @kazeburo Nagano MANABIYA Teratail developer days
2018/03/23
Me • խ • @kazeburo • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability
Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ • झຯDBͷ Restore
Agenda • ϝϧΧϦʹ͍ͭͯ • ϝϧΧϦͷ Infrastructure History #1 - Multi-Cloud
• ϝϧΧϦͷ Infrastructure History #2 - Microservices on Multi-Cloud • Microservices on Multi-Cloud ͷ՝
None
ϝϧΧϦ • ຊ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ
ถࠃ/ӳࠃ ͷల։ JP UK US
KPI μϯϩʔυ GMV(૯औҾֹ) 1ԯDLҎ্(JP+US+UK) ݄ؒ100ԯԁҎ্ ग़ 1100ສҎ্
γεςϜ֓ཁ ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services
LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ը૾ ܾࡁ AI ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏
Infrastructure
Infrastructure in 2017 DNS: Amazon Route53 CDN: Akamai, CloudFront Storage:
Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel JP UK US
Infrastructure in 2018 DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux(JP)
Storage: Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog JP UK US + +
Infrastructure History #1 2013 - 2017 / Multi-Cloud
Infrastructure History (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷʮ͘͞ΒͷVPSʯ1ʹWebDBࡌͤͨߏͰ։࢝ •
Infrastructure ઐऀ͕͍ͳ͍தͰɺ։ൃऀʹۙͳج൫Λબ • ϦϦʔεޙ2ϲ݄Ͱʮ͘͞ΒͷΫϥυʯʮઐ༻αʔόʯҠߦ
ʮ͘͞Βͷઐ༻αʔόʯ • Metal as a Service • ཧαʔόΛΫϥυͷΑ͏ʹѻ͑Δ • ཧαʔόͳΒͰͷύϑΥʔϚϯε
• ωοτϫʔΫͱϋʔυΣΞͷอक ͘͞ΒΠϯλʔωοτ༷͕୲ • ʮ͘͞ΒͷΫϥυʯͱଓ͕Մೳ • ίετύϑΥʔϚϯεʹ༏ΕΔ
Infrastructure History (2) • 2014/09 US ϦϦʔε • AWS (Oregon)
ʹͯαʔϏεߏங • JPϦϦʔε͔Β͠Β͘ܦͪɺ։ൃऀʹAWSܦݧऀ͕૿Ճ • ͦΕͰ Infrastructure ઐऀগͳ͘ɺRDSElastiCacheϚωʔδυαʔϏεΛ ར༻ͯ͠αʔϏεΛߏங • USࠃͷ MaaS Λݕ౼͕ͨ͠ɺUSͰͷαʔϏεͷ༧͕͘͠ɺΫϥυͷॊ ೈ͞Λ JP ΑΓॏཁࢹ
Infrastructure History (3) • 2015/11 SREνʔϜൃ • JP/US ͷΞʔΩςΫνϟΛվળ͠ɺαʔϏεͷ৴པੑͱεέʔϥϏϦςΟͷ ্ʹͱऔΓΉ
• 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़ͱͯ͠ʮGCPʯ্ͰαʔϏεΛߏங
Multi-Cloud in 2017/03 JP UK US ઐ༻αʔό EC2 GCE IaaS
Λத৺ͱͨ͠ Multi-Cloud (Hybrid Cloud) ͨͩ͠ɺͦΕͧΕͷαʔϏεΛΈΔͱ୯ಠͷCloudΛར༻
Multi-Cloud Operations • ՄೳͳݶΓڞ௨ͷΞʔΩςΫνϟΛ࠾༻ • ଞͷΫϥυʹଘࡏ͠ͳ͍ϚωʔδυαʔϏεͷϦϓϨΠε • Consul/Local DNSͷಋೖ •
ΦϖϨʔγϣϯͷڞ௨ԽɾগਓͰͷӡ༻ͷ࣮ݱ • JP ͷنͰ࣮ͷ͋ΔߏɻUS AppStoreͰ3Ґ࣌ͷτϥϑΟοΫΛ҆ఆͯ͠ॲཧ • Ansible playbookɺDBͷϚΠάϨʔγϣϯ࡞ۀͷڞ௨Խ
Architecture nginx nginx nginx DNS-RR App App App App App
App MySQL MySQL memcached memcached util util cloud cloud JP nginx nginx nginx App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK γϯϓϧͳ3ߏ ΫϥυͰEC2/GCE (αʔό) Λ த৺ʹߏ ɾ USಠࣗͷαʔϏε খنDBʹ RDSΛ͏͜ͱ UKͰCloud Load BalancerΛར༻
Internal DNS App App App App App App DNS DNS
unbound unbound unbound unbound unbound unbound DNS unbound Consul DNS *.consul *.local • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε্ • resolv.conf ΑΓোʹڧ͍ • αʔϏεͷՄ༻ੑͱॊೈੑΛ֬อ • ΞϓϦέʔγϣϯIPΞυϨεͰͳ͘ϗετ໊Λར༻ • ΞϓϦέʔγϣϯίʔυͷมߋͳ͠ʹߏมߋ͕Մೳ • Internal LBସͱͯ͠consul Λ͔ͭͬͨԽͱෛՙࢄΛଟ༻
Infrastructure History #2 2018 - / Microservices on Multi-Cloud
Microservices • αʔϏεͷ Resilience Λ্ͤ͞Δ • ࡉ͔͍୯ҐͰͷεέʔϦϯάɺোͷ • νʔϜɾ৫ͷ Scalability
ΛߴΊΔ • 1000໊Ҏ্ͷΤϯδχΞ৫Λࢤ • αʔϏε։ൃͷΛ͞Βʹ͍͋͛ͯͨ͘Ί
US Re-Architecture • US marketʹΑΓ࠷దԽ͘͢ Client ΛFull Renewal • MicroservicesͷroutingΛߦ͏API
GatewayΛGolangͰ࣮ • AWS্ͷMonolith APIΛWrap • ؇͔ͳҠߦΛ࣮ݱ API Gateway search personalization offer gRPC JSON over HTTPs Protocol Buffers over HTTPs gRPC gRPC Monolith API
API Fork • 3ͭͷRegionͰڞ༗͍ͯͨ͠Monolith APIͷίʔυΛ US,UK ͱ JP Ͱ •
ࣗregionͷมߋ͕ଞregionʹӨڹ͢Δ͜ͱΛ͑ΔɻௐɾQAίετݮ • ΑΓ֤ࠃͷࣄʹ͋ͬͨ։ൃΛ֤ࠃͰߦ͏ • US,UKͷݱ࠾༻ਐల
API Gateway in JP • Monolith API͔ΒݺΕΔ Microservices ͢Ͱʹӡ༻த •
JPͰMicroservicesΛ͞ΒʹਐΊΔͨΊ API GatewayΛಋೖ • Golang͕ͩɺUSͱҟͳΔ࣮ • Clientͷมߋͳ͘Protocolҡ࣋ • DNS cacheɺRequest bufferingͳͲͷՃ API Gateway JSON over HTTPs JSON over HTTPs ServiceA ServiceC ServiceB
Infrastructure in 2018 JP UK US + + ͦΕͧΕͷRegionʹ͋Θͤͨ Microservices
on Multi-Cloud
Microservices Tech Stack • Container / Docker • Kubernetes •
Spinnaker
Container / Docker • Container • Ϧιʔεͷɾ੍ޚ • VMΑΓܰྔͳOSڥΛ࣮ݱ •
Docker • ϙʔλϏςΟͷ࣮ݱ • DockerfileʹΑΔҰ؏ͨ͠Πϝʔδͷ࡞
Container use case Github PR Daily job BigQuery (app-log) index
Container Registory DEPLOY!! Application͚ͩͰͳ͘ MLRecommendͷσʔλΛؚΉContainerΛ࡞ ෳࡶͳMiddleware҆ఆͯ͠ఏڙ container for keyword suggest service
Kubernetes • Container ͷ Orchestration Platform • ࣗಈScalingɺࣗಈhealing • Container
ӡ༻ίετͷݮ • GKE(Google Kubernetes Engine) Λத৺ʹར༻ • k8s͕MicroservicesͷKey factor • AWS EKS/Fargateͷݕূ • ͘͞ΒͷΫϥυɺk8s on Metalͷݕ౼ɾݕূ
Spinnaker • Continuous Delivery Platform • Developed by Netflix •
googleͳͲͷڠྗɾOSSԽ • Deploy pipelineΛఆٛ͠ɺࣗಈ࣮ߦ͢Δ • Multi-Cloud ରԠ • k8s, ECS, OpenStack... • SpinnakerʹΑΔContinuous Delivery http://tech.mercari.com/entry/2017/08/21/092743
Microservices on Multi-Cloud ͷ՝
Microservices on Multi-Cloud Pros/Cons • Pros: Service ʹద࣮ͨ͠ߦڥͷબ • σʔλϕʔεɾMLܥαʔϏεͳͲ৽͍ٕ͠ज़Λૉૣ͘औΓࠐΉ
• ։ൃऀ͕ٕज़બݖΛͭ͜ͱͰɺΦʔφʔγοϓΛΑΓڧ͘ • Cons: Ϋϥυؒ࿈ܞͷޮੑ • ωοτϫʔΫίετ • Ϋϥυؒͷڑ • Cons: αʔϏεͷՄ༻ੑҡ࣋
Distance between clouds ੴङ DC Cloud Service Mircoservices Infrastructure ઐ༻αʔό
Monolith API Infrastructure 1,000 km
Distance between clouds $ ping -c 3 example.mercari.jp PING example.mercari.jp
(x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=50 time=18.6 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=50 time=18.4 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=50 time=20.6 ms ੴङ(ઐ༻αʔό) ▶︎ ౦ژ(Google Cloud Load Balancer) $ ping -c 3 example.mercari.jp PING example.mercari.jp (x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=56 time=1.09 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=56 time=1.08 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=56 time=1.14 ms ౦ژ(͘͞ΒͷΫϥυ) ▶︎ ౦ژ(Google Cloud Load Balancer) 18-20 ms 1 ms ಉ͡DCͰ͋Ε 0.1 ms
Distance between clouds by HTTPS $ ./httpstat.sh https://example.mercari.jp/hc HTTP/1.1 200
OK Server: nginx/1.13.3 Date: Wed, 11 Oct 2017 01:59:15 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Wed, 11 Oct 2017 02:59:15 GMT Cache-Control: max-age=3600 Cache-Control: public Via: 1.1 google Alt-Svc: clear DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 1ms | 19ms | 165ms | 20ms | 0ms ] | | | | | namelookup:1ms | | | | connect:20ms | | | pretransfer:185ms | | starttransfer:205ms | total:205ms
How to beyond the distance • 3 way handshakeΛආ͚ΔɻTLS ͷ
handshake ආ͚Δ • HTTP/1, HTTP/2 ͷKeepAlive Λ׆༻͢Δ • ChoconͰͷConnection Aggregation
chocon • GoͰ࣮ͨ͠γϯϓϧͳ Proxy Server • OSSͱͯ͠ެ։ • github.com/kazeburo/chocon •
1Ҏ্ͷՔಇ࣮
chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME
chocon.local. ෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
After Chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control:
max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms pingͱಉͷ
Durability, Availability • Multi-CloudͰՄ༻ੑԼ͕Δ • ͲͷΫϥυ͕མͪͯαʔϏεͷܧଓʹӨڹ • Քಇ 99.99% ͱ
99.95% ͷΫϥυΛ͍ͬͯΔ߹ɺՔಇ 99.95%ʹͳΔ • MicroservicesͰಛఆͷαʔϏε͕མͪͯશମʹӨڹ͠ͳ͍ͤ͞ͳ͍ • Өڹ͕͑ΒΕΔMicroservicesಛఆͷCloudͰӡ༻ • ߴ͍Մ༻ੑ͕ඞཁͱ͞ΕΔMicroservicesMulti-CloudͰల։
Massive Computing Resource Service Mesh Service Mesh J Infrastructure in
the near future? Security / DDoS mitigation API Gateway A B C D D E CloudA CloudB F CloudC (Monolith API) H K L M
ॊೈͰ৴པੑͷߴ͍ Infrastructure Λ Microservices ͱ Multi-Cloud Ͱ࣮ݱ
We’re Hiring! careers.mercari.com