Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud connect the world as a Glue - AWS Dev Day 2017

Cloud connect the world as a Glue - AWS Dev Day 2017

kazeburo

June 02, 2017
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. Cloud connect the world as a Glue AWS Dev Day

    2017 Track 2 Masahiro Nagano @kazeburo
  2. Me • Masahiro Nagano / ௕໺խ޿ • @kazeburo • Mercari,

    Inc
 Principal Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc Technical Advisor
  3. SRE • Site Reliability Engineering ͷུ • Google ͷӡ༻νʔϜΛ཰͍Δ Ben

    Treynor ͕ఏএ • Google ͷ༷ʑͳϓϩμΫτɾαʔϏεΛԣஅͯ͠ɺιϑτ΢ΣΞΤ ϯδχΞϦϯάΑΓαΠτ/αʔϏεͷ৴པੑΛ޲্ͤ͞Δ Software Engineering/Teamͱͦͷ࣮ફ = Google SRE
  4. Mercari SRE ͷۀ຿ൣғ Operations Software Eng. ج൫ߏங OnCall (ো֐ରԠ) Automation

    εέʔϥϏϦςΟɾՄ༻ੑվળ ϛυϧ΢ΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾ෼ੳج൫ͷߏஙɺӡ༻ αʔόϓϩϏδϣχϯάɾσϓϩΠͷ੔උ ηΩϡϦςΟʗෆਖ਼ར༻ݕग़
  5. Global Development Team Tokyo San Francisco London San Francisco/London ʹΦϑΟε

    ݱ஍࠾༻ɺग़޲ऀɺ௕ظग़ு߹Θͤͯ ΤϯδχΞ͕਺໊͔Β਺े໊
  6. Global Development Team • Tokyo • ։ൃͷத৺ɻJPʹՃ͑ͯશͯͷregionͷ։ൃ • San Fransisco

    • αʔϏεͷϩʔΧϥΠζ • ݴޠ͚ͩͰ͸ͳ͘ɺจԽ΍श׳ͷϩʔΧϥΠζ • London • αʔϏε্ཱͪ͛ϑΣʔζ / ݱ஍ͷ๏ྩͳͲʹ߹ΘͤͨϩʔΧϥΠζ
  7. Global Development ͷ೉͠͞ 10:00 1:00 18:00 9࣌ؒ 7࣌ؒ લ೔ Tokyo

    San Francisco London 3ڌ఺ἧͬͯإΛ߹ΘͤΔ͜ͱ͕͔ͳΓ೉͍͠
  8. SREͷέʔε • 6ਓͷ͏ͪɺ1ਓ͕௕ظUSग़ுத • ݱ஍։ൃͷαʔϏεͷΦϖϨʔγϣϯͷ೺Ѳ • ि1ͰUSͱͷSync MTG • ே9࣌(PDT

    17:00) ʹࣗ୐ʹͯ Video Conference • UKͱ͸Ҋ݅ϕʔεͰ༦ํʹMTG • OnCall ౰൪͸ே9͔࣌Βࣗ୐଴ػɻUS͔Βͷ࡞ۀґཔʹ͋ͨΔ
  9. Infrastructure history (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷVPS 1୆ʹWeb΋DB΋͢΂ͯࡌͤͨ

    • ΠϯϑϥετϥΫνϟઐ೚ऀ͍ͳ͍தͰɺ਎ۙͳٕज़Λબ୒ • ϦϦʔεޙ2ϲ݄Ͱ͘͞ΒΫϥ΢υɺઐ༻αʔόʹҠߦ͖ͯͨ͠
  10. Infrastructure history (2) • 2014/09 US ϦϦʔε • AWS (Oregon)

    ʹͯαʔϏεߏங • JPϦϦʔε౰ॳʹൺ΂ͯΤϯδχΞ͕૿͑ɺAWSܦݧऀ΋ଟ͘ͳͬͨ • ͦΕͰ΋·ͩΠϯϑϥετϥΫνϟઐ೚ऀ͸গͳ͘ɺAWSͷϚωʔδυ αʔϏεΛଟ͘ར༻ͯ͠αʔϏεΛߏங • USࠃ಺ͷઐ༻αʔόར༻΋ݕ౼͕ͨ͠ɺUSͷεέʔϧ͸༧૝ͮ͠Β͘ɺ Ϋϥ΢υͷॊೈ͞Λ೔ຊΑΓ΋ॏཁࢹͨ͠
  11. Infrastructure history (3) • (2015/02 kazeburo ೖࣾ) • 2015/11 SREνʔϜൃ଍

    • ͘͞ΒΠϯλʔωοτͱAWSͷϋΠϒϦουͳΠϯϑϥετϥΫνϟ ͷ্ͷΞʔΩςΫνϟΛਐԽͤ͞ɺ৴པੑͱεέʔϥϏϦςΟͷ޲্ • 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़తνϟϨϯδͱͯ͠GCPΛબ୒
  12. Architecture • ࡾ૚+αͳγϯϓϧͳΞʔΩςΫνϟ • Reverse Proxy = nginx
 Application =

    Apache+mod_php
 Database = MySQL
 Cache = memcached
 Search = Solr • ଟ͘Λ෺ཧαʔόʹͯߏ੒ • εέʔϧΞοϓ΋εέʔϧΞ΢τ΋ߦ͏Diagonal Scaleࢦ޲ • Databaseʹ͸ ioMemory ΍ NVMe Λ౥ࡌͨ͠αʔόΛ࠾༻ nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client WS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers fic DNS-RR App App App App App App MySQL MySQL memcached memcached util util cloud cloud JP
  13. Architecture JP ͷΞʔΩςΫνϟΛجຊ౿ऻ EC2/GCE (αʔό) Λத৺ͨ͠ߏ੒ ɾ USಠࣗͷαʔϏε΍
 খن໛ʙதن໛DBʹ͸ RDS

    UKͰ͸Cloud Load BalancerΛར༻ nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client anagement onsole IAM Add-on Example: IAM Add-on man Intelligence Tasks (HIT) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached util util US RDS EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corpora data cen Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK
  14. αʔόத৺ͷ Architecture • ϝϯςϯεϏϦςΟɾεέʔϥϏϦςΟઓུͷڞ௨Խ • গਓ਺Ͱͷӡ༻ • Ansible Playbook ࠶ར༻

    • εέʔϧ͕ઌߦ͍ͯ͠ΔJPͰ࣮੷͋Δߏ੒ • US Ͱͷ App Store ϥϯΩϯά3ҐͷτϥϑΟοΫ΋໰୊ͳ͘ӡ༻ • EC2ͷIaaSͱͯ͠ͷύϑΥʔϚϯεɺ৴པੑ͸͔ͳΓ޲্͍ͯ͠Δ
  15. Mercari Architecture ·ͱΊ • 3ͭͷRegionͰ࠾༻͢ΔΠϯϑϥετϥΫνϟ͕ҟͳΔ • JP/US/UK ͸αʔόΛத৺ͱͨ͠ArchitectureΛ࠾༻ • AWSͰ΋Ϋϥ΢υΒ͍͠ઃܭ͸ͤͣɺن໛Ͱઌߦ͢ΔJPʹ߹Θͤ

    Δ͜ͱͰɺӡ༻ͷڞ௨ԽͱলྗԽ • ϝϧΧϦͰ͸Ϋϥ΢υΛੵۃతʹ࢖͍ͬͯͳ͍ʁ • JP/US/UK ڞ௨ͷΠϯϑϥετϥΫνϟͰར༻͍ͯ͠·͢
  16. Global Infrastructure JP US UK DNS: Amazon Route53 CDN: Akamai,

    CloudFront Storage: Amazon S3 Analysis: Google BigQuery Common Micro Services ܾࡁ/෺ྲྀ/Domestic Service ܾࡁ/෺ྲྀ/Domestic Service ܾࡁ/෺ྲྀ/Domestic Service ڞ௨ΞʔΩςΫνϟ αʔό͕த৺ ֤Region Ϋϥ΢υ͕த৺ ৴པੑͷߴ͍AWSͷ αʔϏε͕ڬΈࠐΉ
  17. Amazon Route53 • ߴ͍Մ༻ੑͱ৴པੑͷDNS • Roadworker Λར༻ • github.com/codenize-tools/roadworker •

    Routefile ΛGithubͰ؅ཧ • Pull RequestͷmergeޙɺCIΛܦ༝ͯ͠ ࣗಈ൓ө hosted_zone "mercari.jp." do rrset "api.mercari.jp.", "CNAME" do ttl 30 resource_records( "endpoint-api.mercari.jp" ) end end #Routefile PR Github Travis-CI Route53
  18. Amazon Route53 + HealthCheck • DNS-RR ӡ༻࣌ͷ໰୊఺ • αʔόো֐࣌ʹDNSͷॻ͖׵͑ʹ͕͔͔࣌ؒΔ •

    ϒϥ΢βͳͲͷҰ෦ΫϥΠΞϯτ͸DNS-RRͷ৔߹ɺҰ෦ͷαʔόʹ઀ଓ͕Ͱ͖ ͳ͍৔߹ɺଞͷαʔό΁઀ଓ͠௚͢ͷͰো֐ʹΑΔӨڹ͸େ͖͘ͳΓʹ͍͘ɻ • ϚΠΫϩαʔϏεԽ͕ਐΉͱ༷ʑͳϒϥ΢βҎ֎ͷΫϥΠΞϯτ͕઀ଓ͢Δɻ ଟ͘͸DNS-RRͷো֐࣌ͷ࠶઀ଓ͸࣮૷͞Εͯͳ͍ • Route53 ͷ Health CheckΛ࢖͍ղܾ(Λݕূத)
  19. Route53 + Health Check with Roadworker [“153.x.y.150”, "153.x.y.151"].each do |ip|

    rrset "endpoint-ha.mercari.jp.", "A" do ttl 30 weight 1 set_identifier “endpoint-ha-“ + ip.gsub(/\./,'-') health_check "http://#{ip}/hc", :request_interval => 30, :failure_threshold => 3 resource_records( "#{ip}" ) end end Health CheckʹΑΓ DNS-RR Ͱ΋Մ༻ੑΛߴΊΒΕΔ #Routefile
  20. (࿩͸ͦΕ·͕͢) ಺෦ DNS • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε޲্ • resolv.confΑΓՄ༻ੑ্͕͕Δ •

    DNSαʔόͷunboundͰϦΫΤετΛৼΓ෼͚ • *.local ͸BIND͕ݖҖαʔό • *.consul ͸consul DNS interface App App App App App App DNS DNS unbound unbound unbound unbound unbound unbound DNS unbound Consul/DNS BIND *.consul *.local
  21. ͋ΒΏΔσʔλͷετϨʔδ: ঎඼ը૾ App App App App App App Client Multimedia

    Corporate data center Traditional server Mobile Client ent IAM Add-on Example: IAM Add-on ligence HIT) Assignment/ Task Requester Workers ঎඼ը૾σʔλ͸ಉظతʹॖখ/Ξοϓϩʔυ AWS SDK for PHPΛར༻ɻෳ਺ͷը૾Λฒߦͯ͠PUTͯ͠଎౓޲্ ग़඼! ਺ඦສຕ/day
  22. Amazon S3 as a Hub MySQL SaaS / ࣾ಺ Microservices

    + IAM ػցֶशʗෆਖ਼ݕ஌ ෺ྲྀɾܾࡁ ৴པੑͷߴ͍S3ΛHubͱͯ͠ɺૄ݁߹Λ࣮ݱ SFTP Partner goofys App App App App App App nginx nginx nginx consulͰઃఆΛ഑෍ σʔλͷimport/export ACL ML API ML API ֶशσʔλͷimport
  23. ࠃ಺ͱࠃ֎ͷϨΠςϯγ ੴ AWS ౦ GCP 18ms 70-100ms 140ms GCP 6ms

    ͍͍ͩͨͷ஋ ଠฏ༸/๺ถେ཮/େ੢༸͸΋ͱΑΓɺੴङ΋ԕ͍
  24. ΫϥΠΞϯτ͔Βͷ઀ଓվળ • CDNΛར༻͢Δ • Cloudfront, Akamai, Fastly • ΫϥΠΞϯτ͸ۙ͘ʹ͋ΔCDNͷΤοδαʔόͱTLS Handshaking

    • CDN ͱ Origin ؒ͸ίωΫγϣϯू໿΍ઐ༻ωοτϫʔΫΛར༻͢ Δ͜ͱͰߴ଎Խ • www.mercari.com ͸CDNΛར༻
  25. CDNͷར༻: mercari Web JP US UK mercari.com/ mercari.com/jp/ mercari.com/uk/ CDN

    azon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client AM Add-on Example: IAM Add-on Assignment/ Task Requester Workers ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client anagement nsole IAM Add-on Example: IAM Add-on man Intelligence Tasks (HIT) Assignment/ Task Requester Workers ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client et AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers on l Turk vice Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Reques Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights re User Users Client Mul Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk Human Intelligence Tasks (HIT) Assignment/ Task Workers Amazon Mechanical Turk Non-Service Specific ੴङDC
  26. chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME

    chocon.local. ಺෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
  27. Before chocon $ ./httpstat.sh /dev/null https://microservice.example.com/hc HTTP/1.1 200 OK Server:

    nginx/1.11.5 Date: Thu, 01 Jun 2017 00:43:49 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Thu, 01 Jun 2017 01:43:49 GMT Cache-Control: max-age=3600,public Body stored in: /tmp/httpstat-body.263264511496278239 DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 2ms | 24ms | 197ms | 25ms | 0ms ] | | | | | namelookup:2ms | | | | connect:26ms | | | pretransfer:223ms | | starttransfer:248ms | total:248ms
  28. After chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control:

    max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms
  29. Why chocon? • ࣅͨmiddleware͸ݟ͔ͭΒͳ͍ • ୯७ͳforward proxyͰ͸HTTPS௨৴ͷू໿͸Ͱ͖ͳ͍ • HTTPS͸end to

    endͰ҉߸ԽɻMITM Proxy͕ඞཁʹͳΔ • Goݴޠඪ४ͷHTTP/2ʹΑΓޮ཰ͷྑ͍ू໿ɺߴ଎ͳΞΫηε͕ظ଴
  30. chocon in JP App App App App App App App

    App chocon DC(Cloud) Cloud(DC) Microservices SaaS Cloud API endpoint 90msec ͕19msec ͱੴङ౦ژؒͷRTT࣮ଌ஋·Ͱվળ AWS SDK΋endpointΛ੾Γସ͑Δ͜ͱͰར༻Մೳ http or https keepAlive
  31. chocon & Pacific Ocean App App App App chocon US

    Cloud HTTPS, HTTP/2 Keepalive 100msecఔ౓·Ͱ஗Ԇ͕཈͑ΒΕɺଞRegionͱͷ࿈ܞͷ࣮ݱɻ USͷઌਐతͳΫϥ΢υαʔϏεʹΞΫηε͠΍͘͢ͳΔ Cloudfront/CDN ւఈέʔϒϧ
  32. ·ͱΊ • ϝϧΧϦ͸ JP/US/UK ͷ3ڌ఺ͰαʔϏεల։ɺ։ൃ΋ߦ͏ • ֤Region͸αʔόΛத৺ͱͨ͠ڞ௨ͨ͠ΞʔΩςΫνϟ • άϩʔόϧͰ͸Amazon Route53,

    Amazon S3ͷߴ͍৴པੑʹࢧ͑ΒΕ͍ͯΔ • ੈքΛ݁ͿͨΊʹΫϥ΢υαʔϏε΍ಠࣗ։ൃͷιϑτ΢ΣΞΛར༻