Cloud connect the world as a Glue - AWS Dev Day 2017

1. Cloud connect the world as a Glue AWS Dev Day 2017 Track 2 Masahiro Nagano @kazeburo
2. Me • Masahiro Nagano / ௕໺խ޿ • @kazeburo • Mercari, Inc
 Principal Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc Technical Advisor
3. SRE Team ͷ঺հ
4. SRE • Site Reliability Engineering ͷུ • Google ͷӡ༻νʔϜΛ཰͍Δ Ben Treynor ͕ఏএ • Google ͷ༷ʑͳϓϩμΫτɾαʔϏεΛԣஅͯ͠ɺιϑτ΢ΣΞΤ ϯδχΞϦϯάΑΓαΠτ/αʔϏεͷ৴པੑΛ޲্ͤ͞Δ Software Engineering/Teamͱͦͷ࣮ફ = Google SRE
5. Google SRE • ιϑτ΢ΣΞΤϯδχΞ(SWE)ͱͯ͠࠾༻ • ӡ༻ͷۀ຿Λ50%ҎԼʹ཈͑Δ • 50%͸ΦϖϨʔγϣϯͷࣗಈԽɺιϑτ΢ΣΞͷ৴པੑ޲্ʹ͋ͯΔ • ΤϥʔόδΣοτͱ͍͏ߟ͑ํ • SREͱSWEͷSLAΛऔΓܾΊɺར֐ΛҰகͤ͞Δ http://landing.google.com/sre/book.html
7. Mercari SRE • ͍ͭͰ΋շద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯά͸શ෦΍Δʯ • 2015/11 ʮΠϯϑϥνʔϜʯ͔ΒSRE΁ • ʮΠϯϑϥʯΑΓ΋αʔϏεࢦ޲ • ݱࡏϝϯόʔ͸ʮ6ਓʯઈࢍืूத
8. Mercari SRE ͷۀ຿ൣғ Operations Software Eng. ج൫ߏங OnCall (ো֐ରԠ) Automation εέʔϥϏϦςΟɾՄ༻ੑվળ ϛυϧ΢ΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾ෼ੳج൫ͷߏஙɺӡ༻
   αʔόϓϩϏδϣχϯάɾσϓϩΠͷ੔උ
   ηΩϡϦςΟʗෆਖ਼ར༻ݕग़
9. Agenda • ϝϧΧϦͱ͸ / ੈք3ڌ఺Ͱͷ։ൃӡ༻ମ੍ʹ͍ͭͯ • ϝϧΧϦͷΞʔΩςΫνϟ / Ϋϥ΢υͷར༻ • ڑ཭Λ௒͑ΔɺੈքΛܨ͙γεςϜ
10. Mercari • ࠃ಺࠷େڃͷϑϦϚΞϓϦ • 3෼Ͱ؆୯ʹग़඼ • ҆৺҆શͳܾࡁ
11. Mercari KPI μ΢ϯϩʔυ਺ GMV(૯औҾֹ) 6500ສDL(JP+US) ݄ؒ100ԯԁҎ্ ग़඼਺ 1೔100ສ඼Ҏ্
12. μ΢ϯϩʔυ਺ਪҠ(JP) JP μ΢ϯϩʔυ਺ 4000ສ (2016/11)
13. ೔ຊ࠷େͷϑϦϚΞϓϦ 1,200 1෼ؒͷग़඼਺(peek࣌ؒଳ) ඼Ҏ্
14. ग़඼͔Β͙͢ʹചΕΔ 24 ࣌ؒҎ಺ ചΕͨ঎඼ͷ໿50%͕ 24࣌ؒҎ಺ʹऔҾ੒ཱ
15. Global Service JP 2016/08 US AppStore 3Ґ US UK 2017/03/15 ϦϦʔε
16. Global Development Team Tokyo San Francisco London San Francisco/London ʹΦϑΟε ݱ஍࠾༻ɺग़޲ऀɺ௕ظग़ு߹Θͤͯ ΤϯδχΞ͕਺໊͔Β਺े໊
17. Global Development Team • Tokyo • ։ൃͷத৺ɻJPʹՃ͑ͯશͯͷregionͷ։ൃ • San Fransisco • αʔϏεͷϩʔΧϥΠζ • ݴޠ͚ͩͰ͸ͳ͘ɺจԽ΍श׳ͷϩʔΧϥΠζ • London • αʔϏε্ཱͪ͛ϑΣʔζ / ݱ஍ͷ๏ྩͳͲʹ߹ΘͤͨϩʔΧϥΠζ
18. Global Development ͷ೉͠͞ 10:00 1:00 18:00 9࣌ؒ 7࣌ؒ લ೔ Tokyo San Francisco London 3ڌ఺ἧͬͯإΛ߹ΘͤΔ͜ͱ͕͔ͳΓ೉͍͠
19. Global Development ͷਐΊํ(1) • Ϋϥ΢υΛ׆༻ͯ͠ίϛϡχέʔγϣϯΛਤΔ • ଠฏ༸ɾେ੢༸Λ·͍ͨͩPull ReqeuestϨϏϡʔ • Slack • Video Conference • ϦϞʔτϖΞϓϩ(εΫϦʔϯڞ༗)
20. Global Development ͷਐΊํ(2) • ཱࣗͨ͠νʔϜͱͯ͠՝୊ղܾ͢Δ • ϓϩδΣΫτϚωʔδϟͱɺΫϥΠΞϯτ͔ΒαʔόαΠυΤϯ δχΞ·ͰϑϧελοΫͷϓϩμΫτνʔϜΛݱ஍Ͱ݁੒ • νʔϜؙ͝ͱग़ு • iOS/Android͸RegionʹΑͬͯfork͋Δ͍͸ɺbranchΛ෼͚ɺޓ͍ ͷӨڹΛݮΒ͢
21. SREͷέʔε • 6ਓͷ͏ͪɺ1ਓ͕௕ظUSग़ுத • ݱ஍։ൃͷαʔϏεͷΦϖϨʔγϣϯͷ೺Ѳ • ि1ͰUSͱͷSync MTG • ே9࣌(PDT 17:00) ʹࣗ୐ʹͯ Video Conference • UKͱ͸Ҋ݅ϕʔεͰ༦ํʹMTG • OnCall ౰൪͸ே9͔࣌Βࣗ୐଴ػɻUS͔Βͷ࡞ۀґཔʹ͋ͨΔ
22. Mercari Architecture
23. Infrastructure ੴङDC ઐ༻αʔό JP Cloud US Cloud UK Hybrid & Multi Cloud
24. Infrastructure history (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷVPS 1୆ʹWeb΋DB΋͢΂ͯࡌͤͨ • ΠϯϑϥετϥΫνϟઐ೚ऀ͍ͳ͍தͰɺ਎ۙͳٕज़Λબ୒ • ϦϦʔεޙ2ϲ݄Ͱ͘͞ΒΫϥ΢υɺઐ༻αʔόʹҠߦ͖ͯͨ͠
25. Infrastructure history (2) • 2014/09 US ϦϦʔε • AWS (Oregon) ʹͯαʔϏεߏங • JPϦϦʔε౰ॳʹൺ΂ͯΤϯδχΞ͕૿͑ɺAWSܦݧऀ΋ଟ͘ͳͬͨ • ͦΕͰ΋·ͩΠϯϑϥετϥΫνϟઐ೚ऀ͸গͳ͘ɺAWSͷϚωʔδυ αʔϏεΛଟ͘ར༻ͯ͠αʔϏεΛߏங • USࠃ಺ͷઐ༻αʔόར༻΋ݕ౼͕ͨ͠ɺUSͷεέʔϧ͸༧૝ͮ͠Β͘ɺ Ϋϥ΢υͷॊೈ͞Λ೔ຊΑΓ΋ॏཁࢹͨ͠
26. Infrastructure history (3) • (2015/02 kazeburo ೖࣾ) • 2015/11 SREνʔϜൃ଍ • ͘͞ΒΠϯλʔωοτͱAWSͷϋΠϒϦουͳΠϯϑϥετϥΫνϟ ͷ্ͷΞʔΩςΫνϟΛਐԽͤ͞ɺ৴པੑͱεέʔϥϏϦςΟͷ޲্ • 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़తνϟϨϯδͱͯ͠GCPΛબ୒
27. Architecture • ࡾ૚+αͳγϯϓϧͳΞʔΩςΫνϟ • Reverse Proxy = nginx
 Application = Apache+mod_php
 Database = MySQL
 Cache = memcached
 Search = Solr • ଟ͘Λ෺ཧαʔόʹͯߏ੒ • εέʔϧΞοϓ΋εέʔϧΞ΢τ΋ߦ͏Diagonal Scaleࢦ޲ • Databaseʹ͸ ioMemory ΍ NVMe Λ౥ࡌͨ͠αʔόΛ࠾༻ nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client WS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers fic DNS-RR App App App App App App MySQL MySQL memcached memcached util util cloud cloud JP
28. Architecture JP ͷΞʔΩςΫνϟΛجຊ౿ऻ EC2/GCE (αʔό) Λத৺ͨ͠ߏ੒ ɾ USಠࣗͷαʔϏε΍
 খن໛ʙதن໛DBʹ͸ RDS UKͰ͸Cloud Load BalancerΛར༻ nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client anagement onsole IAM Add-on Example: IAM Add-on man Intelligence Tasks (HIT) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached util util US RDS EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corpora data cen Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK
29. αʔόத৺ͷ Architecture • ϝϯςϯεϏϦςΟɾεέʔϥϏϦςΟઓུͷڞ௨Խ • গਓ਺Ͱͷӡ༻ • Ansible Playbook ࠶ར༻ • εέʔϧ͕ઌߦ͍ͯ͠ΔJPͰ࣮੷͋Δߏ੒ • US Ͱͷ App Store ϥϯΩϯά3ҐͷτϥϑΟοΫ΋໰୊ͳ͘ӡ༻ • EC2ͷIaaSͱͯ͠ͷύϑΥʔϚϯεɺ৴པੑ͸͔ͳΓ޲্͍ͯ͠Δ
30. Mercari Architecture ·ͱΊ • 3ͭͷRegionͰ࠾༻͢ΔΠϯϑϥετϥΫνϟ͕ҟͳΔ • JP/US/UK ͸αʔόΛத৺ͱͨ͠ArchitectureΛ࠾༻ • AWSͰ΋Ϋϥ΢υΒ͍͠ઃܭ͸ͤͣɺن໛Ͱઌߦ͢ΔJPʹ߹Θͤ Δ͜ͱͰɺӡ༻ͷڞ௨ԽͱলྗԽ • ϝϧΧϦͰ͸Ϋϥ΢υΛੵۃతʹ࢖͍ͬͯͳ͍ʁ • JP/US/UK ڞ௨ͷΠϯϑϥετϥΫνϟͰར༻͍ͯ͠·͢
31. Mercari Global Infrastructure
32. Global Infrastructure • Mercari JP/US/UK ͷΠϯϑϥετϥΫνϟ͸ಠཱ͍ͯ͠Δ • σʔλΛαʔϏεΛߦ͏Ҭ಺ʹཹΊΔඞཁੑ • ͍͔ͭ͘ͷΫϥ΢υαʔϏεΛڞ௨ͯ͠ར༻ • ւ֎ͰͷΞΫηεվળ • Ϋϥ΢υͷߴ͍εέʔϥϏϦςΟɾ৴པੑʹΑΓαʔϏεͷՄ༻ ੑΛอͭ
33. Global Infrastructure JP US UK DNS: Amazon Route53 CDN: Akamai, CloudFront Storage: Amazon S3 Analysis: Google BigQuery Common Micro Services ܾࡁ/෺ྲྀ/Domestic Service ܾࡁ/෺ྲྀ/Domestic Service ܾࡁ/෺ྲྀ/Domestic Service ڞ௨ΞʔΩςΫνϟ αʔό͕த৺ ֤Region Ϋϥ΢υ͕த৺ ৴པੑͷߴ͍AWSͷ αʔϏε͕ڬΈࠐΉ
34. Amazon Route53 • ߴ͍Մ༻ੑͱ৴པੑͷDNS • Roadworker Λར༻ • github.com/codenize-tools/roadworker • Routefile ΛGithubͰ؅ཧ • Pull RequestͷmergeޙɺCIΛܦ༝ͯ͠ ࣗಈ൓ө hosted_zone "mercari.jp." do rrset "api.mercari.jp.", "CNAME" do ttl 30 resource_records( "endpoint-api.mercari.jp" ) end end #Routefile PR Github Travis-CI Route53
35. Amazon Route53 + HealthCheck • DNS-RR ӡ༻࣌ͷ໰୊఺ • αʔόো֐࣌ʹDNSͷॻ͖׵͑ʹ͕͔͔࣌ؒΔ • ϒϥ΢βͳͲͷҰ෦ΫϥΠΞϯτ͸DNS-RRͷ৔߹ɺҰ෦ͷαʔόʹ઀ଓ͕Ͱ͖ ͳ͍৔߹ɺଞͷαʔό΁઀ଓ͠௚͢ͷͰো֐ʹΑΔӨڹ͸େ͖͘ͳΓʹ͍͘ɻ • ϚΠΫϩαʔϏεԽ͕ਐΉͱ༷ʑͳϒϥ΢βҎ֎ͷΫϥΠΞϯτ͕઀ଓ͢Δɻ ଟ͘͸DNS-RRͷো֐࣌ͷ࠶઀ଓ͸࣮૷͞Εͯͳ͍ • Route53 ͷ Health CheckΛ࢖͍ղܾ(Λݕূத)
36. Route53 + Health Check with Roadworker [“153.x.y.150”, "153.x.y.151"].each do |ip| rrset "endpoint-ha.mercari.jp.", "A" do ttl 30 weight 1 set_identifier “endpoint-ha-“ + ip.gsub(/\./,'-') health_check "http://#{ip}/hc", :request_interval => 30, :failure_threshold => 3 resource_records( "#{ip}" ) end end Health CheckʹΑΓ DNS-RR Ͱ΋Մ༻ੑΛߴΊΒΕΔ #Routefile
37. (࿩͸ͦΕ·͕͢) ಺෦ DNS • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε޲্ • resolv.confΑΓՄ༻ੑ্͕͕Δ • DNSαʔόͷunboundͰϦΫΤετΛৼΓ෼͚ • *.local ͸BIND͕ݖҖαʔό • *.consul ͸consul DNS interface App App App App App App DNS DNS unbound unbound unbound unbound unbound unbound DNS unbound Consul/DNS BIND *.consul *.local
38. (࿩͸ͦΕ·͕͢) ಺෦DNSͰCNAME • ಺෦DNSͰϚωʔδυαʔϏεͷΤϯυϙΠϯτͷCNAMEΛઃఆ • ΞϓϦέʔγϣϯ͔Β઀ଓ͸CNAMEܦ༝ • ϚωʔδυαʔϏε͔ΒϚωʔδυαʔϏε΁ͷҠߦɺϚωʔδυ αʔϏε͔ΒEC2΁ͷҠߦɺ·ͨͦͷٯͷҠߦ͕΍Γ΍͍͢ db-cstool-master IN CNAME cstool-db.XXXXX.us-west-2.rds.amazonaws.com.
39. Amazon S3 • ߴ͍Մ༻ੑͱ৴པੑͷετϨʔδ • ঎඼ը૾ɺϩάɺσʔλϕʔεͷόοΫΞοϓͳͲ͋ΒΏΔσʔλ Λ֨ೲ • IAMΛར༻ͨ͠ߴ౓ͳΞΫηε؅ཧͱૄ݁߹ͷ࣮ݱ • αϒγεςϜ͔ΒͷσʔλΠϯϙʔτɾΤΫεϙʔτ • ֎෦αʔϏεɾύʔτφʔͱͷσʔλड͚౉͠खஈ
40. ͋ΒΏΔσʔλͷετϨʔδ: ঎඼ը૾ App App App App App App Client Multimedia Corporate data center Traditional server Mobile Client ent IAM Add-on Example: IAM Add-on ligence HIT) Assignment/ Task Requester Workers ঎඼ը૾σʔλ͸ಉظతʹॖখ/Ξοϓϩʔυ AWS SDK for PHPΛར༻ɻෳ਺ͷը૾Λฒߦͯ͠PUTͯ͠଎౓޲্ ग़඼! ਺ඦສຕ/day
41. ͋ΒΏΔσʔλͷετϨʔδ: ϩά Log Log ΞΫηεϩάʗΤϥʔϩάͳͲ֤छϩά͸fluentܦ༝Ͱू໿ͯ͠S3ʹ֨ೲ aws-cli ·ͨ͸ fluent-plugin-s3 batch + aws-cli > 1TB/day App App App App App App
42. ͋ΒΏΔσʔλͷετϨʔδ: όοΫΞοϓ MySQL Master MySQL BackupSlave MySQL͸ຖ೔xtrabackup(ि1Ͱmysqldump) backup༻slave͔ΒbackupΛऔಘɻaws-cliͰసૹ > 1.2TB(ѹॖࡁ)/day xtrabackup + aws-cli MySQL Master MySQL BackupSlave
43. Amazon S3 as a Hub MySQL SaaS / ࣾ಺ Microservices + IAM ػցֶशʗෆਖ਼ݕ஌ ෺ྲྀɾܾࡁ ৴པੑͷߴ͍S3ΛHubͱͯ͠ɺૄ݁߹Λ࣮ݱ SFTP Partner goofys App App App App App App nginx nginx nginx consulͰઃఆΛ഑෍ σʔλͷimport/export ACL ML API ML API ֶशσʔλͷimport
44. ػցֶश΁ͷऔΓ૊Έ • αʔϏεͰར༻தɾݕূத • ݕࡧ݁ՌͷվળɻߦಈղੳʹΑΓɺ঎඼ͷݕࡧΠϯσοΫεʹΩʔ ϫʔυΛ௥Ճ͠ɺΑΓݟ͚ͭ΍͘͢ • ग़඼࣌ͷՁ֨αδΣετ • ػցֶशΛͩΕͰ΋ࢼ͢͜ͱ͕Ͱ͖Δ؀ڥΛ • Amazon ML΋ݕ౼
45. ڑ཭Λ௒͑ͯੈքΛܨ͙
46. ڑ཭ͱϨΠςϯγ • ޫ͸50msecʹ஍ٿ൒प΋Ͱ͖ͳ͍ɻԕڑ཭ͱͷ௨৴͸ίετ͕ߴ͍ • σʔληϯλʔؒɺΫϥ΢υؒͷڑ཭͕͋Δ৔߹ʹ͸ɺͦΕΛࠀ෰ ͠ɺޮ཰ͷྑ͍௨৴Λߦ͏ඞཁ • (ੴङԕ͍໰୊)
47. ࠃ಺ͱࠃ֎ͷϨΠςϯγ ੴ AWS ౦ GCP 18ms 70-100ms 140ms GCP 6ms ͍͍ͩͨͷ஋ ଠฏ༸/๺ถେ཮/େ੢༸͸΋ͱΑΓɺੴङ΋ԕ͍
48. ߴϨΠςϯγ؀ڥͰͷHTTPS௨৴ • ௨ৗͷTCP HandshakingʹՃ͑਺ճͷ΍ΓͱΓ͕ඞཁ • RTT 26msecͰHTTPSͷ௨৴Λߦͳͬͨ৔߹ɺ200msecҎ্͔͔Δ • RTT 100msec௒͑Δͱɺ600msecҎ্ • ࢀߟ) mercari APIͷϨεϙϯελΠϜ(90percentile)͸ 100msec
49. ԕڑ཭઀ଓ͢ΔϢʔεέʔε • ΫϥΠΞϯτ͕ԕ͍ͱ͜Ζ͔ΒαʔϏεʹΞΫηε͢Δ • ւ֎, US౦ւ؛/੢ւ؛ • ΞϓϦέʔγϣϯͷίʔυ͔ΒଞͷΫϥ΢υ(σʔληϯλʔ)ʹ
 ΞΫηε͢Δ • SaaSɺϚΠΫϩαʔϏε
50. ΫϥΠΞϯτ͔Βͷ઀ଓվળ • CDNΛར༻͢Δ • Cloudfront, Akamai, Fastly • ΫϥΠΞϯτ͸ۙ͘ʹ͋ΔCDNͷΤοδαʔόͱTLS Handshaking • CDN ͱ Origin ؒ͸ίωΫγϣϯू໿΍ઐ༻ωοτϫʔΫΛར༻͢ Δ͜ͱͰߴ଎Խ • www.mercari.com ͸CDNΛར༻
51. CDNͷར༻: mercari Web JP US UK mercari.com/ mercari.com/jp/ mercari.com/uk/ CDN azon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client AM Add-on Example: IAM Add-on Assignment/ Task Requester Workers ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client anagement nsole IAM Add-on Example: IAM Add-on man Intelligence Tasks (HIT) Assignment/ Task Requester Workers ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client et AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers on l Turk vice Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Reques Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights re User Users Client Mul Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk Human Intelligence Tasks (HIT) Assignment/ Task Workers Amazon Mechanical Turk Non-Service Specific ੴङDC
52. ΞϓϦέʔγϣϯ͔ΒΫϥ΢υ΁ΞΫηε • ΞϓϦέʔγϣϯͰHTTPS௨৴ͷKeepAliveΛߦ͏ • PHP ApplicationͰͷKeepAlive͸೉͍͠ • ϦΫΤετॲཧޙʹϝϞϦ͕ΫϦΞ͞Εɺ TCP઀ଓ΋੾ΕΔ • ϚϧνϓϩηεͰ͋ΓɺKeepAliveͯ͠΋ޮ཰͕ѱ͍ • => ͦ͜Ͱ Connection PoolingΛ໨తͱͨ͠ Proxy ServerΛ։ൃ
53. chocon • GoͰ࣮૷ͨ͠γϯϓϧͳ
 Proxy Server • OSSͱͯ͠ެ։ • github.com/kazeburo/chocon • ൒೥Ҏ্ͷՔಇ࣮੷
54. chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME chocon.local. ಺෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
55. Before chocon $ ./httpstat.sh /dev/null https://microservice.example.com/hc HTTP/1.1 200 OK Server: nginx/1.11.5 Date: Thu, 01 Jun 2017 00:43:49 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Thu, 01 Jun 2017 01:43:49 GMT Cache-Control: max-age=3600,public Body stored in: /tmp/httpstat-body.263264511496278239 DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 2ms | 24ms | 197ms | 25ms | 0ms ] | | | | | namelookup:2ms | | | | connect:26ms | | | pretransfer:223ms | | starttransfer:248ms | total:248ms
56. After chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control: max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms
57. Why chocon? • ࣅͨmiddleware͸ݟ͔ͭΒͳ͍ • ୯७ͳforward proxyͰ͸HTTPS௨৴ͷू໿͸Ͱ͖ͳ͍ • HTTPS͸end to endͰ҉߸ԽɻMITM Proxy͕ඞཁʹͳΔ • Goݴޠඪ४ͷHTTP/2ʹΑΓޮ཰ͷྑ͍ू໿ɺߴ଎ͳΞΫηε͕ظ଴
58. chocon in JP App App App App App App App App chocon DC(Cloud) Cloud(DC) Microservices SaaS Cloud API endpoint 90msec ͕19msec ͱੴङ౦ژؒͷRTT࣮ଌ஋·Ͱվળ AWS SDK΋endpointΛ੾Γସ͑Δ͜ͱͰར༻Մೳ http or https keepAlive
59. chocon & Pacific Ocean App App App App chocon US Cloud HTTPS, HTTP/2 Keepalive 100msecఔ౓·Ͱ஗Ԇ͕཈͑ΒΕɺଞRegionͱͷ࿈ܞͷ࣮ݱɻ USͷઌਐతͳΫϥ΢υαʔϏεʹΞΫηε͠΍͘͢ͳΔ Cloudfront/CDN ւఈέʔϒϧ
60. ·ͱΊ
61. ·ͱΊ • ϝϧΧϦ͸ JP/US/UK ͷ3ڌ఺ͰαʔϏεల։ɺ։ൃ΋ߦ͏ • ֤Region͸αʔόΛத৺ͱͨ͠ڞ௨ͨ͠ΞʔΩςΫνϟ • άϩʔόϧͰ͸Amazon Route53, Amazon S3ͷߴ͍৴པੑʹࢧ͑ΒΕ͍ͯΔ • ੈքΛ݁ͿͨΊʹΫϥ΢υαʔϏε΍ಠࣗ։ൃͷιϑτ΢ΣΞΛར༻
62. We’re Hiring! ੈքʹ௅ΉɺϝϧΧϦ ݴ͍༁φγͷύϑΥʔϚϯεͱ৴པੑͰࢧ͑ΔSRE www.mercari.com/jp/jobs/