Hybrid SERVER Architecture in Mercari

Hybrid SERVER Architecture in Mercari / SRE Tech Talks #1 #sretalks

Transcript

1. Hybrid SERVER Architecture in Mercari Masahiro Nagano @kazéburo SRE Tech Talks #1
2. SRE εςοΧʔ࡞Γ·ͨ͠!!
3. Me • Masahiro Nagano • @kazeburo • Mercari, Inc.
 Principal Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
5. SRE@Mercari • Operations • γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺαʔϏεͷ৴པੑΛ ޲্ͤ͞Δ • Software Engineering • αʔϏεΛεέʔϧͤ͞Δϛυϧ΢ΣΞͷ։ൃɾӡ༻ • ։ൃ؀ڥͷվળɾσϓϩΠπʔϧͷ੔උ
6. SRE@Mercari ࣄྫ • Gaurun - push notification server • ngx_luaʹΑΔBalancer֦ு
 - http://tech.mercari.com/entry/2015/11/25/170049 • ChatOps - http://tech.mercari.com/entry/2015/10/15/183000 • 20+ Mackerel Plugins
 - https://github.com/kazeburo/custom-mackerel-plugins
7. Hybrid SERVER Architecture
 in Mercari
8. Mercari Your Friendly Mobile MarketPlace JP US
9. Mercari KPI DOWNLOAD GMV LISTED ITEMS 4000ສDL(JP+US) ݄ؒ100ԯԁҎ্ 1೔50ສ඼Ҏ্
10. Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US
11. ͘͞Βͷઐ༻αʔό
12. ͘͞Βͷઐ༻αʔό • ίϩέʔγϣϯ/ϨϯλϧαʔόͰ͸ͳ͘ɺMaaS • ioMemoryɺNVMe SSD ౥ࡌՄೳ • ίϯτϩʔϧύωϧ͔Β࠶ىಈɺOSͷ࠶Πϯετʔϧɺ
 ίϯιʔϧ΁ͷΞΫηε • ϓϥΠϕʔτωοτϫʔΫͷߏங
 ͘͞ΒͷΫϥ΢υɺίϩέʔγϣϯαʔϏεͱͷ૬ޓ઀ଓ
13. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP util util cloud cloud
14. Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP ಉ͡ߏ੒ util util cloud cloud util util EC2 EC2
15. HyBrid Architecture • ΞϓϦέʔγϣϯίʔυͷෳࡶੑͷআڈ • `if $region === “US”` `if $region === “JP”` • playbookΛڞ༗ɺಉ͡ϝϯςφϏϦςΟ • ಉ͡εέʔϧઓུ
16. Hyper Hybrid Architecture UK ४උத JP US ͘͞Β AWS ??? Region/λΠϛϯάͰ࠷దͳΠϯϑϥετϥΫνϟͷબ୒ ٕज़ͷ෯͕޿͕Δ
17. US Architecture history • ELB • Internal ELB • ElastiCache • RDS HTTP/2(SPDY), ngx_dynamic_upstreamʹΑΔແఀࢭdeploy ಥൃతͳΞΫηε଱ੑ, ઃఆ൓ө଎౓ͷվળ OS/Kernel tuning, Scalability ޲্ όοΫΞοϓӡ༻खॱͷڞ௨Խ, Rolling Schema Upgrade DNS-RR, nginx Internal DNS, Consul memcached on EC2 MySQL on EC2 ϚωʔδυαʔϏε͔Βαʔόத৺ͷΞʔΩςΫνϟ΁
18. Why SERVER Architecture?
19. Why SERVER Architecture • ৽͍ٕ͠ज़΁ͷରԠ • εέʔϥϏϦςΟͷ੍ޚ • OS/Kernel, ϛυϧ΢ΣΞͷνϡʔχϯά
 ϞχλϦϯά • ن໛ͷ֦େ͕ઌߦ͍ͯ͠ΔJPͷӡ༻ͱڞ௨Խ
20. Why Not 
 SERVERLESS Architecture • ʮϚωʔδυαʔϏεΛ࢖͏͜ͱͰΞϓϦέʔ γϣϯ։ൃʹઐ೦ʯͱ͍͏ϋφγ • ΑΓߴ͍Ϩϕϧͷ৴པੑΛSRE͕ఏڙ͢Δ • ௿͍ϨΠϠʔ͔Βͷ࠷దԽͷੵΈॏͶ • SREʹΑΔ৴པੑͷఏڙ͕೉͍͠΋ͷ͸ੵۃతʹ
 ϚωʔδυαʔϏεʗΫϥ΢υαʔϏεΛ׆༻
21. SERVER Architecture
 Λࢧ͑Δٕज़
22. JP US Akamai, BigQuery, mackerel SendGrid, Route53, S3 Ansible, OS/Kernel/Middleware Tuning ಺෦DNS, Consul ֤Region/Serverʹಋೖ CloudαʔϏε
23. Consul
24. Usage of Consul in Mercari • Service Discovery • SMTP Server • Endpoint of Internal API • Configuration Deployment • TLS Session Ticket • Auto update IP black list • Distributed Lock
25. SMTP Server Discovery { "service": { "name": “mail-delivery“, "port": 25, "checks": [ { "script": "sh /etc/consul.d/check_mail.sh","interval": "10s" } ] } } #!/bin/bash set -e echo "test" | mail -S "smtp=smtp://localhost:25" -r no-reply@shiroyagi no- reply@kuroyagi find /var/log/maillog -mmin -3 | xargs grep kuroyagi|grep "250 Ok" check_mail.sh mail-delivery.json ϝʔϧૹ৴ʹҟৗ͕ ͋ΔͱαʔϏε͔Β ੾Γ཭͞ΕΔ
26. Consul maint is Useful $ consul maint -enable $ consul maint -disable shell shell ଈ࣌(+DNS cache)ͰαʔϏε͔Β֎ΕΔ ϝϯςφϯεʗ࠶ىಈ ଈ࣌(+DNS cache)ͰαʔϏε෮چ ໭͠๨Εʹ ஫ҙ!!!
27. http://connpass.com/event/35836/ @cubicdaiya ͕ MercariͰͷConsulͷࣄྫΛ࿩͠·͢
28. mackerel
29. Migration from Zabbix ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US Z ZABBIX Z ZABBIX MackerelͰͷJP/USҰׅ؂ࢹ RoleΛ࢖͏͜ͱͰ؂ࢹϧʔϧͷJP/USڞ௨Խ
30. Prevent unmonitored Host Ҡߦʗ࡞ۀ࣌ʹ؂ࢹ࿙ΕΛ๷͙ͨΊʹ ϨϙʔτΛ1೔ʹ2ճ࡞੒ͯ͠Slackʹ౤ߘ
31. Inside mackerel report JP US fping -a -g 10.0.x.0/24 Λ࢖͍ IPΞυϨεҰཡΛ࡞੒ aws ίϚϯυΛ࢖͍
 IPΞυϨεҰཡΛऔಘ mkr ίϚϯυͰऔಘͨ͠؂ࢹର৅αʔόҰཡͰϑΟϧλ Slack௨஌
32. ·ͱΊ
33. Conclusion • ෺ཧαʔόͱԾ૝αʔόͷ Hybrid Architecture • ServerΛத৺ͱͨ͠ߏ੒Ͱɺߴ͍৴པੑΛ
 SRE͕ఏڙ͢Δ • mercariͷ Hybrid SERVER Architecture Λࢧ͑Δ
 ٕज़ͱͯ͠Consul΍mackerelΛࣄྫΛ঺հ
34. We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
35. Ҏ্