Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hybrid SERVER Architecture in Mercari

Hybrid SERVER Architecture in Mercari

Hybrid SERVER Architecture in Mercari / SRE Tech Talks #1 #sretalks

kazeburo

July 25, 2016
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. Me • Masahiro Nagano • @kazeburo • Mercari, Inc.
 Principal

    Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
  2. SRE@Mercari • Operations • γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺαʔϏεͷ৴པੑΛ ޲্ͤ͞Δ • Software Engineering •

    αʔϏεΛεέʔϧͤ͞Δϛυϧ΢ΣΞͷ։ൃɾӡ༻ • ։ൃ؀ڥͷվળɾσϓϩΠπʔϧͷ੔උ
  3. SRE@Mercari ࣄྫ • Gaurun - push notification server • ngx_luaʹΑΔBalancer֦ு


    - http://tech.mercari.com/entry/2015/11/25/170049 • ChatOps - http://tech.mercari.com/entry/2015/10/15/183000 • 20+ Mackerel Plugins
 - https://github.com/kazeburo/custom-mackerel-plugins
  4. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or

    its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP util util cloud cloud
  5. Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC

    or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP ಉ͡ߏ੒ util util cloud cloud util util EC2 EC2
  6. HyBrid Architecture • ΞϓϦέʔγϣϯίʔυͷෳࡶੑͷআڈ • `if $region === “US”` `if

    $region === “JP”` • playbookΛڞ༗ɺಉ͡ϝϯςφϏϦςΟ • ಉ͡εέʔϧઓུ
  7. Hyper Hybrid Architecture UK ४උத JP US ͘͞Β AWS ???

    Region/λΠϛϯάͰ࠷దͳΠϯϑϥετϥΫνϟͷબ୒ ٕज़ͷ෯͕޿͕Δ
  8. US Architecture history • ELB • Internal ELB • ElastiCache

    • RDS HTTP/2(SPDY), ngx_dynamic_upstreamʹΑΔແఀࢭdeploy ಥൃతͳΞΫηε଱ੑ, ઃఆ൓ө଎౓ͷվળ OS/Kernel tuning, Scalability ޲্ όοΫΞοϓӡ༻खॱͷڞ௨Խ, Rolling Schema Upgrade DNS-RR, nginx Internal DNS, Consul memcached on EC2 MySQL on EC2 ϚωʔδυαʔϏε͔Βαʔόத৺ͷΞʔΩςΫνϟ΁
  9. Why Not 
 SERVERLESS Architecture • ʮϚωʔδυαʔϏεΛ࢖͏͜ͱͰΞϓϦέʔ γϣϯ։ൃʹઐ೦ʯͱ͍͏ϋφγ • ΑΓߴ͍Ϩϕϧͷ৴པੑΛSRE͕ఏڙ͢Δ

    • ௿͍ϨΠϠʔ͔Βͷ࠷దԽͷੵΈॏͶ • SREʹΑΔ৴པੑͷఏڙ͕೉͍͠΋ͷ͸ੵۃతʹ
 ϚωʔδυαʔϏεʗΫϥ΢υαʔϏεΛ׆༻
  10. JP US Akamai, BigQuery, mackerel SendGrid, Route53, S3 Ansible, OS/Kernel/Middleware

    Tuning ಺෦DNS, Consul ֤Region/Serverʹಋೖ CloudαʔϏε
  11. Usage of Consul in Mercari • Service Discovery • SMTP

    Server • Endpoint of Internal API • Configuration Deployment • TLS Session Ticket • Auto update IP black list • Distributed Lock
  12. SMTP Server Discovery { "service": { "name": “mail-delivery“, "port": 25,

    "checks": [ { "script": "sh /etc/consul.d/check_mail.sh","interval": "10s" } ] } } #!/bin/bash set -e echo "test" | mail -S "smtp=smtp://localhost:25" -r no-reply@shiroyagi no- reply@kuroyagi find /var/log/maillog -mmin -3 | xargs grep kuroyagi|grep "250 Ok" check_mail.sh mail-delivery.json ϝʔϧૹ৴ʹҟৗ͕ ͋ΔͱαʔϏε͔Β ੾Γ཭͞ΕΔ
  13. Consul maint is Useful $ consul maint -enable $ consul

    maint -disable shell shell ଈ࣌(+DNS cache)ͰαʔϏε͔Β֎ΕΔ ϝϯςφϯεʗ࠶ىಈ ଈ࣌(+DNS cache)ͰαʔϏε෮چ ໭͠๨Εʹ ஫ҙ!!!
  14. Migration from Zabbix ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US

    West (Oregon) Region JP US Z ZABBIX Z ZABBIX MackerelͰͷJP/USҰׅ؂ࢹ RoleΛ࢖͏͜ͱͰ؂ࢹϧʔϧͷJP/USڞ௨Խ
  15. Inside mackerel report JP US fping -a -g 10.0.x.0/24 Λ࢖͍

    IPΞυϨεҰཡΛ࡞੒ aws ίϚϯυΛ࢖͍
 IPΞυϨεҰཡΛऔಘ mkr ίϚϯυͰऔಘͨ͠؂ࢹର৅αʔόҰཡͰϑΟϧλ Slack௨஌