Hybrid SERVER Architecture in Mercari

by kazeburo

Slide 1

Slide 1 text

Hybrid SERVER Architecture in Mercari Masahiro Nagano @kazéburo SRE Tech Talks #1

Slide 2

Slide 2 text

SRE εςοΧʔ࡞Γ·ͨ͠!!

Slide 3

Slide 3 text

Me • Masahiro Nagano • @kazeburo • Mercari, Inc.  Principal Engineer  Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

SRE@Mercari • Operations • γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺαʔϏεͷ৴པੑΛ ޲্ͤ͞Δ • Software Engineering • αʔϏεΛεέʔϧͤ͞Δϛυϧ΢ΣΞͷ։ൃɾӡ༻ • ։ൃ؀ڥͷվળɾσϓϩΠπʔϧͷ੔උ

Slide 6

Slide 6 text

SRE@Mercari ࣄྫ • Gaurun - push notiﬁcation server • ngx_luaʹΑΔBalancer֦ு  - http://tech.mercari.com/entry/2015/11/25/170049 • ChatOps - http://tech.mercari.com/entry/2015/10/15/183000 • 20+ Mackerel Plugins  - https://github.com/kazeburo/custom-mackerel-plugins

Slide 7

Slide 7 text

Hybrid SERVER Architecture  in Mercari

Slide 8

Slide 8 text

Mercari Your Friendly Mobile MarketPlace JP US

Slide 9

Slide 9 text

Mercari KPI DOWNLOAD GMV LISTED ITEMS 4000ສDL(JP+US) ݄ؒ100ԯԁҎ্ 1೔50ສ඼Ҏ্

Slide 10

Slide 10 text

Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US

Slide 11

Slide 11 text

͘͞Βͷઐ༻αʔό

Slide 12

Slide 12 text

͘͞Βͷઐ༻αʔό • ίϩέʔγϣϯ/ϨϯλϧαʔόͰ͸ͳ͘ɺMaaS • ioMemoryɺNVMe SSD ౥ࡌՄೳ • ίϯτϩʔϧύωϧ͔Β࠶ىಈɺOSͷ࠶Πϯετʔϧɺ  ίϯιʔϧ΁ͷΞΫηε • ϓϥΠϕʔτωοτϫʔΫͷߏங  ͘͞ΒͷΫϥ΢υɺίϩέʔγϣϯαʔϏεͱͷ૬ޓ઀ଓ

Slide 13

Slide 13 text

Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP util util cloud cloud

Slide 14

Slide 14 text

Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP ಉ͡ߏ੒ util util cloud cloud util util EC2 EC2

Slide 15

Slide 15 text

HyBrid Architecture • ΞϓϦέʔγϣϯίʔυͷෳࡶੑͷআڈ • `if $region === “US”` `if $region === “JP”` • playbookΛڞ༗ɺಉ͡ϝϯςφϏϦςΟ • ಉ͡εέʔϧઓུ

Slide 16

Slide 16 text

Hyper Hybrid Architecture UK ४උத JP US ͘͞Β AWS ??? Region/λΠϛϯάͰ࠷దͳΠϯϑϥετϥΫνϟͷબ୒ ٕज़ͷ෯͕޿͕Δ

Slide 17

Slide 17 text

US Architecture history • ELB • Internal ELB • ElastiCache • RDS HTTP/2(SPDY), ngx_dynamic_upstreamʹΑΔແఀࢭdeploy ಥൃతͳΞΫηε଱ੑ, ઃఆ൓ө଎౓ͷվળ OS/Kernel tuning, Scalability ޲্ όοΫΞοϓӡ༻खॱͷڞ௨Խ, Rolling Schema Upgrade DNS-RR, nginx Internal DNS, Consul memcached on EC2 MySQL on EC2 ϚωʔδυαʔϏε͔Βαʔόத৺ͷΞʔΩςΫνϟ΁

Slide 18

Slide 18 text

Why SERVER Architecture?

Slide 19

Slide 19 text

Why SERVER Architecture • ৽͍ٕ͠ज़΁ͷରԠ • εέʔϥϏϦςΟͷ੍ޚ • OS/Kernel, ϛυϧ΢ΣΞͷνϡʔχϯά  ϞχλϦϯά • ن໛ͷ֦େ͕ઌߦ͍ͯ͠ΔJPͷӡ༻ͱڞ௨Խ

Slide 20

Slide 20 text

Why Not   SERVERLESS Architecture • ʮϚωʔδυαʔϏεΛ࢖͏͜ͱͰΞϓϦέʔ γϣϯ։ൃʹઐ೦ʯͱ͍͏ϋφγ • ΑΓߴ͍Ϩϕϧͷ৴པੑΛSRE͕ఏڙ͢Δ • ௿͍ϨΠϠʔ͔Βͷ࠷దԽͷੵΈॏͶ • SREʹΑΔ৴པੑͷఏڙ͕೉͍͠΋ͷ͸ੵۃతʹ  ϚωʔδυαʔϏεʗΫϥ΢υαʔϏεΛ׆༻

Slide 21

Slide 21 text

SERVER Architecture  Λࢧ͑Δٕज़

Slide 22

Slide 22 text

JP US Akamai, BigQuery, mackerel SendGrid, Route53, S3 Ansible, OS/Kernel/Middleware Tuning ಺෦DNS, Consul ֤Region/Serverʹಋೖ CloudαʔϏε

Slide 23

Slide 23 text

Consul

Slide 24

Slide 24 text

Usage of Consul in Mercari • Service Discovery • SMTP Server • Endpoint of Internal API • Conﬁguration Deployment • TLS Session Ticket • Auto update IP black list • Distributed Lock

Slide 25

Slide 25 text

SMTP Server Discovery { "service": { "name": “mail-delivery“, "port": 25, "checks": [ { "script": "sh /etc/consul.d/check_mail.sh","interval": "10s" } ] } } #!/bin/bash set -e echo "test" | mail -S "smtp=smtp://localhost:25" -r no-reply@shiroyagi no- reply@kuroyagi find /var/log/maillog -mmin -3 | xargs grep kuroyagi|grep "250 Ok" check_mail.sh mail-delivery.json ϝʔϧૹ৴ʹҟৗ͕ ͋ΔͱαʔϏε͔Β ੾Γ཭͞ΕΔ

Slide 26

Slide 26 text

Consul maint is Useful $ consul maint -enable $ consul maint -disable shell shell ଈ࣌(+DNS cache)ͰαʔϏε͔Β֎ΕΔ ϝϯςφϯεʗ࠶ىಈ ଈ࣌(+DNS cache)ͰαʔϏε෮چ ໭͠๨Εʹ ஫ҙ!!!

Slide 27

Slide 27 text

http://connpass.com/event/35836/ @cubicdaiya ͕ MercariͰͷConsulͷࣄྫΛ࿩͠·͢

Slide 28

Slide 28 text

mackerel

Slide 29

Slide 29 text

Migration from Zabbix ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US Z ZABBIX Z ZABBIX MackerelͰͷJP/USҰׅ؂ࢹ RoleΛ࢖͏͜ͱͰ؂ࢹϧʔϧͷJP/USڞ௨Խ

Slide 30

Slide 30 text

Prevent unmonitored Host Ҡߦʗ࡞ۀ࣌ʹ؂ࢹ࿙ΕΛ๷͙ͨΊʹ ϨϙʔτΛ1೔ʹ2ճ࡞੒ͯ͠Slackʹ౤ߘ

Slide 31

Slide 31 text

Inside mackerel report JP US fping -a -g 10.0.x.0/24 Λ࢖͍ IPΞυϨεҰཡΛ࡞੒ aws ίϚϯυΛ࢖͍  IPΞυϨεҰཡΛऔಘ mkr ίϚϯυͰऔಘͨ͠؂ࢹର৅αʔόҰཡͰϑΟϧλ Slack௨஌

Slide 32

Slide 32 text

·ͱΊ

Slide 33

Slide 33 text

Conclusion • ෺ཧαʔόͱԾ૝αʔόͷ Hybrid Architecture • ServerΛத৺ͱͨ͠ߏ੒Ͱɺߴ͍৴པੑΛ  SRE͕ఏڙ͢Δ • mercariͷ Hybrid SERVER Architecture Λࢧ͑Δ  ٕज़ͱͯ͠Consul΍mackerelΛࣄྫΛ঺հ