Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
2年間運用しているソーシャルゲームのサーバ構成の変遷
Search
tkuchiki
January 27, 2017
Programming
3
4.7k
2年間運用しているソーシャルゲームのサーバ構成の変遷
tkuchiki
January 27, 2017
Tweet
Share
More Decks by tkuchiki
See All by tkuchiki
Kubernetesの上に作る、統一されたマイクロサービス運用体験
tkuchiki
1
8.1k
メルペイSREチームのオンコール/oncall-merpay-sre
tkuchiki
0
140
/sre-next-2020
tkuchiki
5
14k
/fastly-meetup-3
tkuchiki
0
86
/fastly-yamagoya-meetup-2019
tkuchiki
0
180
/cloud-spanner-operation-on-merpay
tkuchiki
1
1.3k
ソーシャルゲームにおける AWS 移行事例
tkuchiki
21
23k
Other Decks in Programming
See All in Programming
DynamoDB コスト最適化っぽいことの基本 with Terraform
kuro_kurorrr
2
250
ぼっちを避けて楽しむためのアノテコノテ / Various Tips and Tricks to Avoid Loneliness and Have Fun
nrslib
3
1.7k
小さな開発会社を作った理由
polidog
0
1.9k
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
Trial
cairolibrary720
1
130
企業向け生成AIアプリの 開発から得られた知見
takaakikakei
0
310
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
450
CSC307 Lecture 11
javiergs
PRO
0
240
ドメイン駆動設計の実践
masuda220
PRO
17
5.2k
初心者がおさえておきたいAWS CDKのベストプラクティス 2024
konokenj
15
7.3k
Mastering Developer Experience: A Roadmap for Success 【開発生産性Conference 2024】
findyinc
1
380
Architectures with Lightweight Stores: New Rules and Options
manfredsteyer
PRO
0
100
Featured
See All Featured
Teambox: Starting and Learning
jrom
130
8.6k
We Have a Design System, Now What?
morganepeng
46
7k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
The Invisible Side of Design
smashingmag
294
50k
Documentation Writing (for coders)
carmenintech
63
4.2k
KATA
mclloyd
20
13k
Ruby is Unlike a Banana
tanoku
96
10k
Adopting Sorbet at Scale
ufuk
71
8.8k
GraphQLとの向き合い方2022年版
quramy
36
13k
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
Producing Creativity
orderedlist
PRO
340
39k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.5k
Transcript
2ؒӡ༻͍ͯ͠Δ ιʔγϟϧήʔϜͷ αʔόߏͷมભ @tkuchiki
※ެ։༻ʹҰ෦εϥΠυΛ আ͓ͯ͠Γ·͢
WHOAMI(1) • @tkuchiki • ΠϯϑϥνʔϜ • ୲ • ιʔγϟϧήʔϜ •
ࣗࣾαʔϏε • Web+DB PRESS Vol.94
Agenda • αʔόߏͷมભ • ղܾฤ • Auto Scaling ฤ •
Replace ฤ
ղܾฤ
ϦϦʔε࣌ͷαʔόߏ
ϛυϧΣΞ • શڞ௨ • Perl 5.16.3 • consul + dnsmasq
• ෦ DNS • Service Discovery(DNS) • zabbix-agent
ϛυϧΣΞ • Openresty 1.4.6 • MySQL 5.6(RDS) • zabbix-proxy
ϛυϧΣΞ • Redis + Redis Sentinel • master ͷ IP
Λ Sentinel ʹ͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue
ϛυϧΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3,
numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάूଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd
1core ͔͑͠ͳ͍ • Nginx ͷΞΫηεϩάूͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάूͱͦΕҎ֎Λ͚Δ
σΟεΫ༰ྔ͕ᷓΕͦ͏ʹͳΔ • fluentd ͷෛՙݮޭ • ࠓσΟεΫ༰ྔ͕Γͳ͘ͳΔ • cron.daily ͰϩάͷѹॖΛͯ͠ɺҰఆظؒա ͗ͨΒআ͍ͯͨ͠
• σΟεΫΛՃͯ͠ cron.daily ͰϩάΛୀආ
·ͩσΟεΫ༰ྔ͕ᷓΕͦ͏ • 11ճͷϩάୀආͰؒʹ߹Θͳ͘ͳΔ • ຖ࣌ϩάΛѹॖͱୀආ
MySQL ߴෛՙ • MySQL 1 Ͱࡹ͖͖Εͳ͍ෛՙ • Slave(Read Replica) ΛՃͯ͠ࢀরࢄ
• HAProxy ಋೖ • ͍ͭͰʹཧը໘༻ͷ Slave ΛՃ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
ࢼ߹ Batch ͷ࣮ߦ͕͍࣌ؒ • Ϣʔβ૿Ճʹ͍ࢼ߹૿Ճ • Batch ͷ࣮ߦྃ·Ͱͷ͕࣌ؒͲΜͲΜ৳ͼ ͍ͯ͘ •
εέʔϧΞοϓͰରԠ • ݱࡏεέʔϧΞτ͠ͳ͍ߏͳͷͰࠓ ޙͷ՝
fluent-plugin-redshift͕͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌ͳ͔͕ͬͨɺΞΫηε͕૿
͔͑ͯΒ࣌ʑ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • Ҏ্ଓ͖·ͨ͠
rin • https://github.com/fujiwara/rin • S3 ͷΠϕϯτ௨Ͱ SQS ʹΩϡʔΛՃ • SQS
ͷΩϡʔΛऔΓग़ͯ͠ redshift ʹ COPY ίϚϯυΛ࣮ߦ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)
ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for
Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)
҆ఆظʹಥೖ • ϦϦʔεޙ1͘Β͍Ͱେ͖ͳղফ • վળʹऔΓΊΔΑ͏ʹͳΔ
Auto Scaling ฤ
αʔόͷखಈՃ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰՃ • AWS CLI Ͱ EC2
Λىಈͯ͠ ELB ʹՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞ • Management Console ͕৮Εͳͯ͘࡞ ۀ͕Ͱ͖Δ
αʔόͷखಈՃ • ࡞ۀͰ͖Δ͕... • ෛՙ͕͍ͱαʔόϦιʔε͕༨Δ • ༧ظͤ͵ෛՙ૿େʹରԠͰ͖ͳ͍ • EC2 ͕ಥવࢮ͢ΔͱखಈͰ෮׆ͤ͞Δඞཁ͕
͋ͬͨ • ࠓ CloudWatch Alarm Ͱࣗಈ࠶ىಈͤ͞Δ ͜ͱ͕Ͱ͖·͢
Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ •
Πϕϯτ։࢝લʹ minimum capacity Λ૿ ͢ඞཁ͕͋ΔͨΊશʹͳ͘͢͜ͱͰ͖ ͳ͍
Auto Scaling ಋೖͷ • Ͳ͏ͬͯαʔόΛ࠷৽ͷঢ়ଶʹ͢Δʁ • ΞϓϦέʔγϣϯίʔυɾαʔόઃఆΛम ਖ਼͢Δʹ AMI Λ࡞ʁ
• αʔόىಈ࣌ʹ deploy ͢Δʁ • deploy αʔό͔ΒίʔυΛऔͬͯ͘Δʁ
Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ
औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹར༻͍ͯ͠Δ
# stretcher $ consul watch -type event -name deploy stretcher
# deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱՄೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ
Auto Scaling ͷΈ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest
ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚ΕΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛՃ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ আ & Consul Cluster ͔Β leave
Auto Scaling ಋೖͷޮՌ • ॳͷతୡ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ •
Pull ܕ deploy ʹมߋ͢Δ͜ͱͰ deploy ࣌ؒ ॖ
Replace ฤ
৽ήʔϜαʔόϦϦʔε • 20168݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱશ͘ผͷڥͱͯ͠ߏங • ιʔείʔυڞ༗ • DB ޓ͍ʹࢀর͋͠Θͳ͍
• ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘڥ Replace ͢Δ͜ͱʹ
طଘήʔϜαʔόͷ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓݹ͍
• ߏ͕ݹ͍ɾྑ͘ͳ͍ • ϩάूαʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ Norikra Ͱ... • etc
αʔόߏͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ࠷৽(c4,
m4)ʹ • ϩάूαʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora
αʔόߏͷ Replace • ϛυϧΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy
• td-agent ΛΊͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess ΛΘ ͳ͍Α͏ʹ
αʔόߏͷ Replace • Consul ͷ External Service Λ׆༻ • Consul
Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY 0QFOSFTUZ qVFOUBHFOUIZESB qVFOUE )"1SPYZ qVFOUBHFOUIZESB HFBSNBOE Log Log( +Backup) Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1 qVFOUE SJO qVFOUE /PSJLSB Log Replace ޙ
Replace ͷޮՌ • ϩάूαʔόͷೖΕସ͕͑(΄΅)ແఀࢭͰߦ ͑Δ • Aurora มߋޙɺEBS ͷϨΠςϯγ૿ʹΑΔΫ ΤϦੑೳͷྼԽ͕ൃੜ͠ͳ͘ͳΔ
• Norikra + fluentd + fluent-plugin-slack ͰϦΞ ϧλΠϜͳΤϥʔ௨͕Մೳʹ
Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू αʔόͷෛՙݮ • Zabbix ͷϝτϦΫεૹ৴
https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Βૹ৴
Replace ͷޮՌ • ϝϯςφϯεঢ়ଶͷΓସ͕͑ߴʹ • Nginx ͰϑΝΠϧͷ༗ແΛݟͯ 503 Λฦ͠ ͍ͯͨͨΊɺͦͷͨΊ͚ͩʹ
deploy ͢Δ͔ શʹϑΝΠϧΛΔɾফ͢࡞ۀ͕ඞཁͩͬ ͨ
Replace ͷޮՌ • ΠϯελϯελΠϓΛมߋͨ͜͠ͱͰ CPU ͷ ॲཧੑೳ্͕ • ϨεϙϯελΠϜ͕(ଟগ)ߴʹ
՝ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖͑Δ •
Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ