Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
2年間運用しているソーシャルゲームのサーバ構成の変遷
Search
tkuchiki
January 27, 2017
Programming
3
5k
2年間運用しているソーシャルゲームのサーバ構成の変遷
tkuchiki
January 27, 2017
Tweet
Share
More Decks by tkuchiki
See All by tkuchiki
Kubernetesの上に作る、統一されたマイクロサービス運用体験
tkuchiki
1
8.2k
メルペイSREチームのオンコール/oncall-merpay-sre
tkuchiki
0
270
/sre-next-2020
tkuchiki
5
15k
/fastly-meetup-3
tkuchiki
0
100
/fastly-yamagoya-meetup-2019
tkuchiki
0
220
/cloud-spanner-operation-on-merpay
tkuchiki
1
1.5k
ソーシャルゲームにおける AWS 移行事例
tkuchiki
21
24k
Other Decks in Programming
See All in Programming
Tool Catalog Agent for Bedrock AgentCore Gateway
licux
7
2.5k
Navigating Dependency Injection with Metro
zacsweers
3
2.5k
AIでLINEスタンプを作ってみた
eycjur
1
230
アルテニア コンサル/ITエンジニア向け 採用ピッチ資料
altenir
0
110
個人開発で徳島大学生60%以上の心を掴んだアプリ、そして手放した話
akidon0000
1
150
そのAPI、誰のため? Androidライブラリ設計における利用者目線の実践テクニック
mkeeda
2
1.8k
基礎から学ぶ大画面対応(Learning Large-Screen Support from the Ground Up)
tomoya0x00
0
3.3k
Cache Me If You Can
ryunen344
2
3k
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
250
AWS発のAIエディタKiroを使ってみた
iriikeita
1
190
Performance for Conversion! 分散トレーシングでボトルネックを 特定せよ
inetand
0
2.4k
Amazon RDS 向けに提供されている MCP Server と仕組みを調べてみた/jawsug-okayama-2025-aurora-mcp
takahashiikki
1
110
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
Building Adaptive Systems
keathley
43
2.7k
Practical Orchestrator
shlominoach
190
11k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
Making Projects Easy
brettharned
117
6.4k
Facilitating Awesome Meetings
lara
55
6.5k
Bash Introduction
62gerente
615
210k
Agile that works and the tools we love
rasmusluckow
330
21k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
810
Large-scale JavaScript Application Architecture
addyosmani
513
110k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Transcript
2ؒӡ༻͍ͯ͠Δ ιʔγϟϧήʔϜͷ αʔόߏͷมભ @tkuchiki
※ެ։༻ʹҰ෦εϥΠυΛ আ͓ͯ͠Γ·͢
WHOAMI(1) • @tkuchiki • ΠϯϑϥνʔϜ • ୲ • ιʔγϟϧήʔϜ •
ࣗࣾαʔϏε • Web+DB PRESS Vol.94
Agenda • αʔόߏͷมભ • ղܾฤ • Auto Scaling ฤ •
Replace ฤ
ղܾฤ
ϦϦʔε࣌ͷαʔόߏ
ϛυϧΣΞ • શڞ௨ • Perl 5.16.3 • consul + dnsmasq
• ෦ DNS • Service Discovery(DNS) • zabbix-agent
ϛυϧΣΞ • Openresty 1.4.6 • MySQL 5.6(RDS) • zabbix-proxy
ϛυϧΣΞ • Redis + Redis Sentinel • master ͷ IP
Λ Sentinel ʹ͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue
ϛυϧΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3,
numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάूଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd
1core ͔͑͠ͳ͍ • Nginx ͷΞΫηεϩάूͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάूͱͦΕҎ֎Λ͚Δ
σΟεΫ༰ྔ͕ᷓΕͦ͏ʹͳΔ • fluentd ͷෛՙݮޭ • ࠓσΟεΫ༰ྔ͕Γͳ͘ͳΔ • cron.daily ͰϩάͷѹॖΛͯ͠ɺҰఆظؒա ͗ͨΒআ͍ͯͨ͠
• σΟεΫΛՃͯ͠ cron.daily ͰϩάΛୀආ
·ͩσΟεΫ༰ྔ͕ᷓΕͦ͏ • 11ճͷϩάୀආͰؒʹ߹Θͳ͘ͳΔ • ຖ࣌ϩάΛѹॖͱୀආ
MySQL ߴෛՙ • MySQL 1 Ͱࡹ͖͖Εͳ͍ෛՙ • Slave(Read Replica) ΛՃͯ͠ࢀরࢄ
• HAProxy ಋೖ • ͍ͭͰʹཧը໘༻ͷ Slave ΛՃ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
ࢼ߹ Batch ͷ࣮ߦ͕͍࣌ؒ • Ϣʔβ૿Ճʹ͍ࢼ߹૿Ճ • Batch ͷ࣮ߦྃ·Ͱͷ͕࣌ؒͲΜͲΜ৳ͼ ͍ͯ͘ •
εέʔϧΞοϓͰରԠ • ݱࡏεέʔϧΞτ͠ͳ͍ߏͳͷͰࠓ ޙͷ՝
fluent-plugin-redshift͕͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌ͳ͔͕ͬͨɺΞΫηε͕૿
͔͑ͯΒ࣌ʑ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • Ҏ্ଓ͖·ͨ͠
rin • https://github.com/fujiwara/rin • S3 ͷΠϕϯτ௨Ͱ SQS ʹΩϡʔΛՃ • SQS
ͷΩϡʔΛऔΓग़ͯ͠ redshift ʹ COPY ίϚϯυΛ࣮ߦ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)
ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for
Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)
҆ఆظʹಥೖ • ϦϦʔεޙ1͘Β͍Ͱେ͖ͳղফ • վળʹऔΓΊΔΑ͏ʹͳΔ
Auto Scaling ฤ
αʔόͷखಈՃ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰՃ • AWS CLI Ͱ EC2
Λىಈͯ͠ ELB ʹՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞ • Management Console ͕৮Εͳͯ͘࡞ ۀ͕Ͱ͖Δ
αʔόͷखಈՃ • ࡞ۀͰ͖Δ͕... • ෛՙ͕͍ͱαʔόϦιʔε͕༨Δ • ༧ظͤ͵ෛՙ૿େʹରԠͰ͖ͳ͍ • EC2 ͕ಥવࢮ͢ΔͱखಈͰ෮׆ͤ͞Δඞཁ͕
͋ͬͨ • ࠓ CloudWatch Alarm Ͱࣗಈ࠶ىಈͤ͞Δ ͜ͱ͕Ͱ͖·͢
Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ •
Πϕϯτ։࢝લʹ minimum capacity Λ૿ ͢ඞཁ͕͋ΔͨΊશʹͳ͘͢͜ͱͰ͖ ͳ͍
Auto Scaling ಋೖͷ • Ͳ͏ͬͯαʔόΛ࠷৽ͷঢ়ଶʹ͢Δʁ • ΞϓϦέʔγϣϯίʔυɾαʔόઃఆΛम ਖ਼͢Δʹ AMI Λ࡞ʁ
• αʔόىಈ࣌ʹ deploy ͢Δʁ • deploy αʔό͔ΒίʔυΛऔͬͯ͘Δʁ
Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ
औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹར༻͍ͯ͠Δ
# stretcher $ consul watch -type event -name deploy stretcher
# deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱՄೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ
Auto Scaling ͷΈ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest
ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚ΕΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛՃ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ আ & Consul Cluster ͔Β leave
Auto Scaling ಋೖͷޮՌ • ॳͷతୡ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ •
Pull ܕ deploy ʹมߋ͢Δ͜ͱͰ deploy ࣌ؒ ॖ
Replace ฤ
৽ήʔϜαʔόϦϦʔε • 20168݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱશ͘ผͷڥͱͯ͠ߏங • ιʔείʔυڞ༗ • DB ޓ͍ʹࢀর͋͠Θͳ͍
• ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘڥ Replace ͢Δ͜ͱʹ
طଘήʔϜαʔόͷ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓݹ͍
• ߏ͕ݹ͍ɾྑ͘ͳ͍ • ϩάूαʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ Norikra Ͱ... • etc
αʔόߏͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ࠷৽(c4,
m4)ʹ • ϩάूαʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora
αʔόߏͷ Replace • ϛυϧΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy
• td-agent ΛΊͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess ΛΘ ͳ͍Α͏ʹ
αʔόߏͷ Replace • Consul ͷ External Service Λ׆༻ • Consul
Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY 0QFOSFTUZ qVFOUBHFOUIZESB qVFOUE )"1SPYZ qVFOUBHFOUIZESB HFBSNBOE Log Log( +Backup) Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1 qVFOUE SJO qVFOUE /PSJLSB Log Replace ޙ
Replace ͷޮՌ • ϩάूαʔόͷೖΕସ͕͑(΄΅)ແఀࢭͰߦ ͑Δ • Aurora มߋޙɺEBS ͷϨΠςϯγ૿ʹΑΔΫ ΤϦੑೳͷྼԽ͕ൃੜ͠ͳ͘ͳΔ
• Norikra + fluentd + fluent-plugin-slack ͰϦΞ ϧλΠϜͳΤϥʔ௨͕Մೳʹ
Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू αʔόͷෛՙݮ • Zabbix ͷϝτϦΫεૹ৴
https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Βૹ৴
Replace ͷޮՌ • ϝϯςφϯεঢ়ଶͷΓସ͕͑ߴʹ • Nginx ͰϑΝΠϧͷ༗ແΛݟͯ 503 Λฦ͠ ͍ͯͨͨΊɺͦͷͨΊ͚ͩʹ
deploy ͢Δ͔ શʹϑΝΠϧΛΔɾফ͢࡞ۀ͕ඞཁͩͬ ͨ
Replace ͷޮՌ • ΠϯελϯελΠϓΛมߋͨ͜͠ͱͰ CPU ͷ ॲཧੑೳ্͕ • ϨεϙϯελΠϜ͕(ଟগ)ߴʹ
՝ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖͑Δ •
Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ