Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
2年間運用しているソーシャルゲームのサーバ構成の変遷
Search
tkuchiki
January 27, 2017
Programming
3
4.9k
2年間運用しているソーシャルゲームのサーバ構成の変遷
tkuchiki
January 27, 2017
Tweet
Share
More Decks by tkuchiki
See All by tkuchiki
Kubernetesの上に作る、統一されたマイクロサービス運用体験
tkuchiki
1
8.2k
メルペイSREチームのオンコール/oncall-merpay-sre
tkuchiki
0
270
/sre-next-2020
tkuchiki
5
15k
/fastly-meetup-3
tkuchiki
0
100
/fastly-yamagoya-meetup-2019
tkuchiki
0
210
/cloud-spanner-operation-on-merpay
tkuchiki
1
1.5k
ソーシャルゲームにおける AWS 移行事例
tkuchiki
21
24k
Other Decks in Programming
See All in Programming
AIコーディングエージェント全社導入とセキュリティ対策
hikaruegashira
15
8.4k
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ~継続的な取り組みを添えて~
takutakahashi
4
1.4k
AI Ramen Fight
yusukebe
0
120
No Install CMS戦略 〜 5年先を見据えたフロントエンド開発を考える / no_install_cms
rdlabo
0
370
Gemini CLI のはじめ方
ttnyt8701
1
110
GPUを計算資源として使おう!
primenumber
1
290
ZeroETLで始めるDynamoDBとS3の連携
afooooil
0
130
202507_ADKで始めるエージェント開発の基本 〜デモを通じて紹介〜(奥田りさ)The Basics of Agent Development with ADK — A Demo-Focused Introduction
risatube
PRO
5
1.2k
ソフトウェア設計とAI技術の活用
masuda220
PRO
25
6.9k
変化を楽しむエンジニアリング ~ いままでとこれから ~
murajun1978
0
520
Bedrock AgentCore ObservabilityによるAIエージェントの運用
licux
8
340
なぜあなたのオブザーバビリティ導入は頓挫するのか
ryota_hnk
3
510
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
Facilitating Awesome Meetings
lara
54
6.5k
Designing for humans not robots
tammielis
253
25k
Building Applications with DynamoDB
mza
95
6.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
What's in a price? How to price your products and services
michaelherold
246
12k
Scaling GitHub
holman
461
140k
We Have a Design System, Now What?
morganepeng
53
7.7k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Transcript
2ؒӡ༻͍ͯ͠Δ ιʔγϟϧήʔϜͷ αʔόߏͷมભ @tkuchiki
※ެ։༻ʹҰ෦εϥΠυΛ আ͓ͯ͠Γ·͢
WHOAMI(1) • @tkuchiki • ΠϯϑϥνʔϜ • ୲ • ιʔγϟϧήʔϜ •
ࣗࣾαʔϏε • Web+DB PRESS Vol.94
Agenda • αʔόߏͷมભ • ղܾฤ • Auto Scaling ฤ •
Replace ฤ
ղܾฤ
ϦϦʔε࣌ͷαʔόߏ
ϛυϧΣΞ • શڞ௨ • Perl 5.16.3 • consul + dnsmasq
• ෦ DNS • Service Discovery(DNS) • zabbix-agent
ϛυϧΣΞ • Openresty 1.4.6 • MySQL 5.6(RDS) • zabbix-proxy
ϛυϧΣΞ • Redis + Redis Sentinel • master ͷ IP
Λ Sentinel ʹ͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue
ϛυϧΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3,
numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάूଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd
1core ͔͑͠ͳ͍ • Nginx ͷΞΫηεϩάूͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάूͱͦΕҎ֎Λ͚Δ
σΟεΫ༰ྔ͕ᷓΕͦ͏ʹͳΔ • fluentd ͷෛՙݮޭ • ࠓσΟεΫ༰ྔ͕Γͳ͘ͳΔ • cron.daily ͰϩάͷѹॖΛͯ͠ɺҰఆظؒա ͗ͨΒআ͍ͯͨ͠
• σΟεΫΛՃͯ͠ cron.daily ͰϩάΛୀආ
·ͩσΟεΫ༰ྔ͕ᷓΕͦ͏ • 11ճͷϩάୀආͰؒʹ߹Θͳ͘ͳΔ • ຖ࣌ϩάΛѹॖͱୀආ
MySQL ߴෛՙ • MySQL 1 Ͱࡹ͖͖Εͳ͍ෛՙ • Slave(Read Replica) ΛՃͯ͠ࢀরࢄ
• HAProxy ಋೖ • ͍ͭͰʹཧը໘༻ͷ Slave ΛՃ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
ࢼ߹ Batch ͷ࣮ߦ͕͍࣌ؒ • Ϣʔβ૿Ճʹ͍ࢼ߹૿Ճ • Batch ͷ࣮ߦྃ·Ͱͷ͕࣌ؒͲΜͲΜ৳ͼ ͍ͯ͘ •
εέʔϧΞοϓͰରԠ • ݱࡏεέʔϧΞτ͠ͳ͍ߏͳͷͰࠓ ޙͷ՝
fluent-plugin-redshift͕͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌ͳ͔͕ͬͨɺΞΫηε͕૿
͔͑ͯΒ࣌ʑ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • Ҏ্ଓ͖·ͨ͠
rin • https://github.com/fujiwara/rin • S3 ͷΠϕϯτ௨Ͱ SQS ʹΩϡʔΛՃ • SQS
ͷΩϡʔΛऔΓग़ͯ͠ redshift ʹ COPY ίϚϯυΛ࣮ߦ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)
ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for
Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)
҆ఆظʹಥೖ • ϦϦʔεޙ1͘Β͍Ͱେ͖ͳղফ • վળʹऔΓΊΔΑ͏ʹͳΔ
Auto Scaling ฤ
αʔόͷखಈՃ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰՃ • AWS CLI Ͱ EC2
Λىಈͯ͠ ELB ʹՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞ • Management Console ͕৮Εͳͯ͘࡞ ۀ͕Ͱ͖Δ
αʔόͷखಈՃ • ࡞ۀͰ͖Δ͕... • ෛՙ͕͍ͱαʔόϦιʔε͕༨Δ • ༧ظͤ͵ෛՙ૿େʹରԠͰ͖ͳ͍ • EC2 ͕ಥવࢮ͢ΔͱखಈͰ෮׆ͤ͞Δඞཁ͕
͋ͬͨ • ࠓ CloudWatch Alarm Ͱࣗಈ࠶ىಈͤ͞Δ ͜ͱ͕Ͱ͖·͢
Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ •
Πϕϯτ։࢝લʹ minimum capacity Λ૿ ͢ඞཁ͕͋ΔͨΊશʹͳ͘͢͜ͱͰ͖ ͳ͍
Auto Scaling ಋೖͷ • Ͳ͏ͬͯαʔόΛ࠷৽ͷঢ়ଶʹ͢Δʁ • ΞϓϦέʔγϣϯίʔυɾαʔόઃఆΛम ਖ਼͢Δʹ AMI Λ࡞ʁ
• αʔόىಈ࣌ʹ deploy ͢Δʁ • deploy αʔό͔ΒίʔυΛऔͬͯ͘Δʁ
Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ
औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹར༻͍ͯ͠Δ
# stretcher $ consul watch -type event -name deploy stretcher
# deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱՄೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ
Auto Scaling ͷΈ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest
ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚ΕΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛՃ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ আ & Consul Cluster ͔Β leave
Auto Scaling ಋೖͷޮՌ • ॳͷతୡ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ •
Pull ܕ deploy ʹมߋ͢Δ͜ͱͰ deploy ࣌ؒ ॖ
Replace ฤ
৽ήʔϜαʔόϦϦʔε • 20168݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱશ͘ผͷڥͱͯ͠ߏங • ιʔείʔυڞ༗ • DB ޓ͍ʹࢀর͋͠Θͳ͍
• ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘڥ Replace ͢Δ͜ͱʹ
طଘήʔϜαʔόͷ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓݹ͍
• ߏ͕ݹ͍ɾྑ͘ͳ͍ • ϩάूαʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ Norikra Ͱ... • etc
αʔόߏͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ࠷৽(c4,
m4)ʹ • ϩάूαʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora
αʔόߏͷ Replace • ϛυϧΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy
• td-agent ΛΊͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess ΛΘ ͳ͍Α͏ʹ
αʔόߏͷ Replace • Consul ͷ External Service Λ׆༻ • Consul
Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY 0QFOSFTUZ qVFOUBHFOUIZESB qVFOUE )"1SPYZ qVFOUBHFOUIZESB HFBSNBOE Log Log( +Backup) Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1 qVFOUE SJO qVFOUE /PSJLSB Log Replace ޙ
Replace ͷޮՌ • ϩάूαʔόͷೖΕସ͕͑(΄΅)ແఀࢭͰߦ ͑Δ • Aurora มߋޙɺEBS ͷϨΠςϯγ૿ʹΑΔΫ ΤϦੑೳͷྼԽ͕ൃੜ͠ͳ͘ͳΔ
• Norikra + fluentd + fluent-plugin-slack ͰϦΞ ϧλΠϜͳΤϥʔ௨͕Մೳʹ
Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू αʔόͷෛՙݮ • Zabbix ͷϝτϦΫεૹ৴
https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Βૹ৴
Replace ͷޮՌ • ϝϯςφϯεঢ়ଶͷΓସ͕͑ߴʹ • Nginx ͰϑΝΠϧͷ༗ແΛݟͯ 503 Λฦ͠ ͍ͯͨͨΊɺͦͷͨΊ͚ͩʹ
deploy ͢Δ͔ શʹϑΝΠϧΛΔɾফ͢࡞ۀ͕ඞཁͩͬ ͨ
Replace ͷޮՌ • ΠϯελϯελΠϓΛมߋͨ͜͠ͱͰ CPU ͷ ॲཧੑೳ্͕ • ϨεϙϯελΠϜ͕(ଟগ)ߴʹ
՝ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖͑Δ •
Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ