Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
2年間運用しているソーシャルゲームのサーバ構成の変遷
Search
tkuchiki
January 27, 2017
Programming
3
4.6k
2年間運用しているソーシャルゲームのサーバ構成の変遷
tkuchiki
January 27, 2017
Tweet
Share
More Decks by tkuchiki
See All by tkuchiki
Kubernetesの上に作る、統一されたマイクロサービス運用体験
tkuchiki
1
8.1k
メルペイSREチームのオンコール/oncall-merpay-sre
tkuchiki
0
130
/sre-next-2020
tkuchiki
5
14k
/fastly-meetup-3
tkuchiki
0
83
/fastly-yamagoya-meetup-2019
tkuchiki
0
180
/cloud-spanner-operation-on-merpay
tkuchiki
1
1.2k
ソーシャルゲームにおける AWS 移行事例
tkuchiki
21
23k
Other Decks in Programming
See All in Programming
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
170
Blue/Greenデプロイの導入による 運用フローの改善
kudoas
1
370
try! Swift Tokyo 2024のLT枠に採択されたプロポーザルを出すときに考えていたこと
ski
0
350
Rails と人魚の話/rails-and-mermaid
sanfrecce_osaka
0
100
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
360
PHP8.3の機能を振り返る / Review of PHP 8.3 features
seike460
PRO
1
110
サイコロで理解する統計的仮説検定の考え方
tatamiya
4
900
Git Rebase
bkuhlmann
11
1.6k
Hanami and htmx
bkuhlmann
0
210
Compose-View Interop in Practice (mDevCamp 2024)
stewemetal
0
120
Behind VS Code Extensions for JavaScript / TypeScript Linnting and Formatting
unvalley
5
900
DMMプラットフォームがTiDB Cloudを採用した背景
pospome
8
4k
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
422
63k
Teambox: Starting and Learning
jrom
128
8.4k
WebSockets: Embracing the real-time Web
robhawkes
59
7k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
Building Your Own Lightsaber
phodgson
99
5.7k
We Have a Design System, Now What?
morganepeng
43
6.7k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
Code Reviewing Like a Champion
maltzj
514
39k
Making Projects Easy
brettharned
108
5.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
6
1.5k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
What's new in Ruby 2.0
geeforr
337
31k
Transcript
2ؒӡ༻͍ͯ͠Δ ιʔγϟϧήʔϜͷ αʔόߏͷมભ @tkuchiki
※ެ։༻ʹҰ෦εϥΠυΛ আ͓ͯ͠Γ·͢
WHOAMI(1) • @tkuchiki • ΠϯϑϥνʔϜ • ୲ • ιʔγϟϧήʔϜ •
ࣗࣾαʔϏε • Web+DB PRESS Vol.94
Agenda • αʔόߏͷมભ • ղܾฤ • Auto Scaling ฤ •
Replace ฤ
ղܾฤ
ϦϦʔε࣌ͷαʔόߏ
ϛυϧΣΞ • શڞ௨ • Perl 5.16.3 • consul + dnsmasq
• ෦ DNS • Service Discovery(DNS) • zabbix-agent
ϛυϧΣΞ • Openresty 1.4.6 • MySQL 5.6(RDS) • zabbix-proxy
ϛυϧΣΞ • Redis + Redis Sentinel • master ͷ IP
Λ Sentinel ʹ͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue
ϛυϧΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3,
numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάूଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd
1core ͔͑͠ͳ͍ • Nginx ͷΞΫηεϩάूͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάूͱͦΕҎ֎Λ͚Δ
σΟεΫ༰ྔ͕ᷓΕͦ͏ʹͳΔ • fluentd ͷෛՙݮޭ • ࠓσΟεΫ༰ྔ͕Γͳ͘ͳΔ • cron.daily ͰϩάͷѹॖΛͯ͠ɺҰఆظؒա ͗ͨΒআ͍ͯͨ͠
• σΟεΫΛՃͯ͠ cron.daily ͰϩάΛୀආ
·ͩσΟεΫ༰ྔ͕ᷓΕͦ͏ • 11ճͷϩάୀආͰؒʹ߹Θͳ͘ͳΔ • ຖ࣌ϩάΛѹॖͱୀආ
MySQL ߴෛՙ • MySQL 1 Ͱࡹ͖͖Εͳ͍ෛՙ • Slave(Read Replica) ΛՃͯ͠ࢀরࢄ
• HAProxy ಋೖ • ͍ͭͰʹཧը໘༻ͷ Slave ΛՃ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ
ࢼ߹ Batch ͷ࣮ߦ͕͍࣌ؒ • Ϣʔβ૿Ճʹ͍ࢼ߹૿Ճ • Batch ͷ࣮ߦྃ·Ͱͷ͕࣌ؒͲΜͲΜ৳ͼ ͍ͯ͘ •
εέʔϧΞοϓͰରԠ • ݱࡏεέʔϧΞτ͠ͳ͍ߏͳͷͰࠓ ޙͷ՝
fluent-plugin-redshift͕͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌ͳ͔͕ͬͨɺΞΫηε͕૿
͔͑ͯΒ࣌ʑ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • Ҏ্ଓ͖·ͨ͠
rin • https://github.com/fujiwara/rin • S3 ͷΠϕϯτ௨Ͱ SQS ʹΩϡʔΛՃ • SQS
ͷΩϡʔΛऔΓग़ͯ͠ redshift ʹ COPY ίϚϯυΛ࣮ߦ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)
ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for
Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)
҆ఆظʹಥೖ • ϦϦʔεޙ1͘Β͍Ͱେ͖ͳղফ • վળʹऔΓΊΔΑ͏ʹͳΔ
Auto Scaling ฤ
αʔόͷखಈՃ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰՃ • AWS CLI Ͱ EC2
Λىಈͯ͠ ELB ʹՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞ • Management Console ͕৮Εͳͯ͘࡞ ۀ͕Ͱ͖Δ
αʔόͷखಈՃ • ࡞ۀͰ͖Δ͕... • ෛՙ͕͍ͱαʔόϦιʔε͕༨Δ • ༧ظͤ͵ෛՙ૿େʹରԠͰ͖ͳ͍ • EC2 ͕ಥવࢮ͢ΔͱखಈͰ෮׆ͤ͞Δඞཁ͕
͋ͬͨ • ࠓ CloudWatch Alarm Ͱࣗಈ࠶ىಈͤ͞Δ ͜ͱ͕Ͱ͖·͢
Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ •
Πϕϯτ։࢝લʹ minimum capacity Λ૿ ͢ඞཁ͕͋ΔͨΊશʹͳ͘͢͜ͱͰ͖ ͳ͍
Auto Scaling ಋೖͷ • Ͳ͏ͬͯαʔόΛ࠷৽ͷঢ়ଶʹ͢Δʁ • ΞϓϦέʔγϣϯίʔυɾαʔόઃఆΛम ਖ਼͢Δʹ AMI Λ࡞ʁ
• αʔόىಈ࣌ʹ deploy ͢Δʁ • deploy αʔό͔ΒίʔυΛऔͬͯ͘Δʁ
Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ
औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹར༻͍ͯ͠Δ
# stretcher $ consul watch -type event -name deploy stretcher
# deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱՄೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ
Auto Scaling ͷΈ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest
ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚ΕΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛՃ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ আ & Consul Cluster ͔Β leave
Auto Scaling ಋೖͷޮՌ • ॳͷతୡ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ্ •
Pull ܕ deploy ʹมߋ͢Δ͜ͱͰ deploy ࣌ؒ ॖ
Replace ฤ
৽ήʔϜαʔόϦϦʔε • 20168݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱશ͘ผͷڥͱͯ͠ߏங • ιʔείʔυڞ༗ • DB ޓ͍ʹࢀর͋͠Θͳ͍
• ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘڥ Replace ͢Δ͜ͱʹ
طଘήʔϜαʔόͷ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓݹ͍
• ߏ͕ݹ͍ɾྑ͘ͳ͍ • ϩάूαʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ Norikra Ͱ... • etc
αʔόߏͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ࠷৽(c4,
m4)ʹ • ϩάूαʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora
αʔόߏͷ Replace • ϛυϧΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy
• td-agent ΛΊͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess ΛΘ ͳ͍Α͏ʹ
αʔόߏͷ Replace • Consul ͷ External Service Λ׆༻ • Consul
Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ
ELB Web+App Deploy x 1 Batch x 1 Admin x
1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY 0QFOSFTUZ qVFOUBHFOUIZESB qVFOUE )"1SPYZ qVFOUBHFOUIZESB HFBSNBOE Log Log( +Backup) Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1 qVFOUE SJO qVFOUE /PSJLSB Log Replace ޙ
Replace ͷޮՌ • ϩάूαʔόͷೖΕସ͕͑(΄΅)ແఀࢭͰߦ ͑Δ • Aurora มߋޙɺEBS ͷϨΠςϯγ૿ʹΑΔΫ ΤϦੑೳͷྼԽ͕ൃੜ͠ͳ͘ͳΔ
• Norikra + fluentd + fluent-plugin-slack ͰϦΞ ϧλΠϜͳΤϥʔ௨͕Մೳʹ
Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू αʔόͷෛՙݮ • Zabbix ͷϝτϦΫεૹ৴
https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Βૹ৴
Replace ͷޮՌ • ϝϯςφϯεঢ়ଶͷΓସ͕͑ߴʹ • Nginx ͰϑΝΠϧͷ༗ແΛݟͯ 503 Λฦ͠ ͍ͯͨͨΊɺͦͷͨΊ͚ͩʹ
deploy ͢Δ͔ શʹϑΝΠϧΛΔɾফ͢࡞ۀ͕ඞཁͩͬ ͨ
Replace ͷޮՌ • ΠϯελϯελΠϓΛมߋͨ͜͠ͱͰ CPU ͷ ॲཧੑೳ্͕ • ϨεϙϯελΠϜ͕(ଟগ)ߴʹ
՝ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖͑Δ •
Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ