2年間運用しているソーシャルゲームのサーバ構成の変遷

5292dd0924f5e440a4b7ff40b165a0b4?s=47 tkuchiki
January 27, 2017

 2年間運用しているソーシャルゲームのサーバ構成の変遷

5292dd0924f5e440a4b7ff40b165a0b4?s=128

tkuchiki

January 27, 2017
Tweet

Transcript

  1. 7.

    ϛυϧ΢ΣΞ • શ୆ڞ௨ • Perl 5.16.3 • consul + dnsmasq

    • ಺෦ DNS • Service Discovery(DNS) • zabbix-agent
  2. 9.

    ϛυϧ΢ΣΞ • Redis + Redis Sentinel • master ͷ IP

    Λ Sentinel ʹ໰͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue
  3. 10.

    ϛυϧ΢ΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3,

    numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift
  4. 11.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin  0QFOSFTUZ  qVFOUBHFOUMJUF  UEBHFOU  qVFOUBHFOUMJUF  HFBSNBOE Log  3FEJT  3FEJT 4FOUJOFM  UEBHFOU  3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle  0QFOSFTUZ
  5. 12.

    fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάू໿ଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd

    ͸ 1core ͔͠࢖͑ͳ͍ • Nginx ͷΞΫηεϩάू໿ͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάू໿ͱͦΕҎ֎Λ෼͚Δ
  6. 15.
  7. 16.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y  0QFOSFTUZ  qVFOUBHFOUMJUF  UEBHFOU  )"1SPYZ  qVFOUBHFOUMJUF  HFBSNBOE Log  3FEJT  3FEJT 4FOUJOFM  UEBHFOU  3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle  0QFOSFTUZ
  8. 18.

    fluent-plugin-redshift͕ࢗ͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌͸໰୊ͳ͔͕ͬͨɺΞΫηε͕૿

    ͔͑ͯΒ࣌ʑࢗ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕ࢗ͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • ൒೥Ҏ্ଓ͖·ͨ͠
  9. 19.
  10. 20.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y  0QFOSFTUZ  qVFOUBHFOUMJUF  UEBHFOU  )"1SPYZ  qVFOUBHFOUMJUF  HFBSNBOE Log  3FEJT  3FEJT 4FOUJOFM  UEBHFOU  3FEJT 4FOUJOFM  SJO Log Asset Bundle  0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)
  11. 21.

    ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for

    Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢
  12. 22.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y  0QFOSFTUZ  qVFOUBHFOUMJUF  UEBHFOU  )"1SPYZ  qVFOUBHFOUMJUF  HFBSNBOE Log  UEBHFOU  SJO Log Asset Bundle  0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)
  13. 25.

    αʔόͷखಈ௥Ճ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ৔߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰ௥Ճ • AWS CLI Ͱ EC2

    Λىಈͯ͠ ELB ʹ௥ՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞੒ • Management Console ͕৮Εͳͯ͘΋࡞ ۀ͕Ͱ͖Δ
  14. 27.

    Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ޲্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ •

    Πϕϯτ։࢝લʹ minimum capacity Λ૿΍ ͢ඞཁ͕͋ΔͨΊ׬શʹͳ͘͢͜ͱ͸Ͱ͖ ͳ͍
  15. 29.

    Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ

    औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ೚ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹ΋ར༻͍ͯ͠Δ
  16. 30.

    # stretcher $ consul watch -type event -name deploy stretcher

    # deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱ΋Մೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ
  17. 31.

    Auto Scaling ͷ࢓૊Έ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest

    ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λ࢖ͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷ஋ͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚Ε͹ΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛ௥Ճ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ࡟ আ & Consul Cluster ͔Β leave
  18. 34.

    ৽ήʔϜαʔόϦϦʔε • 2016೥8݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱ͸શ͘ผͷ؀ڥͱͯ͠ߏங • ιʔείʔυ͸ڞ༗ • DB ΋ޓ͍ʹࢀর͋͠Θͳ͍

    • ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ௕࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘ؀ڥ΋ Replace ͢Δ͜ͱʹ
  19. 35.

    طଘήʔϜαʔόͷ໰୊ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓ΋ݹ͍

    • ߏ੒͕ݹ͍ɾྑ͘ͳ͍ • ϩάू໿αʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ͸ Norikra Ͱ... • etc
  20. 36.

    αʔόߏ੒ͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ΋࠷৽(c4,

    m4)ʹ • ϩάू໿αʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora
  21. 37.

    αʔόߏ੒ͷ Replace • ϛυϧ΢ΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy

    • td-agent Λ΍Ίͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess Λ࢖Θ ͳ͍Α͏ʹ
  22. 38.

    αʔόߏ੒ͷ Replace • Consul ͷ External Service Λ׆༻ • Consul

    Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ
  23. 39.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y  0QFOSFTUZ  qVFOUBHFOUMJUF  UEBHFOU  )"1SPYZ  qVFOUBHFOUMJUF  HFBSNBOE Log  UEBHFOU  SJO Log Asset Bundle  0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ
  24. 40.

    ELB Web+App Deploy x 1 Batch x 1 Admin x

    1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY  0QFOSFTUZ  qVFOUBHFOUIZESB  qVFOUE  )"1SPYZ  qVFOUBHFOUIZESB  HFBSNBOE Log Log( +Backup) Log Asset Bundle  0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1  qVFOUE  SJO  qVFOUE  /PSJLSB Log Replace ޙ
  25. 42.

    Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू໿ αʔόͷෛՙ௿ݮ • Zabbix ΁ͷϝτϦΫεૹ৴͸

    https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Β௚઀ૹ৴
  26. 45.

    ՝୊ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖׵͑Δ •

    Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ