ソーシャルゲームにおける AWS 移行事例

5292dd0924f5e440a4b7ff40b165a0b4?s=47 tkuchiki
August 22, 2015

ソーシャルゲームにおける AWS 移行事例

トラックE (607-608) 2015-08-22 14:20:00

5292dd0924f5e440a4b7ff40b165a0b4?s=128

tkuchiki

August 22, 2015
Tweet

Transcript

  1. ιʔγϟϧήʔϜʹ͓͚Δ AWSҠߦࣄྫ YAPC::Asia TOKYO 2015 @tkuchiki

  2. WHOAMI(1) • @tkuchiki • KAYAC Inc. 4೥໨ • ΦϖϨʔγϣϯΤϯδχΞ •

    ओʹιʔγϟϧήʔϜ୲౰ • perl ͸ΞϓϦέʔγϣϯΤϯδχΞͩͬͨ ࠒʹ 3ϲ݄΄Ͳ...
  3. Agenda • AWS Ҡߦͷഎܠ • ࣄྫ 1 • ࣄྫ 2,

    3 • ؂ࢹʹ͍ͭͯ • Jenkins Slave ͷ׆༻ • AWS ҠߦͷϝϦοτɾσϝϦοτ • ஌ݟ
  4. AWS Ҡߦͷഎܠ • 2011 ೥͔Β DC Ͱͷӡ༻։࢝ • 5 ೥ϦʔεͰαʔόػߪೖ

    • Ϧʔεظؒऴྃޙͷ͜ͱΛߟ͑Δඞཁੑ • 2014 ೥Ҏ߱ʹϦϦʔεͨ͠αʔϏε͸ɺࠓޙ ͷ͜ͱΛݟਾ͑ͯ AWS ʹߏங
  5. AWS Ҡߦͷഎܠ • ࣾ಺࠷େن໛ͷαʔϏεͷ AWS Ҡߦͱ਺ݸͷ αʔϏεऴྃ • ࢖༻͠ͳ͍αʔό͕૿Ճ •

    2014 ೥ 3 ݄ʹϥοΫ࡟ݮ • ࠓޙ΋༨৒αʔό͕૿Ճ͢ΔՄೳੑΛߟ͑ͯ ࠷దԽ͠΍͍͢ AWS ʹશ໘Ҡߦ͢Δ͜ͱʹ
  6. ࣄྫ 1

  7. • ๯ݥΫΠζΩϯάμϜ • Ҡߦظؒ 1 ϲ݄ • Ҡߦ࣌ʹ perl ͷϚΠφʔόʔδϣϯΞοϓ

    • cpan module ͷόʔδϣϯ΋ • CentOS 6ΛϚΠφʔόʔδϣϯΞοϓ ࣄྫ 1
  8. • Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ 2 ໊ • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊

    • Φϒβʔόʔ 2 ໊ ࣄྫ 1
  9. Ҡߦ࡞ۀ • dev αʔόߏங • RDS for MySQL ͷݕূ •

    ElastiCache for Redis ͷݕূ • Cpanfile ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • Mozilla::CA ͷ໰୊ • ຊ൪αʔόߏங • ϝϯςφϯε
  10. dev αʔόߏங • dev αʔό͕ 2 ୆͋ͬͨ • ৑௕ߏ੒Ͱ͸ͳ͍ •

    ؅ཧ͕େมʹͳΔ͚ͩͳͷͰ1୆ʹ·ͱΊΔ • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook Խ
  11. RDS for MySQL ͷݕূ • RDS ͸λΠϜκʔϯͷઃఆ͕Ͱ͖ͳ͍ͷͰ઀ ଓ࣌ʹઃఆมߋ͢Δ • MySQL

    ͷ৔߹ɺinit_connect • SET SESSION time_zone = CASE WHEN POSITION('rds' IN CURRENT_USER()) = 1 THEN 'UTC' ELSE '+9:00' END;
  12. RDS for MySQL ͷݕূ • Failover ͯ͠΋໰୊ͳ͍͔ • init_connect ͰΤϥʔ͕ͰΔ৔߹͕͋Δͱ

    ͍͏৘ใ͕... • reboot with failover ػೳΛ࢖༻ • MySQL ͷ reconnect Λ༗ޮʹ͠ͳ͍
  13. ΦϯϓϨ MySQL -> RDS • ΦϯϓϨ - RDS Ͱ Replication

    ͕૊ΊΔ • RDS ͸λΠϜκʔϯ͕มߋͰ͖ͳ͍ • UTC ͷΈ • JST Ͱӡ༻͍ͯ͠Δ৔߹ 9 ࣌ؒͣΕΔ • ࣌ࠁؔ਺Λ࢖͍ͬͯͳ͍৔߹͸໰୊ͳ͍
  14. MySQL ͷσʔλαΠζ • MySQL 5.5 • mysqldump(raw): 14.92GB • mysqldump(zgip):

    2.79GB
  15. $ time zcat dump.sql.gz | mysql -u ... real 209m18.642s

    user 6m42.994s sys 0m16.495s MySQL ͷΠϯϙʔτ
  16. 209෼…

  17. Ͳ͏ʹ͔ͯ͠ ߴ଎Խ͍ͨ͠

  18. None
  19. खॱ͕·ͱΊ ΒΕ͍ͯͨ

  20. MySQL5.5 ࠷଎σʔλҠߦखॱ • ςʔϒϧ࡞੒ • ϓϥΠϚϦΠϯσοΫε࡞੒ • ֎෦Ωʔ੍໿෇༩ • ֎෦Ωʔ੍໿ແޮԽ

    • σʔλϩʔυ • ֎෦Ωʔ੍໿༗ޮԽ • ηΧϯμϦΠϯσοΫε࡞੒ http://dbstudy.info/files/20131007/mysql56_load_r2.pdf
  21. mysqldump • ηΧϯμϦΠϯσοΫεൈ͖ͰdumpͰ͖ͳ͍ • --disable-keys(-K) Λ࢖͏ͱߦ͕͢΂ͯૠೖ͞ Εͨ͋ͱʹΠϯσοΫε࡞੒ • INSERT จ͕

    ALTER TABLE (DISABLE| ENABLE) KEYS ͰғΘΕΔ • MyISAM ͷΈ...
  22. mysqldumpindex • https://github.com/tkuchiki/mysqldumpindex • SHOW TABLES ͱ SHOW INDEXES FROM

    ͷ݁Ռ ͔ΒɺηΧϯμϦΠϯσοΫεͷ࡞੒ɺ࡟আ SQL Λ࡞੒͢Δ • mysqldump --no-data Ͱ dump ͨ͠ SQL ͱɺη ΧϯμϦΠϯσοΫε࡟আɾ࡞੒ޙʹ dump ͨ͠ SQL ͷ diff ΛͱͬͯҰக͢Δ͜ͱΛ֬ೝ
  23. # σʔλΠϯϙʔτ $ time { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo

    "FOREIGN_KEY_CHECKS=1"; } | mysql -u ... real 54m52.894s user 6m43.438s sys 0m15.711s # ηΧϯμϦΠϯσοΫε࡞੒ $ time mysql -u ... < create_index.sql real 38m26.767s user 0m0.010s sys 0m0.018s ࠷଎खॱͰͷσʔλҠߦ࣌ؒ 209 ෼͔Β 93෼ʹ୹ॖʂ
  24. mysqldump(percona-server) • --innodb-optimize-keys • CREATE TABLE จ͔Β UNIQUE ͱ INDEX

    ΛऔΓআ͘ • INSERT จͷޙΖʹ ALTER TABLE ADD [UNIQUE] KEY Λ௥Ճ
  25. mysqldump(percona-server) • --innodb-optimize-keys Λ࢖͏͚ͩͰ࠷଎ͷ खॱʹͳΓͦ͏ • ͱࢥ࣮ͬͯݧͯ͠Έ·͕ͨ͠ૣ͘ͳΒͣ

  26. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • dsn ʹ timeout Λઃఆ •

    mysql_connect_timeout • mysql_read_timeout • mysql_write_timeout • ઃఆ͠ͳ͍ͱϓϩηε͕࠶ੜ੒͞Εͳ͍ͱ࠶઀ଓ ͠ͳ͍ • DNS ΛҾ͖௚͞ͳ͍
  27. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆͯ͠΋ͩΊͳ৔߹͕͋Δ • ωοτϫʔΫஅ •

    MySQL server has gone away • Ұ౓ࣦഊ͢Δͱϓϩηε͕࠶ੜ੒͞ΕΔ·Ͱܨ ͕Βͳ͍... • ping Ͱܨ͕Δ͔֬ೝ͢Δ͔ɺDBIx::Connector Λ࢖͏
  28. if (!$dbh->ping) { $dbh->disconnect; $dbh = DBI->connect($dsn,...); } RDS Ͱਖ਼ৗʹ

    Failover ͤ͞Δ
  29. ElastiCache for Redis ͷݕূ • RDS ͱҧͬͯɺreboot with failover ػೳ͕࢖

    ͑ͳ͍ • 30 ~ 60sec ͔͔ΔॲཧΛͤ͞Δͱ Failover ͢Δ͜ͱ͕͋Δ • iptables ͰύέοτΛམͱ͢ • ࠓͷͱ͜Ζਖ਼֬ʹςετ͢Δ͜ͱ͕Ͱ͖ͳ͍
  30. ElastiCache Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆ • timeout ͯ͠΋

    DNS ͸Ҿ͖௚͞ͳ͍ • ping Ͱ֬ೝͯ͠ɺͩΊͳΒ࠶઀ଓ
  31. if (!$redis->ping) { $redis->quit; $redis = Redis::Fast->new(...); } ElastiCache Ͱਖ਼ৗʹ

    Failover ͤ͞Δ
  32. Cpanfile ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • ؀ڥߏஙΛ༰қʹߦ͑ΔΑ͏ʹ • ࠷৽൛ͷ

    cpan module ΛೖΕͯςετ • ໰୊ͳ͔ͬͨͷͰ࠷৽൛ʹ • DateTime ͚ͩ͏Δ͏ඵૠೖ·͑ʹόʔδϣ ϯμ΢ϯ • :60 Λࠩ͠ࠐΜͰ͘Δ...
  33. Mozilla::CA ͷ໰୊ • Mozilla::CA::SSL_ca_file() ͕ฦ͢ূ໌ॻ͕ݹ ͍ • ΞϓϦ಺՝ۚͷϨγʔτݕূͰΤϥʔ͕ग़ ΔΑ͏ʹ... •

    ෇ଐͷ mk-ca-bundle.pl Λ࢖༻
  34. $ ./perl5/lib/perl5/Mozilla/mk-ca-bundle.pl --help ... -p list of Mozilla trust purposes

    and levels for certificates to include in output. Takes the form of a comma separated list of purposes, a colon, and a comma separated list of levels. (default: SERVER_AUTH:TRUSTED_DELEGATOR) Valid purposes are: ALL, DIGITAL_SIGNATURE, NON_REPUDIATION, KEY_ENCIPHERMENT, DATA_ENCIPHERMENT, KEY_AGREEMENT, KEY_CERT_SIGN, CRL_SIGN, SERVER_AUTH, CLIENT_AUTH, CODE_SIGNING, EMAIL_PROTECTION, IPSEC_END_SYSTEM, IPSEC_TUNNEL, IPSEC_USER, TIME_STAMPING, STEP_UP_APPROVED Valid levels are: ALL, TRUSTED_DELEGATOR, NOT_TRUSTED, MUST_VERIFY_TRUST, TRUSTED Mozilla::CA ͷ໰୊
  35. Mozilla::CA ͷ໰୊ • -p PURPOSES:LEVELS Ͱ໨తͱϨϕϧ͕ࢦఆ Ͱ͖Δ • -p ALL:ALL

    ͰશࢦఆՄ • ੜ੒ͨ͠ূ໌ॻΛࢦఆ͢ΔΑ͏ʹมߋ
  36. my %args = ( ssl_opts => { SSL_ca_file => '/path/to/ca-bundle.crt',

    }, ... ); my $furl = Furl->new(%args); ... Mozilla::CA ͷ໰୊
  37. ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λେ෯ʹมߋ • web

    αʔόͱ app αʔόͷ౷߹ • ࣗલͰ MySQL ͱ Redis Λཱͯͳ͍ • RDS ͱ ElastiCache
  38. -# /HJOY "QQ 3FEJT NBTUFS 3FEJT TMBWF #BUDI .Z42-NBTUFS .Z42-TMBWF

    %FQMPZ STZOD "SDIFS 7*1 DIFGTPMP DBQJTUSBOP
  39. ຊ൪αʔόߏங • consul ͷಋೖ • ಺෦ DNS • stretcher ͷಋೖ

    • Fluentd ͷઃఆݟ௚͠
  40. Consul • Hashicorp ੡ͷΦʔέετϨʔγϣϯπʔϧ • KVSɺDNSɺHTTP ΠϯλϑΣʔε • ಺෦ DNS

    ͱͯ͠׆༻
  41. Consul DNS • 8600 port Ͱఏڙ • node ͱ service

    ʹରԠ • nodename.node(.datacenter).consul • servicename.service(.datacenter).consul
  42. dnsmasq • DNS forwarder, cache • consul ͷ 8600 port

    ʹ forward • શαʔόͰՔಇ
  43. $ cat /etc/dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1 dnsmasq

  44. VPC DHCP Options Set • VPC ͷ DHCP ͷઃఆΛߦ͏ •

    resolv.conf ͕ੜ੒͞ΕΔ • ݱࡏ͸ Management Console ͔Β΋ઃఆՄೳ
  45. $ cat /etc/resolv.conf ; generated by /sbin/dhclient-script search node.consul service.consul

    nameserver 127.0.0.1 nameserver 10.1.0.2 ; VPC ͷ DNS nameserver 10.1.11.100 ; Unbound VPC DHCP Options Set
  46. Unbound • DNS cache, resolver • VPC ͷ DNS ΛΩϟογϡ

    • VPC ͷ resolver ʹ໰୊͕͋ͬͯ΋໊લղܾ ͕Ͱ͖ΔΑ͏ʹ • deploy αʔόͰՔಇ
  47. stretcher ʹΑΔ deploy • https://github.com/fujiwara/stretcher • Go ੡ • S3

    ͔Β tarball Λऔಘͯ͠ల։ɺrsync • consulɺserf ͱ࿈ܞ • ৄ͘͠͸ɺhttps://speakerdeck.com/fujiwara3/ consultozi-zuo-osswohuo-yong-sita100tai-gui-mo- falsewebsabisuyun-yong
  48. &-# /HJOY "QQ &MBTUJ$BDIF NBTUFS &MBTUJ$BDIF TMBWF #BUDI 3%4NBTUFS 3%4TMBWF

    CBDLVQ %FQMPZ DPOTVM FWFOU #VJME 6QMPBE 4 DIFGTPMP DBQJTUSBOP
  49. ϝϯςφϯε (ࣄલ४උ) • ηΧϯμϦΠϯσοΫεΛ࡟আ͓͍ͯͨ͠ςʔ ϒϧΛҠߦઌͷ DB ʹ༻ҙ • RDS Λ

    Single-AZ ʹ͢Δ • ύϥϝʔλάϧʔϓͷௐ੔
  50. ϝϯςφϯε (ࣄલ४උ) • RDS ΛSingle-AZ ʹ͢Δ • Multi-AZ ͷ৔߹ɺ଱ো֐ੑ͸޲্͢Δ͕ॻ ͖ࠐΈ଎౓͕2ഒʹ

  51. ϝϯςφϯε (ࣄલ४උ) • ύϥϝʔλάϧʔϓͷมߋ • innodb_flush_log_at_trx_commit=0 • innodb_support_xa=0 • sync_binlog=0

  52. ϝϯςφϯε • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Redis ͷσʔλҠߦ •

    ಈ࡞֬ೝ • DNS੾Γସ͑ • Multi-AZ Խ & ύϥϝʔλάϧʔϓมߋ
  53. MySQL ͷσʔλҠߦ • εΩʔϚ͸༧Ί৽ DB ʹೖΕͯ͋ΔͷͰɺσʔ λ͚ͩ dump (& gzip)

    • FOREIGN_KEY_CHECKS=0 ͱ 1 Λ zcat ͷલ ޙʹೖΕͯ mysql ʹ౉͢
  54. $ { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo "FOREIGN_KEY_CHECKS=1"; } |

    mysql -u ... MySQL ͷσʔλҠߦ
  55. MySQL ͷσʔλҠߦ • import ͕ऴΘͬͨΒηΧϯμϦΠϯσοΫε Λ࡞੒͢Δ • information_schema.tables Λݟ֤ͯςʔϒϧ ͷߦ਺Λൺֱ

  56. mysql -u USER -B -N -e "SELECT TABLE_NAME, TABLE_ROWS FROM

    information_schema.tables WHERE TABLE_SCHEMA = 'my_db' ORDER BY TABLE_NAME" MySQL ͷσʔλҠߦ
  57. Redis ͷσʔλҠߦ • redis-cli CONFIG SET appendonly yes • appendfsync

    no ʹ͢Δͱߴ଎(fsync͠ͳ͍) • redis-cli BGSAVE • Redis Λఀࢭ • cat appendonly.aof | redis-cli --pipe -h ...
  58. Redis ͷσʔλҠߦ • Ҡߦલޙͷ྆؀ڥͰ KEYS “*” ͷ݁ՌΛϑΝΠ ϧʹམͱ͢ • ݁ՌΛ

    sort • diff Λऔͬͯ֬ೝ • expireͰ࡟আ͞Ε͍ͯΔͱଟগࠩ෼͕ग़Δ
  59. ಈ࡞֬ೝ • Ҡߦޙͷ؀ڥʹ઀ଓͰ͖ΔΞϓϦΛ୺຤ʹೖ Ε͓ͯ͘ • υϝΠϯΛઃఆ • εϚʔτϑΥϯͷ DNS ઃఆΛมߋͨ͠Γɺ

    DNS ͷԠ౴Λِ૷͢ΔΑΓָ
  60. Single-AZ -> Multi-AZ ύϥϝʔλάϧʔϓͷมߋ • Multi-AZ ʹมߋ • Single-AZ ͔Β

    Multi-AZ ΁ͷมߋ͸ΦϯϥΠϯͰ ࣮ߦՄೳ • ύϥϝʔλάϧʔϓͷมߋ • innodb_flush_log_at_trx_commit=1 • innodb_support_xa=1 • sync_binlog=1
  61. DNS ͷ੾Γସ͑ • TTL Λ༧Ί୹͓ͯ͘͘͠ • TTL Λ୹ͯ͘͠΋چ؀ڥʹΞΫηε͕དྷΔՄ ೳੑ •

    DB ΍ KVS ͸ඞͣࢭΊ͓ͯ͘
  62. ࣄྫ 2,3

  63. ࣄྫ 2,3 • ๻ΒͷߕࢠԂʂ೤ಆฤͷ Mobage ൛ͱ iOS / Android ൛

    • Ҡߦظؒ߹Θͤͯ 1 ϲ݄ • ΄ͱΜͲαʔόߏ੒͕ಉ͡ͳͷͰಉ࣌ਐߦ • perlbrew ͰΠϯετʔϧͨ͠ perl ؀ڥΛͦͷ·· ͍࣋ͬͯ͘ • SL6 -> CentOS6 ͳͷͰ
  64. ࣄྫ 2,3 • Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ ֤ 1 ໊ •

    ΦϖϨʔγϣϯΤϯδχΞ 1 ໊ • Φϒβʔόʔ 2 ໊
  65. Ҡߦ࡞ۀ • dev αʔόߏங • (RDS for MySQL ͷݕূ) •

    yrmcds ͷݕূ • ຊ൪αʔόߏங • ϝϯςφϯε • Ҡߦޙʹൃੜͨ͠໰୊
  66. dev αʔόߏங • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook

    Խ
  67. yrmcds ͷݕূ • session ʹ Kyoto Tycoon Λ࢖༻ • ػೳతʹ໰୊͸ͳ͍

    • ٕज़ݕূ΋݉Ͷͯผͷϛυϧ΢ΣΞΛ࢖͏ • Redis ͩͱΞϓϦͷେ෯ͳमਖ਼͕ඞཁ • Memcached ޓ׵ͷ΋ͷ͕ྑ͍ • yrmcds
  68. yrmcds • https://github.com/cybozu/yrmcds • Memcached ΄΅׬શޓ׵ • master / slave

    ϨϓϦέʔγϣϯ • VIP ͕෇͍͍ͯΔαʔό͕ master • ϝϞϦ؅ཧʹ Slab Λ࢖༻͍ͯ͠ͳ͍ • αʔόαΠυϩοΫ
  69. EC2 ʹ VIP Λ͚ͭΔ • EC2 ʹ VIP Λ͚ͭͯ΋௨৴Ͱ͖ͳ͍ •

    ENI ͸σϑΥϧτͰૹ৴ݩ/ૹ৴ઌνΣοΫ ͕༗ޮʹͳ͍ͬͯΔ • ແޮʹ͠ͳ͍ͱࣗ෼ѼͯҎ֎ͷύέοτ Λड͚औΕͳ͍
  70. Keepalived • VIP ͷ੾Γସ͑ʹ࢖༻ • VPC Ͱ͸ϚϧνΩϟετ͕࢖͑ͳ͍ • ؤுΕ͹࢖͑ΔΑ͏Ͱ͕͢ɺKeepalived ͷՄ

    ༻ੑ΋ߟ͑ͳͯ͘͸ͳΒͳ͍ • Consul lock Λ׆༻ͨ͠ Failover γεςϜ
  71. Consul lock • ෼ࢄϩοΫػೳ • consul lock service/lock deploy.sh •

    ෼ࢄηϚϑΥػೳ • consul lock -n 2 service/lock deploy.sh
  72. Consul lock Λ׆༻ͨ͠ Failover γεςϜ • lock Λऔͬͨϗετʹ VIP Λ͚ͭΔ

    • ϩʔΧϧͷ yrmcds ͷࢮ׆؂ࢹ • μ΢ϯͨ͠Β lock Λղ์͢Δ • Slave Ͱ΋ lock ΛऔΓʹߦͬͯ lock ղ์଴ͪͷঢ়ଶʹ ͢Δ • yrmcds ͕μ΢ϯ͢Δͱ Slave ͷ͏ͪҰ͕ͭ lock Λऔͬ ͯ VIP Λ͚ͭΔ(master ʹঢ֨)
  73. Consul lock Λ׆༻ͨ͠ Failover γεςϜ • VPC Route Table ͷॻ͖׵͑

    • yrmcds Λ svc -o Ͱىಈ • উखʹ࠶ىಈ͠ͳ͍Α͏ʹ • /service/yrmcds ΋ىಈ࣌ʹ࡟আ • αʔό͕ಥવࢮͯ͠ VIP ͕͍ͭͨ··࠶ىಈ ͯ͠΋ master ʹͳΒͳ͍Α͏ʹ
  74. ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λมߋ • web

    αʔόͱ app αʔόͷ౷߹ • Kyoto Tycoon Λ yrmcds ʹҠߦ
  75. -# /HJOY "QQ ,ZPUP5ZDPPO TFTTJPO #BUDI .Z42-NBTUFS .Z42-TMBWF %FQMPZ STZOD

    "SDIFS DIFGTPMP DBQJTUSBOP ,ZPUP5ZDPPO TFTTJPO CBDLVQ )"1SPYZ .FNDBDIFE DBDIF )"1SPYZ )"1SPYZ
  76. ຊ൪αʔόߏங • MySQL Λ Slave ࢀর͠ͳͯ͘΋໰୊ͳ͘ͳͬ ͨͷͰ HAProxy ͕ෆཁʹ •

    Consul + dnsmasq ಋೖ • ಺෦DNS
  77. &-# ZSNDET NBTUFS ZSNDET TMBWF #BUDI %FQMPZ STZOD "SDIFS 7*1

    DIFGTPMP DBQJTUSBOP 3%4NBTUFS 3%4TMBWF CBDLVQ /HJOY "QQ
  78. Ҡߦ࡞ۀ • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Session ͷσʔλҠߦ •

    (ಈ࡞֬ೝ) • (DNS੾Γସ͑)
  79. MySQL ͷσʔλҠߦ • MySQL ͷσʔλҠߦΛ࠷଎खॱͰ࣮ࢪ • Duplicate entry ͕ग़ࣦͯഊ... •

    લ೔ͷԋशͰ͸੒ޭ • ະͩʹݪҼෆ໌ • ֎෦Ωʔ੍໿Λ΄ͱΜͲ࢖͍ͬͯͳ͍ͨΊ௨ ৗͷ mysqldump Ͱ΋ 5 ෼͔͕ࠩ͠ͳ͔ͬͨ
  80. Session ͷσʔλҠߦ • ͠·ͤΜͰͨ͠ • ϩάΠϯ͠௚͠ • Ϣʔβ਺తʹ໰୊ͳ͠ • ϩάΠϯ͕ूத͢Δͱࡹ͖੾Εͳ͍৔߹΋

    ͋ΔͷͰ௨ৗ͸Ҡߦͨ͠΄͏͕ྑ͍
  81. Ҡߦޙʹൃੜͨ͠໰୊ • Consul ͸਺ඵ Leader lost ͢Δ͜ͱ͕͋Δ • Leader lost

    ͢Δͱ lock ͕֎ΕΔ... • Ҡߦޙ 2 ೔࿈ଓͰ Failover • dns_config.node_ttl, allow_stale, max_stale Λઃఆ͢Δ • Leader lost ࣌΋਺ඵͳΒ DNS ͕Ҿ͚Δ
  82. ؂ࢹʹ ͍ͭͯ

  83. Zabbix • ࣗࣾαʔϏεͷ؂ࢹ • VPC Peering Connection Ͱ internal ௨৴

    • Ұ෦ Zabbix Proxy Λར༻
  84. Zabbix Proxy • Zabbix Server ͷ୅ΘΓͱͯ͠ಈ࡞ • ෛՙͷܰݮ • ऩूσʔλΛ֤ϗετͰόοϑΝϦϯά

    • Firewall ͷઃఆ͕༰қ • ֎෦νΣοΫ • ίϚϯυΛ࣮ߦͯ͠ MySQL ͷ QPS Λܭଌ
  85. Zabbix Proxy • ศར͕ͩ SPOF ʹͳΔ • ΠϯλʔωοτΛܦ༝ͤͣ؂ࢹ͕Մೳʹͳͬ ͨͷͰ௚઀؂ࢹʹมߋ •

    Active Agent Check • ผͷ໰୊͕...
  86. Zabbix Active Check • ֎෦νΣοΫͷεΫϦϓτʹύεϫʔυΛॻ ͍͍ͯΔ • Zabbix Proxy ͕ೖ͍ͬͯΔϗετʹઃఆ

    • ௚઀؂ࢹʹͳΓ࢖͑ͳ͘ͳΔ • ؂ࢹର৅͕૿͑Δͨͼʹ Zabbix Server ͷઃ ఆมߋΛߦ͏ͷ͸ආ͚͍ͨ
  87. Zabbix Active Check • UserParameter Λઃఆͨ͠ϗετΛొ࿥ • RDS ΍ ElastiCache

    ͳͲͷ؂ࢹ͕Մೳʹ • ςϯϓϨʔτʹొ࿥ͨ͠ΞΠςϜΛϗετϚ ΫϩͰมߋՄೳʹ • ࠶ར༻͠΍͍͢
  88. None
  89. None
  90. PUSH ௨஌༻ SSL ূ໌ॻͷ؂ࢹ • PUSH ௨஌ʹ࢖͏ SSL ূ໌ॻͷظݶΛ؂ࢹ •

    ূ໌ॻͷ PATH ͸αʔό͝ͱʹҧ͏ • ϓϩδΣΫτ͝ͱʹΞΠςϜΛ௥Ճ͢Δͷ͸ େม • Low Level Discovery(LLD) Ͱղܾ
  91. Low Level Discovery(LLD) • ΞΠςϜɺτϦΨɺάϥϑΛࣗಈͰੜ੒͢Δ ͨΊͷػೳ • ϑΝΠϧγεςϜɾωοτϫʔΫΠϯλ ϑΣʔεݕग़༻ͷLLD͕૊Έࠐ·Εͨςϯϓ Ϩʔτ΋͋Δ

    • LLD Λࣗ࡞͢Δ͜ͱ΋Մೳ
  92. Low Level Discovery(LLD) • UserParameter ʹ ΞΠςϜऩू༻ͷઃఆ௥Ճ • ΞΠςϜऩू͸ɺܾΊΒΕͨܗࣜͷ JSON

    σʔλΛฦ͢
  93. #!/bin/bash CERTS="${1}" CERTS_DATA="" for CERT in ${CERTS}; do CERTS_DATA="${CERTS_DATA},{\"{#CERT_PATH}\":\"${CERT}\"}" done

    echo "{\"data\":[$(echo ${CERTS_DATA} | sed -e 's/^,//')]}" Low Level Discovery(LLD)
  94. { "data": [ {"{#CERT_PATH}":"/path/to/cert1"}, {"{#CERT_PATH}":"/path/to/cert2"} ] } Low Level Discovery(LLD)

  95. #!/bin/bash CERT="${1}" NOW=$(date +%s) EXPIRE=$(date -d "$(openssl x509 -in ${CERT}

    -noout -dates | tail -n 1 | sed -e 's/notAfter=//')" +%s) echo $(((${EXPIRE} - ${NOW}) / 60 / 60 / 24)) Low Level Discovery(LLD)
  96. None
  97. None
  98. εϖʔε۠੾ΓͰূ໌ॻΛࢦఆ

  99. RDS ͷ؂ࢹ • fluent-plugin-cloudwatch ͰϝτϦΫεΛऩू • fluent-plugin-zabbix Ͱ Zabbix ʹૹ৴

    • MySQL ͷ stats ͸ SQL Λ࣮ߦ • SHOW ENGINE INNODB STATUS • SHOW GLOBAL STATUS
  100. ElastiCache ͷ؂ࢹ • redis-cli INFO ͷ stats Λऩू • DNS

    ΛҾ͍ͯ Primary Cache Cluster ͷ IP Λ؂ࢹ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover • UserParameter ͰίϚϯυ࣮ߦ • net.dns.record ͸ DNS ͕Ҿ͚ͳ͍࣌͸Ξϥʔ τΛඈ͹ͤͳ͍...
  101. #!/bin/bash host="/usr/bin/host" DOMAIN=${1} TIMEOUT=10 RECORDS=$($host -s -W $TIMEOUT $DOMAIN |

    awk 'NR > 1 {print $NF}' | sort) [ "${RECORDS}" = "" ] && echo "fail" && exit 1 echo $RECORDS ElastiCache ͷ؂ࢹ
  102. yrmcds ͷ؂ࢹ • stats ͰϝτϦΫεΛऔಘՄೳ • https://github.com/cybozu/yrmcds/issues/47 ͰରԠ͍͖ͯͨͩ͠·ͨ͠ • ͋Γ͕ͱ͏͍͟͝·͢ʂ

    • VIP ͕෇͍͍ͯΔΠϯελϯεͷ Private IP Λ νΣοΫ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover ͱͯ͠ݕग़
  103. Jenkins Slave ͷ׆༻

  104. Jenkins EC2 Plugin • Jenkins Slave Λ AMI ͔Βىಈ •

    ςετ͕࣮ߦ͞Εͳ͘ͳ͔ͬͯΒ 30෼͘Β ͍Ͱ Terminate • ىಈ͕࣌ؒ୹͘ͳΔͷͰ҆ՁʹͳΔ • Spot Instance ΋࢖͑ΔͷͰɺΑΓ҆Ձʹ
  105. Jenkins EC2 Plugin • 32 core ͷ CPU ͰίΞ਺෼·Ͱฒྻ਺Λ͋͛ ͯςετΛճ͢ͱ͚͜ΔΑ͏ʹ...

    • MySQL ͕ EAGAIN ΤϥʔΛग़͢Α͏ʹ • Test::mysqld Ͱෳ਺ݸͷ MySQL Λ্ཱͪ ͍͛ͯΔ • fs.aio-max-nr=65536 Λ 4 ഒʹͯ͠ճආ
  106. Jenkins EC2 Plugin • ଞͷΧʔωϧύϥϝʔλ΋νϡʔχϯά • ςετ͸ߴ଎ԽͰ͖ͣ... • MySQL ͷ্ཱͪ͛ʹ͕͔͔࣌ؒΔ

    • Test::mysqld::Multi • ฒྻ਺ΛNഒͯ͠΋࣮ߦ࣌ؒ͸1/NʹͳΒͳ͍ • CPUΛ࢖͍੾Εͳ͍
  107. AWS Ҡߦͷ ϝϦοτɾ σϝϦοτ

  108. AWS Ҡߦͷֹۚ໘Ͱͷ ϝϦοτ • ϓϩδΣΫτ͝ͱʹਖ਼֬ͳར༻ֹ͕ग़ͤΔ • ΦϯϓϨͰ͸ɺαʔό(VM)͕༨ͬͨ৔߹શ ࣾඅ༻ʹճ͍ͯͨ͠ • ݟ্͔͚͸҆ՁʹͳΔ͕ɺશࣾతʹ͸අ༻

    ͕͔͔͍ͬͯΔ͜ͱΛཧղ͞Εͳ͍
  109. AWS Ҡߦͷֹۚ໘Ͱͷ σϝϦοτ • අ༻͸ଟগ૿Ճ͢Δ • େن໛Ͱͳ͚Ε͹ਓ݅අΛؚΊΔͱ҆Ձ(ͳ ͸ͣ) • ࣗલͰαʔόӡ༻͢ΔΑΓָ

    • దਖ਼ͳඅ༻͕ܭ্͞ΕΔͨΊɺඅ༻͕૿Ճ͠ ͨΑ͏ʹࢥΘΕͯ͠·͏
  110. AWS Ҡߦͷػೳ໘Ͱͷ ϝϦοτ • ϚωʔδυαʔϏεͰӡ༻ෛՙͷ௿ݮ • ҆Ձʹαʔόͷަ׵͕Մೳ • ϋʔυ΢ΣΞੑೳͷ޲্ •

    ΞϓϦͷνϡʔχϯάͳ͠ͰϨεϙϯελ ΠϜվળ
  111. None
  112. AWS Ҡߦͷػೳ໘Ͱͷ σϝϦοτ • ඞཁͳػೳ͕ͳ͍͜ͱ͕͋Δ • ElastiCache ͷ reboot with

    failover • RDS ͷ λΠϜκʔϯରԠ
  113. ஌ݟ

  114. ஌ݟ • ҆ఆӡ༻Ͱ͖͍ͯΔͱݱঢ়ҡ࣋ʹͳΓ͕ͪ • αʔόߏ੒ͷݟ௚͕͠Ͱ͖Δνϟϯε • ྑ͘ͳ͔ͬͨ෦෼͕ු͖ூΓʹͳΔ • ؂ࢹ͢΂͖͜ͱ͕Ͱ͖͍ͯͳ͍ •

    Chef Ͱ؅ཧͰ͖͍ͯΔ΋ͷͱͰ͖͍ͯͳ͍ ΋ͷ͕ࠞࡏ͍ͯ͠Δ
  115. ஌ݟ • े਺୆ن໛ͷγεςϜͰ͋Ε͹ 2~3 ໊ఔ౓Ͱ 1 ϲ݄͋Ε͹ҠߦͰ͖Δ • ҙ֎ͱԿͱ͔ͳΔ •

    ϝϯςφϯε೔ͷௐ੔͕ҙ֎ͱେม
  116. ஌ݟ • ࣄલԋशΛԿ౓΋΍Δͱ҆৺ • MySQL ͷσʔλҠߦ͸ݕূ࣌ʹ3ɾ4ճɺલ ೔ʹ 2 ճ࣮ࢪ •

    ίϚϯυΛίϐϖ࣮ͯ͠ߦ͢Δ͚ͩͰྑ͍ ঢ়ଶʹ͓ͯ͘͠ͱߋʹ҆৺
  117. ·ͱΊ • AWS ͷҠߦࣄྫ঺հ • ݱঢ়ҡ࣋ΑΓ΋վળ • ߏ੒Λݟ௚͢ྑ͍ػձ • ػձ͕ͳ͍ͱվળ͠ʹ͍͘