Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ソーシャルゲームにおける AWS 移行事例

tkuchiki
August 22, 2015

ソーシャルゲームにおける AWS 移行事例

トラックE (607-608) 2015-08-22 14:20:00

tkuchiki

August 22, 2015
Tweet

More Decks by tkuchiki

Other Decks in Programming

Transcript

  1. ιʔγϟϧήʔϜʹ͓͚Δ
    AWSҠߦࣄྫ
    YAPC::Asia TOKYO 2015
    @tkuchiki

    View Slide

  2. WHOAMI(1)
    • @tkuchiki
    • KAYAC Inc. 4೥໨
    • ΦϖϨʔγϣϯΤϯδχΞ
    • ओʹιʔγϟϧήʔϜ୲౰
    • perl ͸ΞϓϦέʔγϣϯΤϯδχΞͩͬͨ
    ࠒʹ 3ϲ݄΄Ͳ...

    View Slide

  3. Agenda
    • AWS Ҡߦͷഎܠ
    • ࣄྫ 1
    • ࣄྫ 2, 3
    • ؂ࢹʹ͍ͭͯ
    • Jenkins Slave ͷ׆༻
    • AWS ҠߦͷϝϦοτɾσϝϦοτ
    • ஌ݟ

    View Slide

  4. AWS Ҡߦͷഎܠ
    • 2011 ೥͔Β DC Ͱͷӡ༻։࢝
    • 5 ೥ϦʔεͰαʔόػߪೖ
    • Ϧʔεظؒऴྃޙͷ͜ͱΛߟ͑Δඞཁੑ
    • 2014 ೥Ҏ߱ʹϦϦʔεͨ͠αʔϏε͸ɺࠓޙ
    ͷ͜ͱΛݟਾ͑ͯ AWS ʹߏங

    View Slide

  5. AWS Ҡߦͷഎܠ
    • ࣾ಺࠷େن໛ͷαʔϏεͷ AWS Ҡߦͱ਺ݸͷ
    αʔϏεऴྃ
    • ࢖༻͠ͳ͍αʔό͕૿Ճ
    • 2014 ೥ 3 ݄ʹϥοΫ࡟ݮ
    • ࠓޙ΋༨৒αʔό͕૿Ճ͢ΔՄೳੑΛߟ͑ͯ
    ࠷దԽ͠΍͍͢ AWS ʹશ໘Ҡߦ͢Δ͜ͱʹ

    View Slide

  6. ࣄྫ 1

    View Slide

  7. • ๯ݥΫΠζΩϯάμϜ
    • Ҡߦظؒ 1 ϲ݄
    • Ҡߦ࣌ʹ perl ͷϚΠφʔόʔδϣϯΞοϓ
    • cpan module ͷόʔδϣϯ΋
    • CentOS 6ΛϚΠφʔόʔδϣϯΞοϓ
    ࣄྫ 1

    View Slide

  8. • Ҡߦମ੍
    • ΞϓϦέʔγϣϯΤϯδχΞ 2 ໊
    • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊
    • Φϒβʔόʔ 2 ໊
    ࣄྫ 1

    View Slide

  9. Ҡߦ࡞ۀ
    • dev αʔόߏங
    • RDS for MySQL ͷݕূ
    • ElastiCache for Redis ͷݕূ
    • Cpanfile ࡞੒ / cpan module ͷόʔδϣϯΞοϓ
    • Mozilla::CA ͷ໰୊
    • ຊ൪αʔόߏங
    • ϝϯςφϯε

    View Slide

  10. dev αʔόߏங
    • dev αʔό͕ 2 ୆͋ͬͨ
    • ৑௕ߏ੒Ͱ͸ͳ͍
    • ؅ཧ͕େมʹͳΔ͚ͩͳͷͰ1୆ʹ·ͱΊΔ
    • Chef cookbook ͷ੔උ
    • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠
    cookbook Խ

    View Slide

  11. RDS for MySQL ͷݕূ
    • RDS ͸λΠϜκʔϯͷઃఆ͕Ͱ͖ͳ͍ͷͰ઀
    ଓ࣌ʹઃఆมߋ͢Δ
    • MySQL ͷ৔߹ɺinit_connect
    • SET SESSION time_zone = CASE WHEN
    POSITION('rds' IN CURRENT_USER()) = 1
    THEN 'UTC' ELSE '+9:00' END;

    View Slide

  12. RDS for MySQL ͷݕূ
    • Failover ͯ͠΋໰୊ͳ͍͔
    • init_connect ͰΤϥʔ͕ͰΔ৔߹͕͋Δͱ
    ͍͏৘ใ͕...
    • reboot with failover ػೳΛ࢖༻
    • MySQL ͷ reconnect Λ༗ޮʹ͠ͳ͍

    View Slide

  13. ΦϯϓϨ MySQL -> RDS
    • ΦϯϓϨ - RDS Ͱ Replication ͕૊ΊΔ
    • RDS ͸λΠϜκʔϯ͕มߋͰ͖ͳ͍
    • UTC ͷΈ
    • JST Ͱӡ༻͍ͯ͠Δ৔߹ 9 ࣌ؒͣΕΔ
    • ࣌ࠁؔ਺Λ࢖͍ͬͯͳ͍৔߹͸໰୊ͳ͍

    View Slide

  14. MySQL ͷσʔλαΠζ
    • MySQL 5.5
    • mysqldump(raw): 14.92GB
    • mysqldump(zgip): 2.79GB

    View Slide

  15. $ time zcat dump.sql.gz | mysql -u ...
    real 209m18.642s
    user 6m42.994s
    sys 0m16.495s
    MySQL ͷΠϯϙʔτ

    View Slide

  16. 209෼…

    View Slide

  17. Ͳ͏ʹ͔ͯ͠
    ߴ଎Խ͍ͨ͠

    View Slide

  18. View Slide

  19. खॱ͕·ͱΊ
    ΒΕ͍ͯͨ

    View Slide

  20. MySQL5.5 ࠷଎σʔλҠߦखॱ
    • ςʔϒϧ࡞੒
    • ϓϥΠϚϦΠϯσοΫε࡞੒
    • ֎෦Ωʔ੍໿෇༩
    • ֎෦Ωʔ੍໿ແޮԽ
    • σʔλϩʔυ
    • ֎෦Ωʔ੍໿༗ޮԽ
    • ηΧϯμϦΠϯσοΫε࡞੒
    http://dbstudy.info/files/20131007/mysql56_load_r2.pdf

    View Slide

  21. mysqldump
    • ηΧϯμϦΠϯσοΫεൈ͖ͰdumpͰ͖ͳ͍
    • --disable-keys(-K) Λ࢖͏ͱߦ͕͢΂ͯૠೖ͞
    Εͨ͋ͱʹΠϯσοΫε࡞੒
    • INSERT จ͕ ALTER TABLE (DISABLE|
    ENABLE) KEYS ͰғΘΕΔ
    • MyISAM ͷΈ...

    View Slide

  22. mysqldumpindex
    • https://github.com/tkuchiki/mysqldumpindex
    • SHOW TABLES ͱ SHOW INDEXES FROM ͷ݁Ռ
    ͔ΒɺηΧϯμϦΠϯσοΫεͷ࡞੒ɺ࡟আ SQL
    Λ࡞੒͢Δ
    • mysqldump --no-data Ͱ dump ͨ͠ SQL ͱɺη
    ΧϯμϦΠϯσοΫε࡟আɾ࡞੒ޙʹ dump ͨ͠
    SQL ͷ diff ΛͱͬͯҰக͢Δ͜ͱΛ֬ೝ

    View Slide

  23. # σʔλΠϯϙʔτ
    $ time { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz;
    echo "FOREIGN_KEY_CHECKS=1"; } | mysql -u ...
    real 54m52.894s
    user 6m43.438s
    sys 0m15.711s
    # ηΧϯμϦΠϯσοΫε࡞੒
    $ time mysql -u ... < create_index.sql
    real 38m26.767s
    user 0m0.010s
    sys 0m0.018s
    ࠷଎खॱͰͷσʔλҠߦ࣌ؒ
    209 ෼͔Β 93෼ʹ୹ॖʂ

    View Slide

  24. mysqldump(percona-server)
    • --innodb-optimize-keys
    • CREATE TABLE จ͔Β UNIQUE ͱ INDEX
    ΛऔΓআ͘
    • INSERT จͷޙΖʹ ALTER TABLE ADD
    [UNIQUE] KEY Λ௥Ճ

    View Slide

  25. mysqldump(percona-server)
    • --innodb-optimize-keys Λ࢖͏͚ͩͰ࠷଎ͷ
    खॱʹͳΓͦ͏
    • ͱࢥ࣮ͬͯݧͯ͠Έ·͕ͨ͠ૣ͘ͳΒͣ

    View Slide

  26. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ
    • dsn ʹ timeout Λઃఆ
    • mysql_connect_timeout
    • mysql_read_timeout
    • mysql_write_timeout
    • ઃఆ͠ͳ͍ͱϓϩηε͕࠶ੜ੒͞Εͳ͍ͱ࠶઀ଓ
    ͠ͳ͍
    • DNS ΛҾ͖௚͞ͳ͍

    View Slide

  27. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ
    • timeout Λઃఆͯ͠΋ͩΊͳ৔߹͕͋Δ
    • ωοτϫʔΫஅ
    • MySQL server has gone away
    • Ұ౓ࣦഊ͢Δͱϓϩηε͕࠶ੜ੒͞ΕΔ·Ͱܨ
    ͕Βͳ͍...
    • ping Ͱܨ͕Δ͔֬ೝ͢Δ͔ɺDBIx::Connector
    Λ࢖͏

    View Slide

  28. if (!$dbh->ping) {
    $dbh->disconnect;
    $dbh = DBI->connect($dsn,...);
    }
    RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ

    View Slide

  29. ElastiCache for Redis ͷݕূ
    • RDS ͱҧͬͯɺreboot with failover ػೳ͕࢖
    ͑ͳ͍
    • 30 ~ 60sec ͔͔ΔॲཧΛͤ͞Δͱ Failover
    ͢Δ͜ͱ͕͋Δ
    • iptables ͰύέοτΛམͱ͢
    • ࠓͷͱ͜Ζਖ਼֬ʹςετ͢Δ͜ͱ͕Ͱ͖ͳ͍

    View Slide

  30. ElastiCache Ͱਖ਼ৗʹ Failover
    ͤ͞Δ
    • timeout Λઃఆ
    • timeout ͯ͠΋ DNS ͸Ҿ͖௚͞ͳ͍
    • ping Ͱ֬ೝͯ͠ɺͩΊͳΒ࠶઀ଓ

    View Slide

  31. if (!$redis->ping) {
    $redis->quit;
    $redis = Redis::Fast->new(...);
    }
    ElastiCache Ͱਖ਼ৗʹ Failover
    ͤ͞Δ

    View Slide

  32. Cpanfile ࡞੒ /
    cpan module ͷόʔδϣϯΞοϓ
    • ؀ڥߏஙΛ༰қʹߦ͑ΔΑ͏ʹ
    • ࠷৽൛ͷ cpan module ΛೖΕͯςετ
    • ໰୊ͳ͔ͬͨͷͰ࠷৽൛ʹ
    • DateTime ͚ͩ͏Δ͏ඵૠೖ·͑ʹόʔδϣ
    ϯμ΢ϯ
    • :60 Λࠩ͠ࠐΜͰ͘Δ...

    View Slide

  33. Mozilla::CA ͷ໰୊
    • Mozilla::CA::SSL_ca_file() ͕ฦ͢ূ໌ॻ͕ݹ
    ͍
    • ΞϓϦ಺՝ۚͷϨγʔτݕূͰΤϥʔ͕ग़
    ΔΑ͏ʹ...
    • ෇ଐͷ mk-ca-bundle.pl Λ࢖༻

    View Slide

  34. $ ./perl5/lib/perl5/Mozilla/mk-ca-bundle.pl --help
    ...
    -p list of Mozilla trust purposes and levels for certificates
    to include in output. Takes the form of a comma separated
    list of purposes, a colon, and a comma separated list of
    levels. (default: SERVER_AUTH:TRUSTED_DELEGATOR)
    Valid purposes are:
    ALL, DIGITAL_SIGNATURE, NON_REPUDIATION,
    KEY_ENCIPHERMENT, DATA_ENCIPHERMENT, KEY_AGREEMENT,
    KEY_CERT_SIGN, CRL_SIGN, SERVER_AUTH, CLIENT_AUTH,
    CODE_SIGNING, EMAIL_PROTECTION, IPSEC_END_SYSTEM,
    IPSEC_TUNNEL, IPSEC_USER, TIME_STAMPING,
    STEP_UP_APPROVED
    Valid levels are:
    ALL, TRUSTED_DELEGATOR, NOT_TRUSTED, MUST_VERIFY_TRUST,
    TRUSTED
    Mozilla::CA ͷ໰୊

    View Slide

  35. Mozilla::CA ͷ໰୊
    • -p PURPOSES:LEVELS Ͱ໨తͱϨϕϧ͕ࢦఆ
    Ͱ͖Δ
    • -p ALL:ALL ͰશࢦఆՄ
    • ੜ੒ͨ͠ূ໌ॻΛࢦఆ͢ΔΑ͏ʹมߋ

    View Slide

  36. my %args = (
    ssl_opts => {
    SSL_ca_file => '/path/to/ca-bundle.crt',
    },
    ...
    );
    my $furl = Furl->new(%args);
    ...
    Mozilla::CA ͷ໰୊

    View Slide

  37. ຊ൪αʔόߏங
    • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ
    cookbook Λվྑ
    • αʔόߏ੒Λେ෯ʹมߋ
    • web αʔόͱ app αʔόͷ౷߹
    • ࣗલͰ MySQL ͱ Redis Λཱͯͳ͍
    • RDS ͱ ElastiCache

    View Slide

  38. -#
    /HJOY
    "QQ
    3FEJT
    NBTUFS
    3FEJT
    TMBWF
    #BUDI
    .Z42-NBTUFS .Z42-TMBWF
    %FQMPZ
    STZOD
    "SDIFS

    7*1
    DIFGTPMP
    DBQJTUSBOP

    View Slide

  39. ຊ൪αʔόߏங
    • consul ͷಋೖ
    • ಺෦ DNS
    • stretcher ͷಋೖ
    • Fluentd ͷઃఆݟ௚͠

    View Slide

  40. Consul
    • Hashicorp ੡ͷΦʔέετϨʔγϣϯπʔϧ
    • KVSɺDNSɺHTTP ΠϯλϑΣʔε
    • ಺෦ DNS ͱͯ͠׆༻

    View Slide

  41. Consul DNS
    • 8600 port Ͱఏڙ
    • node ͱ service ʹରԠ
    • nodename.node(.datacenter).consul
    • servicename.service(.datacenter).consul

    View Slide

  42. dnsmasq
    • DNS forwarder, cache
    • consul ͷ 8600 port ʹ forward
    • શαʔόͰՔಇ

    View Slide

  43. $ cat /etc/dnsmasq.conf
    server=/consul/127.0.0.1#8600
    bind-interfaces
    listen-address=127.0.0.1
    dnsmasq

    View Slide

  44. VPC DHCP Options Set
    • VPC ͷ DHCP ͷઃఆΛߦ͏
    • resolv.conf ͕ੜ੒͞ΕΔ
    • ݱࡏ͸ Management Console ͔Β΋ઃఆՄೳ

    View Slide

  45. $ cat /etc/resolv.conf
    ; generated by /sbin/dhclient-script
    search node.consul service.consul
    nameserver 127.0.0.1
    nameserver 10.1.0.2 ; VPC ͷ DNS
    nameserver 10.1.11.100 ; Unbound
    VPC DHCP Options Set

    View Slide

  46. Unbound
    • DNS cache, resolver
    • VPC ͷ DNS ΛΩϟογϡ
    • VPC ͷ resolver ʹ໰୊͕͋ͬͯ΋໊લղܾ
    ͕Ͱ͖ΔΑ͏ʹ
    • deploy αʔόͰՔಇ

    View Slide

  47. stretcher ʹΑΔ deploy
    • https://github.com/fujiwara/stretcher
    • Go ੡
    • S3 ͔Β tarball Λऔಘͯ͠ల։ɺrsync
    • consulɺserf ͱ࿈ܞ
    • ৄ͘͠͸ɺhttps://speakerdeck.com/fujiwara3/
    consultozi-zuo-osswohuo-yong-sita100tai-gui-mo-
    falsewebsabisuyun-yong

    View Slide

  48. &-#
    /HJOY
    "QQ
    &MBTUJ$BDIF
    NBTUFS
    &MBTUJ$BDIF
    TMBWF
    #BUDI
    3%4NBTUFS
    3%4TMBWF
    CBDLVQ

    %FQMPZ
    DPOTVM
    FWFOU
    #VJME
    6QMPBE
    4
    DIFGTPMP
    DBQJTUSBOP

    View Slide

  49. ϝϯςφϯε (ࣄલ४උ)
    • ηΧϯμϦΠϯσοΫεΛ࡟আ͓͍ͯͨ͠ςʔ
    ϒϧΛҠߦઌͷ DB ʹ༻ҙ
    • RDS Λ Single-AZ ʹ͢Δ
    • ύϥϝʔλάϧʔϓͷௐ੔

    View Slide

  50. ϝϯςφϯε (ࣄલ४උ)
    • RDS ΛSingle-AZ ʹ͢Δ
    • Multi-AZ ͷ৔߹ɺ଱ো֐ੑ͸޲্͢Δ͕ॻ
    ͖ࠐΈ଎౓͕2ഒʹ

    View Slide

  51. ϝϯςφϯε (ࣄલ४උ)
    • ύϥϝʔλάϧʔϓͷมߋ
    • innodb_flush_log_at_trx_commit=0
    • innodb_support_xa=0
    • sync_binlog=0

    View Slide

  52. ϝϯςφϯε
    • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ
    • MySQL ͷσʔλҠߦ
    • Redis ͷσʔλҠߦ
    • ಈ࡞֬ೝ
    • DNS੾Γସ͑
    • Multi-AZ Խ & ύϥϝʔλάϧʔϓมߋ

    View Slide

  53. MySQL ͷσʔλҠߦ
    • εΩʔϚ͸༧Ί৽ DB ʹೖΕͯ͋ΔͷͰɺσʔ
    λ͚ͩ dump (& gzip)
    • FOREIGN_KEY_CHECKS=0 ͱ 1 Λ zcat ͷલ
    ޙʹೖΕͯ mysql ʹ౉͢

    View Slide

  54. $ { echo "FOREIGN_KEY_CHECKS=0";
    zcat data.sql.gz; echo
    "FOREIGN_KEY_CHECKS=1"; } |
    mysql -u ...
    MySQL ͷσʔλҠߦ

    View Slide

  55. MySQL ͷσʔλҠߦ
    • import ͕ऴΘͬͨΒηΧϯμϦΠϯσοΫε
    Λ࡞੒͢Δ
    • information_schema.tables Λݟ֤ͯςʔϒϧ
    ͷߦ਺Λൺֱ

    View Slide

  56. mysql -u USER -B -N -e "SELECT TABLE_NAME,
    TABLE_ROWS FROM information_schema.tables
    WHERE TABLE_SCHEMA = 'my_db' ORDER BY
    TABLE_NAME"
    MySQL ͷσʔλҠߦ

    View Slide

  57. Redis ͷσʔλҠߦ
    • redis-cli CONFIG SET appendonly yes
    • appendfsync no ʹ͢Δͱߴ଎(fsync͠ͳ͍)
    • redis-cli BGSAVE
    • Redis Λఀࢭ
    • cat appendonly.aof | redis-cli --pipe -h ...

    View Slide

  58. Redis ͷσʔλҠߦ
    • Ҡߦલޙͷ྆؀ڥͰ KEYS “*” ͷ݁ՌΛϑΝΠ
    ϧʹམͱ͢
    • ݁ՌΛ sort
    • diff Λऔͬͯ֬ೝ
    • expireͰ࡟আ͞Ε͍ͯΔͱଟগࠩ෼͕ग़Δ

    View Slide

  59. ಈ࡞֬ೝ
    • Ҡߦޙͷ؀ڥʹ઀ଓͰ͖ΔΞϓϦΛ୺຤ʹೖ
    Ε͓ͯ͘
    • υϝΠϯΛઃఆ
    • εϚʔτϑΥϯͷ DNS ઃఆΛมߋͨ͠Γɺ
    DNS ͷԠ౴Λِ૷͢ΔΑΓָ

    View Slide

  60. Single-AZ -> Multi-AZ
    ύϥϝʔλάϧʔϓͷมߋ
    • Multi-AZ ʹมߋ
    • Single-AZ ͔Β Multi-AZ ΁ͷมߋ͸ΦϯϥΠϯͰ
    ࣮ߦՄೳ
    • ύϥϝʔλάϧʔϓͷมߋ
    • innodb_flush_log_at_trx_commit=1
    • innodb_support_xa=1
    • sync_binlog=1

    View Slide

  61. DNS ͷ੾Γସ͑
    • TTL Λ༧Ί୹͓ͯ͘͘͠
    • TTL Λ୹ͯ͘͠΋چ؀ڥʹΞΫηε͕དྷΔՄ
    ೳੑ
    • DB ΍ KVS ͸ඞͣࢭΊ͓ͯ͘

    View Slide

  62. ࣄྫ 2,3

    View Slide

  63. ࣄྫ 2,3
    • ๻ΒͷߕࢠԂʂ೤ಆฤͷ Mobage ൛ͱ iOS / Android

    • Ҡߦظؒ߹Θͤͯ 1 ϲ݄
    • ΄ͱΜͲαʔόߏ੒͕ಉ͡ͳͷͰಉ࣌ਐߦ
    • perlbrew ͰΠϯετʔϧͨ͠ perl ؀ڥΛͦͷ··
    ͍࣋ͬͯ͘
    • SL6 -> CentOS6 ͳͷͰ

    View Slide

  64. ࣄྫ 2,3
    • Ҡߦମ੍
    • ΞϓϦέʔγϣϯΤϯδχΞ ֤ 1 ໊
    • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊
    • Φϒβʔόʔ 2 ໊

    View Slide

  65. Ҡߦ࡞ۀ
    • dev αʔόߏங
    • (RDS for MySQL ͷݕূ)
    • yrmcds ͷݕূ
    • ຊ൪αʔόߏங
    • ϝϯςφϯε
    • Ҡߦޙʹൃੜͨ͠໰୊

    View Slide

  66. dev αʔόߏங
    • Chef cookbook ͷ੔උ
    • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠
    cookbook Խ

    View Slide

  67. yrmcds ͷݕূ
    • session ʹ Kyoto Tycoon Λ࢖༻
    • ػೳతʹ໰୊͸ͳ͍
    • ٕज़ݕূ΋݉Ͷͯผͷϛυϧ΢ΣΞΛ࢖͏
    • Redis ͩͱΞϓϦͷେ෯ͳमਖ਼͕ඞཁ
    • Memcached ޓ׵ͷ΋ͷ͕ྑ͍
    • yrmcds

    View Slide

  68. yrmcds
    • https://github.com/cybozu/yrmcds
    • Memcached ΄΅׬શޓ׵
    • master / slave ϨϓϦέʔγϣϯ
    • VIP ͕෇͍͍ͯΔαʔό͕ master
    • ϝϞϦ؅ཧʹ Slab Λ࢖༻͍ͯ͠ͳ͍
    • αʔόαΠυϩοΫ

    View Slide

  69. EC2 ʹ VIP Λ͚ͭΔ
    • EC2 ʹ VIP Λ͚ͭͯ΋௨৴Ͱ͖ͳ͍
    • ENI ͸σϑΥϧτͰૹ৴ݩ/ૹ৴ઌνΣοΫ
    ͕༗ޮʹͳ͍ͬͯΔ
    • ແޮʹ͠ͳ͍ͱࣗ෼ѼͯҎ֎ͷύέοτ
    Λड͚औΕͳ͍

    View Slide

  70. Keepalived
    • VIP ͷ੾Γସ͑ʹ࢖༻
    • VPC Ͱ͸ϚϧνΩϟετ͕࢖͑ͳ͍
    • ؤுΕ͹࢖͑ΔΑ͏Ͱ͕͢ɺKeepalived ͷՄ
    ༻ੑ΋ߟ͑ͳͯ͘͸ͳΒͳ͍
    • Consul lock Λ׆༻ͨ͠ Failover γεςϜ

    View Slide

  71. Consul lock
    • ෼ࢄϩοΫػೳ
    • consul lock service/lock deploy.sh
    • ෼ࢄηϚϑΥػೳ
    • consul lock -n 2 service/lock deploy.sh

    View Slide

  72. Consul lock Λ׆༻ͨ͠
    Failover γεςϜ
    • lock Λऔͬͨϗετʹ VIP Λ͚ͭΔ
    • ϩʔΧϧͷ yrmcds ͷࢮ׆؂ࢹ
    • μ΢ϯͨ͠Β lock Λղ์͢Δ
    • Slave Ͱ΋ lock ΛऔΓʹߦͬͯ lock ղ์଴ͪͷঢ়ଶʹ
    ͢Δ
    • yrmcds ͕μ΢ϯ͢Δͱ Slave ͷ͏ͪҰ͕ͭ lock Λऔͬ
    ͯ VIP Λ͚ͭΔ(master ʹঢ֨)

    View Slide

  73. Consul lock Λ׆༻ͨ͠
    Failover γεςϜ
    • VPC Route Table ͷॻ͖׵͑
    • yrmcds Λ svc -o Ͱىಈ
    • উखʹ࠶ىಈ͠ͳ͍Α͏ʹ
    • /service/yrmcds ΋ىಈ࣌ʹ࡟আ
    • αʔό͕ಥવࢮͯ͠ VIP ͕͍ͭͨ··࠶ىಈ
    ͯ͠΋ master ʹͳΒͳ͍Α͏ʹ

    View Slide

  74. ຊ൪αʔόߏங
    • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ
    cookbook Λվྑ
    • αʔόߏ੒Λมߋ
    • web αʔόͱ app αʔόͷ౷߹
    • Kyoto Tycoon Λ yrmcds ʹҠߦ

    View Slide

  75. -#
    /HJOY
    "QQ
    ,ZPUP5ZDPPO
    TFTTJPO
    #BUDI
    .Z42-NBTUFS
    .Z42-TMBWF
    %FQMPZ
    STZOD
    "SDIFS

    DIFGTPMP
    DBQJTUSBOP

    ,ZPUP5ZDPPO
    TFTTJPO CBDLVQ

    )"1SPYZ
    .FNDBDIFE
    DBDIF

    )"1SPYZ
    )"1SPYZ

    View Slide

  76. ຊ൪αʔόߏங
    • MySQL Λ Slave ࢀর͠ͳͯ͘΋໰୊ͳ͘ͳͬ
    ͨͷͰ HAProxy ͕ෆཁʹ
    • Consul + dnsmasq ಋೖ
    • ಺෦DNS

    View Slide

  77. &-#
    ZSNDET
    NBTUFS
    ZSNDET
    TMBWF
    #BUDI
    %FQMPZ
    STZOD
    "SDIFS

    7*1
    DIFGTPMP
    DBQJTUSBOP

    3%4NBTUFS 3%4TMBWF
    CBDLVQ

    /HJOY
    "QQ

    View Slide

  78. Ҡߦ࡞ۀ
    • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ
    • MySQL ͷσʔλҠߦ
    • Session ͷσʔλҠߦ
    • (ಈ࡞֬ೝ)
    • (DNS੾Γସ͑)

    View Slide

  79. MySQL ͷσʔλҠߦ
    • MySQL ͷσʔλҠߦΛ࠷଎खॱͰ࣮ࢪ
    • Duplicate entry ͕ग़ࣦͯഊ...
    • લ೔ͷԋशͰ͸੒ޭ
    • ະͩʹݪҼෆ໌
    • ֎෦Ωʔ੍໿Λ΄ͱΜͲ࢖͍ͬͯͳ͍ͨΊ௨
    ৗͷ mysqldump Ͱ΋ 5 ෼͔͕ࠩ͠ͳ͔ͬͨ

    View Slide

  80. Session ͷσʔλҠߦ
    • ͠·ͤΜͰͨ͠
    • ϩάΠϯ͠௚͠
    • Ϣʔβ਺తʹ໰୊ͳ͠
    • ϩάΠϯ͕ूத͢Δͱࡹ͖੾Εͳ͍৔߹΋
    ͋ΔͷͰ௨ৗ͸Ҡߦͨ͠΄͏͕ྑ͍

    View Slide

  81. Ҡߦޙʹൃੜͨ͠໰୊
    • Consul ͸਺ඵ Leader lost ͢Δ͜ͱ͕͋Δ
    • Leader lost ͢Δͱ lock ͕֎ΕΔ...
    • Ҡߦޙ 2 ೔࿈ଓͰ Failover
    • dns_config.node_ttl, allow_stale, max_stale
    Λઃఆ͢Δ
    • Leader lost ࣌΋਺ඵͳΒ DNS ͕Ҿ͚Δ

    View Slide

  82. ؂ࢹʹ
    ͍ͭͯ

    View Slide

  83. Zabbix
    • ࣗࣾαʔϏεͷ؂ࢹ
    • VPC Peering Connection Ͱ internal ௨৴
    • Ұ෦ Zabbix Proxy Λར༻

    View Slide

  84. Zabbix Proxy
    • Zabbix Server ͷ୅ΘΓͱͯ͠ಈ࡞
    • ෛՙͷܰݮ
    • ऩूσʔλΛ֤ϗετͰόοϑΝϦϯά
    • Firewall ͷઃఆ͕༰қ
    • ֎෦νΣοΫ
    • ίϚϯυΛ࣮ߦͯ͠ MySQL ͷ QPS Λܭଌ

    View Slide

  85. Zabbix Proxy
    • ศར͕ͩ SPOF ʹͳΔ
    • ΠϯλʔωοτΛܦ༝ͤͣ؂ࢹ͕Մೳʹͳͬ
    ͨͷͰ௚઀؂ࢹʹมߋ
    • Active Agent Check
    • ผͷ໰୊͕...

    View Slide

  86. Zabbix Active Check
    • ֎෦νΣοΫͷεΫϦϓτʹύεϫʔυΛॻ
    ͍͍ͯΔ
    • Zabbix Proxy ͕ೖ͍ͬͯΔϗετʹઃఆ
    • ௚઀؂ࢹʹͳΓ࢖͑ͳ͘ͳΔ
    • ؂ࢹର৅͕૿͑Δͨͼʹ Zabbix Server ͷઃ
    ఆมߋΛߦ͏ͷ͸ආ͚͍ͨ

    View Slide

  87. Zabbix Active Check
    • UserParameter Λઃఆͨ͠ϗετΛొ࿥
    • RDS ΍ ElastiCache ͳͲͷ؂ࢹ͕Մೳʹ
    • ςϯϓϨʔτʹొ࿥ͨ͠ΞΠςϜΛϗετϚ
    ΫϩͰมߋՄೳʹ
    • ࠶ར༻͠΍͍͢

    View Slide

  88. View Slide

  89. View Slide

  90. PUSH ௨஌༻ SSL ূ໌ॻͷ؂ࢹ
    • PUSH ௨஌ʹ࢖͏ SSL ূ໌ॻͷظݶΛ؂ࢹ
    • ূ໌ॻͷ PATH ͸αʔό͝ͱʹҧ͏
    • ϓϩδΣΫτ͝ͱʹΞΠςϜΛ௥Ճ͢Δͷ͸
    େม
    • Low Level Discovery(LLD) Ͱղܾ

    View Slide

  91. Low Level Discovery(LLD)
    • ΞΠςϜɺτϦΨɺάϥϑΛࣗಈͰੜ੒͢Δ
    ͨΊͷػೳ
    • ϑΝΠϧγεςϜɾωοτϫʔΫΠϯλ
    ϑΣʔεݕग़༻ͷLLD͕૊Έࠐ·Εͨςϯϓ
    Ϩʔτ΋͋Δ
    • LLD Λࣗ࡞͢Δ͜ͱ΋Մೳ

    View Slide

  92. Low Level Discovery(LLD)
    • UserParameter ʹ ΞΠςϜऩू༻ͷઃఆ௥Ճ
    • ΞΠςϜऩू͸ɺܾΊΒΕͨܗࣜͷ JSON
    σʔλΛฦ͢

    View Slide

  93. #!/bin/bash
    CERTS="${1}"
    CERTS_DATA=""
    for CERT in ${CERTS}; do
    CERTS_DATA="${CERTS_DATA},{\"{#CERT_PATH}\":\"${CERT}\"}"
    done
    echo "{\"data\":[$(echo ${CERTS_DATA} | sed -e 's/^,//')]}"
    Low Level Discovery(LLD)

    View Slide

  94. {
    "data": [
    {"{#CERT_PATH}":"/path/to/cert1"},
    {"{#CERT_PATH}":"/path/to/cert2"}
    ]
    }
    Low Level Discovery(LLD)

    View Slide

  95. #!/bin/bash
    CERT="${1}"
    NOW=$(date +%s)
    EXPIRE=$(date -d "$(openssl x509 -in ${CERT} -noout -dates |
    tail -n 1 | sed -e 's/notAfter=//')" +%s)
    echo $(((${EXPIRE} - ${NOW}) / 60 / 60 / 24))
    Low Level Discovery(LLD)

    View Slide

  96. View Slide

  97. View Slide

  98. εϖʔε۠੾ΓͰূ໌ॻΛࢦఆ

    View Slide

  99. RDS ͷ؂ࢹ
    • fluent-plugin-cloudwatch ͰϝτϦΫεΛऩू
    • fluent-plugin-zabbix Ͱ Zabbix ʹૹ৴
    • MySQL ͷ stats ͸ SQL Λ࣮ߦ
    • SHOW ENGINE INNODB STATUS
    • SHOW GLOBAL STATUS

    View Slide

  100. ElastiCache ͷ؂ࢹ
    • redis-cli INFO ͷ stats Λऩू
    • DNS ΛҾ͍ͯ Primary Cache Cluster ͷ IP Λ؂ࢹ
    • લޙͰ஋͕ҟͳΔ৔߹͸ Failover
    • UserParameter ͰίϚϯυ࣮ߦ
    • net.dns.record ͸ DNS ͕Ҿ͚ͳ͍࣌͸Ξϥʔ
    τΛඈ͹ͤͳ͍...

    View Slide

  101. #!/bin/bash
    host="/usr/bin/host"
    DOMAIN=${1}
    TIMEOUT=10
    RECORDS=$($host -s -W $TIMEOUT $DOMAIN | awk 'NR >
    1 {print $NF}' | sort)
    [ "${RECORDS}" = "" ] && echo "fail" && exit 1
    echo $RECORDS
    ElastiCache ͷ؂ࢹ

    View Slide

  102. yrmcds ͷ؂ࢹ
    • stats ͰϝτϦΫεΛऔಘՄೳ
    • https://github.com/cybozu/yrmcds/issues/47
    ͰରԠ͍͖ͯͨͩ͠·ͨ͠
    • ͋Γ͕ͱ͏͍͟͝·͢ʂ
    • VIP ͕෇͍͍ͯΔΠϯελϯεͷ Private IP Λ
    νΣοΫ
    • લޙͰ஋͕ҟͳΔ৔߹͸ Failover ͱͯ͠ݕग़

    View Slide

  103. Jenkins
    Slave
    ͷ׆༻

    View Slide

  104. Jenkins EC2 Plugin
    • Jenkins Slave Λ AMI ͔Βىಈ
    • ςετ͕࣮ߦ͞Εͳ͘ͳ͔ͬͯΒ 30෼͘Β
    ͍Ͱ Terminate
    • ىಈ͕࣌ؒ୹͘ͳΔͷͰ҆ՁʹͳΔ
    • Spot Instance ΋࢖͑ΔͷͰɺΑΓ҆Ձʹ

    View Slide

  105. Jenkins EC2 Plugin
    • 32 core ͷ CPU ͰίΞ਺෼·Ͱฒྻ਺Λ͋͛
    ͯςετΛճ͢ͱ͚͜ΔΑ͏ʹ...
    • MySQL ͕ EAGAIN ΤϥʔΛग़͢Α͏ʹ
    • Test::mysqld Ͱෳ਺ݸͷ MySQL Λ্ཱͪ
    ͍͛ͯΔ
    • fs.aio-max-nr=65536 Λ 4 ഒʹͯ͠ճආ

    View Slide

  106. Jenkins EC2 Plugin
    • ଞͷΧʔωϧύϥϝʔλ΋νϡʔχϯά
    • ςετ͸ߴ଎ԽͰ͖ͣ...
    • MySQL ͷ্ཱͪ͛ʹ͕͔͔࣌ؒΔ
    • Test::mysqld::Multi
    • ฒྻ਺ΛNഒͯ͠΋࣮ߦ࣌ؒ͸1/NʹͳΒͳ͍
    • CPUΛ࢖͍੾Εͳ͍

    View Slide

  107. AWS Ҡߦͷ
    ϝϦοτɾ
    σϝϦοτ

    View Slide

  108. AWS Ҡߦͷֹۚ໘Ͱͷ
    ϝϦοτ
    • ϓϩδΣΫτ͝ͱʹਖ਼֬ͳར༻ֹ͕ग़ͤΔ
    • ΦϯϓϨͰ͸ɺαʔό(VM)͕༨ͬͨ৔߹શ
    ࣾඅ༻ʹճ͍ͯͨ͠
    • ݟ্͔͚͸҆ՁʹͳΔ͕ɺશࣾతʹ͸අ༻
    ͕͔͔͍ͬͯΔ͜ͱΛཧղ͞Εͳ͍

    View Slide

  109. AWS Ҡߦͷֹۚ໘Ͱͷ
    σϝϦοτ
    • අ༻͸ଟগ૿Ճ͢Δ
    • େن໛Ͱͳ͚Ε͹ਓ݅අΛؚΊΔͱ҆Ձ(ͳ
    ͸ͣ)
    • ࣗલͰαʔόӡ༻͢ΔΑΓָ
    • దਖ਼ͳඅ༻͕ܭ্͞ΕΔͨΊɺඅ༻͕૿Ճ͠
    ͨΑ͏ʹࢥΘΕͯ͠·͏

    View Slide

  110. AWS Ҡߦͷػೳ໘Ͱͷ
    ϝϦοτ
    • ϚωʔδυαʔϏεͰӡ༻ෛՙͷ௿ݮ
    • ҆Ձʹαʔόͷަ׵͕Մೳ
    • ϋʔυ΢ΣΞੑೳͷ޲্
    • ΞϓϦͷνϡʔχϯάͳ͠ͰϨεϙϯελ
    ΠϜվળ

    View Slide

  111. View Slide

  112. AWS Ҡߦͷػೳ໘Ͱͷ
    σϝϦοτ
    • ඞཁͳػೳ͕ͳ͍͜ͱ͕͋Δ
    • ElastiCache ͷ reboot with failover
    • RDS ͷ λΠϜκʔϯରԠ

    View Slide

  113. ஌ݟ

    View Slide

  114. ஌ݟ
    • ҆ఆӡ༻Ͱ͖͍ͯΔͱݱঢ়ҡ࣋ʹͳΓ͕ͪ
    • αʔόߏ੒ͷݟ௚͕͠Ͱ͖Δνϟϯε
    • ྑ͘ͳ͔ͬͨ෦෼͕ු͖ூΓʹͳΔ
    • ؂ࢹ͢΂͖͜ͱ͕Ͱ͖͍ͯͳ͍
    • Chef Ͱ؅ཧͰ͖͍ͯΔ΋ͷͱͰ͖͍ͯͳ͍
    ΋ͷ͕ࠞࡏ͍ͯ͠Δ

    View Slide

  115. ஌ݟ
    • े਺୆ن໛ͷγεςϜͰ͋Ε͹ 2~3 ໊ఔ౓Ͱ
    1 ϲ݄͋Ε͹ҠߦͰ͖Δ
    • ҙ֎ͱԿͱ͔ͳΔ
    • ϝϯςφϯε೔ͷௐ੔͕ҙ֎ͱେม

    View Slide

  116. ஌ݟ
    • ࣄલԋशΛԿ౓΋΍Δͱ҆৺
    • MySQL ͷσʔλҠߦ͸ݕূ࣌ʹ3ɾ4ճɺલ
    ೔ʹ 2 ճ࣮ࢪ
    • ίϚϯυΛίϐϖ࣮ͯ͠ߦ͢Δ͚ͩͰྑ͍
    ঢ়ଶʹ͓ͯ͘͠ͱߋʹ҆৺

    View Slide

  117. ·ͱΊ
    • AWS ͷҠߦࣄྫ঺հ
    • ݱঢ়ҡ࣋ΑΓ΋վળ
    • ߏ੒Λݟ௚͢ྑ͍ػձ
    • ػձ͕ͳ͍ͱվળ͠ʹ͍͘

    View Slide