ソーシャルゲームにおける AWS 移行事例

トラックE (607-608) 2015-08-22 14:20:00

Transcript

1. ιʔγϟϧήʔϜʹ͓͚Δ AWSҠߦࣄྫ YAPC::Asia TOKYO 2015 @tkuchiki
2. WHOAMI(1) • @tkuchiki • KAYAC Inc. 4೥໨ • ΦϖϨʔγϣϯΤϯδχΞ • ओʹιʔγϟϧήʔϜ୲౰ • perl ͸ΞϓϦέʔγϣϯΤϯδχΞͩͬͨ ࠒʹ 3ϲ݄΄Ͳ...
3. Agenda • AWS Ҡߦͷഎܠ • ࣄྫ 1 • ࣄྫ 2, 3 • ؂ࢹʹ͍ͭͯ • Jenkins Slave ͷ׆༻ • AWS ҠߦͷϝϦοτɾσϝϦοτ • ஌ݟ
4. AWS Ҡߦͷഎܠ • 2011 ೥͔Β DC Ͱͷӡ༻։࢝ • 5 ೥ϦʔεͰαʔόػߪೖ • Ϧʔεظؒऴྃޙͷ͜ͱΛߟ͑Δඞཁੑ • 2014 ೥Ҏ߱ʹϦϦʔεͨ͠αʔϏε͸ɺࠓޙ ͷ͜ͱΛݟਾ͑ͯ AWS ʹߏங
5. AWS Ҡߦͷഎܠ • ࣾ಺࠷େن໛ͷαʔϏεͷ AWS Ҡߦͱ਺ݸͷ αʔϏεऴྃ • ࢖༻͠ͳ͍αʔό͕૿Ճ • 2014 ೥ 3 ݄ʹϥοΫ࡟ݮ • ࠓޙ΋༨৒αʔό͕૿Ճ͢ΔՄೳੑΛߟ͑ͯ ࠷దԽ͠΍͍͢ AWS ʹશ໘Ҡߦ͢Δ͜ͱʹ
6. ࣄྫ 1
7. • ๯ݥΫΠζΩϯάμϜ • Ҡߦظؒ 1 ϲ݄ • Ҡߦ࣌ʹ perl ͷϚΠφʔόʔδϣϯΞοϓ • cpan module ͷόʔδϣϯ΋ • CentOS 6ΛϚΠφʔόʔδϣϯΞοϓ ࣄྫ 1
8. • Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ 2 ໊ • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊ • Φϒβʔόʔ 2 ໊ ࣄྫ 1
9. Ҡߦ࡞ۀ • dev αʔόߏங • RDS for MySQL ͷݕূ • ElastiCache for Redis ͷݕূ • Cpanfile ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • Mozilla::CA ͷ໰୊ • ຊ൪αʔόߏங • ϝϯςφϯε
10. dev αʔόߏங • dev αʔό͕ 2 ୆͋ͬͨ • ৑௕ߏ੒Ͱ͸ͳ͍ • ؅ཧ͕େมʹͳΔ͚ͩͳͷͰ1୆ʹ·ͱΊΔ • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook Խ
11. RDS for MySQL ͷݕূ • RDS ͸λΠϜκʔϯͷઃఆ͕Ͱ͖ͳ͍ͷͰ઀ ଓ࣌ʹઃఆมߋ͢Δ • MySQL ͷ৔߹ɺinit_connect • SET SESSION time_zone = CASE WHEN POSITION('rds' IN CURRENT_USER()) = 1 THEN 'UTC' ELSE '+9:00' END;
12. RDS for MySQL ͷݕূ • Failover ͯ͠΋໰୊ͳ͍͔ • init_connect ͰΤϥʔ͕ͰΔ৔߹͕͋Δͱ ͍͏৘ใ͕... • reboot with failover ػೳΛ࢖༻ • MySQL ͷ reconnect Λ༗ޮʹ͠ͳ͍
13. ΦϯϓϨ MySQL -> RDS • ΦϯϓϨ - RDS Ͱ Replication ͕૊ΊΔ • RDS ͸λΠϜκʔϯ͕มߋͰ͖ͳ͍ • UTC ͷΈ • JST Ͱӡ༻͍ͯ͠Δ৔߹ 9 ࣌ؒͣΕΔ • ࣌ࠁؔ਺Λ࢖͍ͬͯͳ͍৔߹͸໰୊ͳ͍
14. MySQL ͷσʔλαΠζ • MySQL 5.5 • mysqldump(raw): 14.92GB • mysqldump(zgip): 2.79GB
15. $ time zcat dump.sql.gz | mysql -u ... real 209m18.642s user 6m42.994s sys 0m16.495s MySQL ͷΠϯϙʔτ
16. 209෼…
17. Ͳ͏ʹ͔ͯ͠ ߴ଎Խ͍ͨ͠
19. खॱ͕·ͱΊ ΒΕ͍ͯͨ
20. MySQL5.5 ࠷଎σʔλҠߦखॱ • ςʔϒϧ࡞੒ • ϓϥΠϚϦΠϯσοΫε࡞੒ • ֎෦Ωʔ੍໿෇༩ • ֎෦Ωʔ੍໿ແޮԽ • σʔλϩʔυ • ֎෦Ωʔ੍໿༗ޮԽ • ηΧϯμϦΠϯσοΫε࡞੒ http://dbstudy.info/files/20131007/mysql56_load_r2.pdf
21. mysqldump • ηΧϯμϦΠϯσοΫεൈ͖ͰdumpͰ͖ͳ͍ • --disable-keys(-K) Λ࢖͏ͱߦ͕͢΂ͯૠೖ͞ Εͨ͋ͱʹΠϯσοΫε࡞੒ • INSERT จ͕ ALTER TABLE (DISABLE| ENABLE) KEYS ͰғΘΕΔ • MyISAM ͷΈ...
22. mysqldumpindex • https://github.com/tkuchiki/mysqldumpindex • SHOW TABLES ͱ SHOW INDEXES FROM ͷ݁Ռ ͔ΒɺηΧϯμϦΠϯσοΫεͷ࡞੒ɺ࡟আ SQL Λ࡞੒͢Δ • mysqldump --no-data Ͱ dump ͨ͠ SQL ͱɺη ΧϯμϦΠϯσοΫε࡟আɾ࡞੒ޙʹ dump ͨ͠ SQL ͷ diff ΛͱͬͯҰக͢Δ͜ͱΛ֬ೝ
23. # σʔλΠϯϙʔτ $ time { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo "FOREIGN_KEY_CHECKS=1"; } | mysql -u ... real 54m52.894s user 6m43.438s sys 0m15.711s # ηΧϯμϦΠϯσοΫε࡞੒ $ time mysql -u ... < create_index.sql real 38m26.767s user 0m0.010s sys 0m0.018s ࠷଎खॱͰͷσʔλҠߦ࣌ؒ 209 ෼͔Β 93෼ʹ୹ॖʂ
24. mysqldump(percona-server) • --innodb-optimize-keys • CREATE TABLE จ͔Β UNIQUE ͱ INDEX ΛऔΓআ͘ • INSERT จͷޙΖʹ ALTER TABLE ADD [UNIQUE] KEY Λ௥Ճ
25. mysqldump(percona-server) • --innodb-optimize-keys Λ࢖͏͚ͩͰ࠷଎ͷ खॱʹͳΓͦ͏ • ͱࢥ࣮ͬͯݧͯ͠Έ·͕ͨ͠ૣ͘ͳΒͣ
26. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • dsn ʹ timeout Λઃఆ • mysql_connect_timeout • mysql_read_timeout • mysql_write_timeout • ઃఆ͠ͳ͍ͱϓϩηε͕࠶ੜ੒͞Εͳ͍ͱ࠶઀ଓ ͠ͳ͍ • DNS ΛҾ͖௚͞ͳ͍
27. RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆͯ͠΋ͩΊͳ৔߹͕͋Δ • ωοτϫʔΫஅ • MySQL server has gone away • Ұ౓ࣦഊ͢Δͱϓϩηε͕࠶ੜ੒͞ΕΔ·Ͱܨ ͕Βͳ͍... • ping Ͱܨ͕Δ͔֬ೝ͢Δ͔ɺDBIx::Connector Λ࢖͏
28. if (!$dbh->ping) { $dbh->disconnect; $dbh = DBI->connect($dsn,...); } RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ
29. ElastiCache for Redis ͷݕূ • RDS ͱҧͬͯɺreboot with failover ػೳ͕࢖ ͑ͳ͍ • 30 ~ 60sec ͔͔ΔॲཧΛͤ͞Δͱ Failover ͢Δ͜ͱ͕͋Δ • iptables ͰύέοτΛམͱ͢ • ࠓͷͱ͜Ζਖ਼֬ʹςετ͢Δ͜ͱ͕Ͱ͖ͳ͍
30. ElastiCache Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆ • timeout ͯ͠΋ DNS ͸Ҿ͖௚͞ͳ͍ • ping Ͱ֬ೝͯ͠ɺͩΊͳΒ࠶઀ଓ
31. if (!$redis->ping) { $redis->quit; $redis = Redis::Fast->new(...); } ElastiCache Ͱਖ਼ৗʹ Failover ͤ͞Δ
32. Cpanfile ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • ؀ڥߏஙΛ༰қʹߦ͑ΔΑ͏ʹ • ࠷৽൛ͷ cpan module ΛೖΕͯςετ • ໰୊ͳ͔ͬͨͷͰ࠷৽൛ʹ • DateTime ͚ͩ͏Δ͏ඵૠೖ·͑ʹόʔδϣ ϯμ΢ϯ • :60 Λࠩ͠ࠐΜͰ͘Δ...
33. Mozilla::CA ͷ໰୊ • Mozilla::CA::SSL_ca_file() ͕ฦ͢ূ໌ॻ͕ݹ ͍ • ΞϓϦ಺՝ۚͷϨγʔτݕূͰΤϥʔ͕ग़ ΔΑ͏ʹ... • ෇ଐͷ mk-ca-bundle.pl Λ࢖༻
34. $ ./perl5/lib/perl5/Mozilla/mk-ca-bundle.pl --help ... -p list of Mozilla trust purposes and levels for certificates to include in output. Takes the form of a comma separated list of purposes, a colon, and a comma separated list of levels. (default: SERVER_AUTH:TRUSTED_DELEGATOR) Valid purposes are: ALL, DIGITAL_SIGNATURE, NON_REPUDIATION, KEY_ENCIPHERMENT, DATA_ENCIPHERMENT, KEY_AGREEMENT, KEY_CERT_SIGN, CRL_SIGN, SERVER_AUTH, CLIENT_AUTH, CODE_SIGNING, EMAIL_PROTECTION, IPSEC_END_SYSTEM, IPSEC_TUNNEL, IPSEC_USER, TIME_STAMPING, STEP_UP_APPROVED Valid levels are: ALL, TRUSTED_DELEGATOR, NOT_TRUSTED, MUST_VERIFY_TRUST, TRUSTED Mozilla::CA ͷ໰୊
35. Mozilla::CA ͷ໰୊ • -p PURPOSES:LEVELS Ͱ໨తͱϨϕϧ͕ࢦఆ Ͱ͖Δ • -p ALL:ALL ͰશࢦఆՄ • ੜ੒ͨ͠ূ໌ॻΛࢦఆ͢ΔΑ͏ʹมߋ
36. my %args = ( ssl_opts => { SSL_ca_file => '/path/to/ca-bundle.crt', }, ... ); my $furl = Furl->new(%args); ... Mozilla::CA ͷ໰୊
37. ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λେ෯ʹมߋ • web αʔόͱ app αʔόͷ౷߹ • ࣗલͰ MySQL ͱ Redis Λཱͯͳ͍ • RDS ͱ ElastiCache
38. -# /HJOY "QQ 3FEJT
    NBTUFS 3FEJT
    TMBWF #BUDI .Z42-
    NBTUFS .Z42-
    TMBWF %FQMPZ STZOD
    "SDIFS 7*1 DIFGTPMP
    DBQJTUSBOP
39. ຊ൪αʔόߏங • consul ͷಋೖ • ಺෦ DNS • stretcher ͷಋೖ • Fluentd ͷઃఆݟ௚͠
40. Consul • Hashicorp ੡ͷΦʔέετϨʔγϣϯπʔϧ • KVSɺDNSɺHTTP ΠϯλϑΣʔε • ಺෦ DNS ͱͯ͠׆༻
41. Consul DNS • 8600 port Ͱఏڙ • node ͱ service ʹରԠ • nodename.node(.datacenter).consul • servicename.service(.datacenter).consul
42. dnsmasq • DNS forwarder, cache • consul ͷ 8600 port ʹ forward • શαʔόͰՔಇ
43. $ cat /etc/dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1 dnsmasq
44. VPC DHCP Options Set • VPC ͷ DHCP ͷઃఆΛߦ͏ • resolv.conf ͕ੜ੒͞ΕΔ • ݱࡏ͸ Management Console ͔Β΋ઃఆՄೳ
45. $ cat /etc/resolv.conf ; generated by /sbin/dhclient-script search node.consul service.consul nameserver 127.0.0.1 nameserver 10.1.0.2 ; VPC ͷ DNS nameserver 10.1.11.100 ; Unbound VPC DHCP Options Set
46. Unbound • DNS cache, resolver • VPC ͷ DNS ΛΩϟογϡ • VPC ͷ resolver ʹ໰୊͕͋ͬͯ΋໊લղܾ ͕Ͱ͖ΔΑ͏ʹ • deploy αʔόͰՔಇ
47. stretcher ʹΑΔ deploy • https://github.com/fujiwara/stretcher • Go ੡ • S3 ͔Β tarball Λऔಘͯ͠ల։ɺrsync • consulɺserf ͱ࿈ܞ • ৄ͘͠͸ɺhttps://speakerdeck.com/fujiwara3/ consultozi-zuo-osswohuo-yong-sita100tai-gui-mo- falsewebsabisuyun-yong
48. &-# /HJOY
    "QQ &MBTUJ$BDIF
    NBTUFS &MBTUJ$BDIF
    TMBWF #BUDI 3%4
    NBTUFS 3%4
    TMBWF
    CBDLVQ %FQMPZ DPOTVM
    FWFOU #VJME
    6QMPBE 4 DIFGTPMP
    DBQJTUSBOP
49. ϝϯςφϯε (ࣄલ४උ) • ηΧϯμϦΠϯσοΫεΛ࡟আ͓͍ͯͨ͠ςʔ ϒϧΛҠߦઌͷ DB ʹ༻ҙ • RDS Λ Single-AZ ʹ͢Δ • ύϥϝʔλάϧʔϓͷௐ੔
50. ϝϯςφϯε (ࣄલ४උ) • RDS ΛSingle-AZ ʹ͢Δ • Multi-AZ ͷ৔߹ɺ଱ো֐ੑ͸޲্͢Δ͕ॻ ͖ࠐΈ଎౓͕2ഒʹ
51. ϝϯςφϯε (ࣄલ४උ) • ύϥϝʔλάϧʔϓͷมߋ • innodb_flush_log_at_trx_commit=0 • innodb_support_xa=0 • sync_binlog=0
52. ϝϯςφϯε • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Redis ͷσʔλҠߦ • ಈ࡞֬ೝ • DNS੾Γସ͑ • Multi-AZ Խ & ύϥϝʔλάϧʔϓมߋ
53. MySQL ͷσʔλҠߦ • εΩʔϚ͸༧Ί৽ DB ʹೖΕͯ͋ΔͷͰɺσʔ λ͚ͩ dump (& gzip) • FOREIGN_KEY_CHECKS=0 ͱ 1 Λ zcat ͷલ ޙʹೖΕͯ mysql ʹ౉͢
54. $ { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo "FOREIGN_KEY_CHECKS=1"; } | mysql -u ... MySQL ͷσʔλҠߦ
55. MySQL ͷσʔλҠߦ • import ͕ऴΘͬͨΒηΧϯμϦΠϯσοΫε Λ࡞੒͢Δ • information_schema.tables Λݟ֤ͯςʔϒϧ ͷߦ਺Λൺֱ
56. mysql -u USER -B -N -e "SELECT TABLE_NAME, TABLE_ROWS FROM information_schema.tables WHERE TABLE_SCHEMA = 'my_db' ORDER BY TABLE_NAME" MySQL ͷσʔλҠߦ
57. Redis ͷσʔλҠߦ • redis-cli CONFIG SET appendonly yes • appendfsync no ʹ͢Δͱߴ଎(fsync͠ͳ͍) • redis-cli BGSAVE • Redis Λఀࢭ • cat appendonly.aof | redis-cli --pipe -h ...
58. Redis ͷσʔλҠߦ • Ҡߦલޙͷ྆؀ڥͰ KEYS “*” ͷ݁ՌΛϑΝΠ ϧʹམͱ͢ • ݁ՌΛ sort • diff Λऔͬͯ֬ೝ • expireͰ࡟আ͞Ε͍ͯΔͱଟগࠩ෼͕ग़Δ
59. ಈ࡞֬ೝ • Ҡߦޙͷ؀ڥʹ઀ଓͰ͖ΔΞϓϦΛ୺຤ʹೖ Ε͓ͯ͘ • υϝΠϯΛઃఆ • εϚʔτϑΥϯͷ DNS ઃఆΛมߋͨ͠Γɺ DNS ͷԠ౴Λِ૷͢ΔΑΓָ
60. Single-AZ -> Multi-AZ ύϥϝʔλάϧʔϓͷมߋ • Multi-AZ ʹมߋ • Single-AZ ͔Β Multi-AZ ΁ͷมߋ͸ΦϯϥΠϯͰ ࣮ߦՄೳ • ύϥϝʔλάϧʔϓͷมߋ • innodb_flush_log_at_trx_commit=1 • innodb_support_xa=1 • sync_binlog=1
61. DNS ͷ੾Γସ͑ • TTL Λ༧Ί୹͓ͯ͘͘͠ • TTL Λ୹ͯ͘͠΋چ؀ڥʹΞΫηε͕དྷΔՄ ೳੑ • DB ΍ KVS ͸ඞͣࢭΊ͓ͯ͘
62. ࣄྫ 2,3
63. ࣄྫ 2,3 • ๻ΒͷߕࢠԂʂ೤ಆฤͷ Mobage ൛ͱ iOS / Android ൛ • Ҡߦظؒ߹Θͤͯ 1 ϲ݄ • ΄ͱΜͲαʔόߏ੒͕ಉ͡ͳͷͰಉ࣌ਐߦ • perlbrew ͰΠϯετʔϧͨ͠ perl ؀ڥΛͦͷ·· ͍࣋ͬͯ͘ • SL6 -> CentOS6 ͳͷͰ
64. ࣄྫ 2,3 • Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ ֤ 1 ໊ • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊ • Φϒβʔόʔ 2 ໊
65. Ҡߦ࡞ۀ • dev αʔόߏங • (RDS for MySQL ͷݕূ) • yrmcds ͷݕূ • ຊ൪αʔόߏங • ϝϯςφϯε • Ҡߦޙʹൃੜͨ͠໰୊
66. dev αʔόߏங • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook Խ
67. yrmcds ͷݕূ • session ʹ Kyoto Tycoon Λ࢖༻ • ػೳతʹ໰୊͸ͳ͍ • ٕज़ݕূ΋݉Ͷͯผͷϛυϧ΢ΣΞΛ࢖͏ • Redis ͩͱΞϓϦͷେ෯ͳमਖ਼͕ඞཁ • Memcached ޓ׵ͷ΋ͷ͕ྑ͍ • yrmcds
68. yrmcds • https://github.com/cybozu/yrmcds • Memcached ΄΅׬શޓ׵ • master / slave ϨϓϦέʔγϣϯ • VIP ͕෇͍͍ͯΔαʔό͕ master • ϝϞϦ؅ཧʹ Slab Λ࢖༻͍ͯ͠ͳ͍ • αʔόαΠυϩοΫ
69. EC2 ʹ VIP Λ͚ͭΔ • EC2 ʹ VIP Λ͚ͭͯ΋௨৴Ͱ͖ͳ͍ • ENI ͸σϑΥϧτͰૹ৴ݩ/ૹ৴ઌνΣοΫ ͕༗ޮʹͳ͍ͬͯΔ • ແޮʹ͠ͳ͍ͱࣗ෼ѼͯҎ֎ͷύέοτ Λड͚औΕͳ͍
70. Keepalived • VIP ͷ੾Γସ͑ʹ࢖༻ • VPC Ͱ͸ϚϧνΩϟετ͕࢖͑ͳ͍ • ؤுΕ͹࢖͑ΔΑ͏Ͱ͕͢ɺKeepalived ͷՄ ༻ੑ΋ߟ͑ͳͯ͘͸ͳΒͳ͍ • Consul lock Λ׆༻ͨ͠ Failover γεςϜ
71. Consul lock • ෼ࢄϩοΫػೳ • consul lock service/lock deploy.sh • ෼ࢄηϚϑΥػೳ • consul lock -n 2 service/lock deploy.sh
72. Consul lock Λ׆༻ͨ͠ Failover γεςϜ • lock Λऔͬͨϗετʹ VIP Λ͚ͭΔ • ϩʔΧϧͷ yrmcds ͷࢮ׆؂ࢹ • μ΢ϯͨ͠Β lock Λղ์͢Δ • Slave Ͱ΋ lock ΛऔΓʹߦͬͯ lock ղ์଴ͪͷঢ়ଶʹ ͢Δ • yrmcds ͕μ΢ϯ͢Δͱ Slave ͷ͏ͪҰ͕ͭ lock Λऔͬ ͯ VIP Λ͚ͭΔ(master ʹঢ֨)
73. Consul lock Λ׆༻ͨ͠ Failover γεςϜ • VPC Route Table ͷॻ͖׵͑ • yrmcds Λ svc -o Ͱىಈ • উखʹ࠶ىಈ͠ͳ͍Α͏ʹ • /service/yrmcds ΋ىಈ࣌ʹ࡟আ • αʔό͕ಥવࢮͯ͠ VIP ͕͍ͭͨ··࠶ىಈ ͯ͠΋ master ʹͳΒͳ͍Α͏ʹ
74. ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λมߋ • web αʔόͱ app αʔόͷ౷߹ • Kyoto Tycoon Λ yrmcds ʹҠߦ
75. -# /HJOY "QQ ,ZPUP
    5ZDPPO
    TFTTJPO #BUDI .Z42-
    NBTUFS .Z42-
    TMBWF %FQMPZ STZOD
    "SDIFS DIFGTPMP
    DBQJTUSBOP ,ZPUP
    5ZDPPO
    TFTTJPO CBDLVQ )"1SPYZ .FNDBDIFE
    DBDIF )"1SPYZ )"1SPYZ
76. ຊ൪αʔόߏங • MySQL Λ Slave ࢀর͠ͳͯ͘΋໰୊ͳ͘ͳͬ ͨͷͰ HAProxy ͕ෆཁʹ • Consul + dnsmasq ಋೖ • ಺෦DNS
77. &-# ZSNDET
    NBTUFS ZSNDET
    TMBWF #BUDI %FQMPZ STZOD
    "SDIFS 7*1 DIFGTPMP
    DBQJTUSBOP 3%4
    NBTUFS 3%4
    TMBWF
    CBDLVQ /HJOY
    "QQ
78. Ҡߦ࡞ۀ • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Session ͷσʔλҠߦ • (ಈ࡞֬ೝ) • (DNS੾Γସ͑)
79. MySQL ͷσʔλҠߦ • MySQL ͷσʔλҠߦΛ࠷଎खॱͰ࣮ࢪ • Duplicate entry ͕ग़ࣦͯഊ... • લ೔ͷԋशͰ͸੒ޭ • ະͩʹݪҼෆ໌ • ֎෦Ωʔ੍໿Λ΄ͱΜͲ࢖͍ͬͯͳ͍ͨΊ௨ ৗͷ mysqldump Ͱ΋ 5 ෼͔͕ࠩ͠ͳ͔ͬͨ
80. Session ͷσʔλҠߦ • ͠·ͤΜͰͨ͠ • ϩάΠϯ͠௚͠ • Ϣʔβ਺తʹ໰୊ͳ͠ • ϩάΠϯ͕ूத͢Δͱࡹ͖੾Εͳ͍৔߹΋ ͋ΔͷͰ௨ৗ͸Ҡߦͨ͠΄͏͕ྑ͍
81. Ҡߦޙʹൃੜͨ͠໰୊ • Consul ͸਺ඵ Leader lost ͢Δ͜ͱ͕͋Δ • Leader lost ͢Δͱ lock ͕֎ΕΔ... • Ҡߦޙ 2 ೔࿈ଓͰ Failover • dns_config.node_ttl, allow_stale, max_stale Λઃఆ͢Δ • Leader lost ࣌΋਺ඵͳΒ DNS ͕Ҿ͚Δ
82. ؂ࢹʹ ͍ͭͯ
83. Zabbix • ࣗࣾαʔϏεͷ؂ࢹ • VPC Peering Connection Ͱ internal ௨৴ • Ұ෦ Zabbix Proxy Λར༻
84. Zabbix Proxy • Zabbix Server ͷ୅ΘΓͱͯ͠ಈ࡞ • ෛՙͷܰݮ • ऩूσʔλΛ֤ϗετͰόοϑΝϦϯά • Firewall ͷઃఆ͕༰қ • ֎෦νΣοΫ • ίϚϯυΛ࣮ߦͯ͠ MySQL ͷ QPS Λܭଌ
85. Zabbix Proxy • ศར͕ͩ SPOF ʹͳΔ • ΠϯλʔωοτΛܦ༝ͤͣ؂ࢹ͕Մೳʹͳͬ ͨͷͰ௚઀؂ࢹʹมߋ • Active Agent Check • ผͷ໰୊͕...
86. Zabbix Active Check • ֎෦νΣοΫͷεΫϦϓτʹύεϫʔυΛॻ ͍͍ͯΔ • Zabbix Proxy ͕ೖ͍ͬͯΔϗετʹઃఆ • ௚઀؂ࢹʹͳΓ࢖͑ͳ͘ͳΔ • ؂ࢹର৅͕૿͑Δͨͼʹ Zabbix Server ͷઃ ఆมߋΛߦ͏ͷ͸ආ͚͍ͨ
87. Zabbix Active Check • UserParameter Λઃఆͨ͠ϗετΛొ࿥ • RDS ΍ ElastiCache ͳͲͷ؂ࢹ͕Մೳʹ • ςϯϓϨʔτʹొ࿥ͨ͠ΞΠςϜΛϗετϚ ΫϩͰมߋՄೳʹ • ࠶ར༻͠΍͍͢
90. PUSH ௨஌༻ SSL ূ໌ॻͷ؂ࢹ • PUSH ௨஌ʹ࢖͏ SSL ূ໌ॻͷظݶΛ؂ࢹ • ূ໌ॻͷ PATH ͸αʔό͝ͱʹҧ͏ • ϓϩδΣΫτ͝ͱʹΞΠςϜΛ௥Ճ͢Δͷ͸ େม • Low Level Discovery(LLD) Ͱղܾ
91. Low Level Discovery(LLD) • ΞΠςϜɺτϦΨɺάϥϑΛࣗಈͰੜ੒͢Δ ͨΊͷػೳ • ϑΝΠϧγεςϜɾωοτϫʔΫΠϯλ ϑΣʔεݕग़༻ͷLLD͕૊Έࠐ·Εͨςϯϓ Ϩʔτ΋͋Δ • LLD Λࣗ࡞͢Δ͜ͱ΋Մೳ
92. Low Level Discovery(LLD) • UserParameter ʹ ΞΠςϜऩू༻ͷઃఆ௥Ճ • ΞΠςϜऩू͸ɺܾΊΒΕͨܗࣜͷ JSON σʔλΛฦ͢
93. #!/bin/bash CERTS="${1}" CERTS_DATA="" for CERT in ${CERTS}; do CERTS_DATA="${CERTS_DATA},{\"{#CERT_PATH}\":\"${CERT}\"}" done echo "{\"data\":[$(echo ${CERTS_DATA} | sed -e 's/^,//')]}" Low Level Discovery(LLD)
94. { "data": [ {"{#CERT_PATH}":"/path/to/cert1"}, {"{#CERT_PATH}":"/path/to/cert2"} ] } Low Level Discovery(LLD)
95. #!/bin/bash CERT="${1}" NOW=$(date +%s) EXPIRE=$(date -d "$(openssl x509 -in ${CERT} -noout -dates | tail -n 1 | sed -e 's/notAfter=//')" +%s) echo $(((${EXPIRE} - ${NOW}) / 60 / 60 / 24)) Low Level Discovery(LLD)
98. εϖʔε۠੾ΓͰূ໌ॻΛࢦఆ
99. RDS ͷ؂ࢹ • fluent-plugin-cloudwatch ͰϝτϦΫεΛऩू • fluent-plugin-zabbix Ͱ Zabbix ʹૹ৴ • MySQL ͷ stats ͸ SQL Λ࣮ߦ • SHOW ENGINE INNODB STATUS • SHOW GLOBAL STATUS
100. ElastiCache ͷ؂ࢹ • redis-cli INFO ͷ stats Λऩू • DNS ΛҾ͍ͯ Primary Cache Cluster ͷ IP Λ؂ࢹ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover • UserParameter ͰίϚϯυ࣮ߦ • net.dns.record ͸ DNS ͕Ҿ͚ͳ͍࣌͸Ξϥʔ τΛඈ͹ͤͳ͍...
101. #!/bin/bash host="/usr/bin/host" DOMAIN=${1} TIMEOUT=10 RECORDS=$($host -s -W $TIMEOUT $DOMAIN | awk 'NR > 1 {print $NF}' | sort) [ "${RECORDS}" = "" ] && echo "fail" && exit 1 echo $RECORDS ElastiCache ͷ؂ࢹ
102. yrmcds ͷ؂ࢹ • stats ͰϝτϦΫεΛऔಘՄೳ • https://github.com/cybozu/yrmcds/issues/47 ͰରԠ͍͖ͯͨͩ͠·ͨ͠ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • VIP ͕෇͍͍ͯΔΠϯελϯεͷ Private IP Λ νΣοΫ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover ͱͯ͠ݕग़
103. Jenkins Slave ͷ׆༻
104. Jenkins EC2 Plugin • Jenkins Slave Λ AMI ͔Βىಈ • ςετ͕࣮ߦ͞Εͳ͘ͳ͔ͬͯΒ 30෼͘Β ͍Ͱ Terminate • ىಈ͕࣌ؒ୹͘ͳΔͷͰ҆ՁʹͳΔ • Spot Instance ΋࢖͑ΔͷͰɺΑΓ҆Ձʹ
105. Jenkins EC2 Plugin • 32 core ͷ CPU ͰίΞ਺෼·Ͱฒྻ਺Λ͋͛ ͯςετΛճ͢ͱ͚͜ΔΑ͏ʹ... • MySQL ͕ EAGAIN ΤϥʔΛग़͢Α͏ʹ • Test::mysqld Ͱෳ਺ݸͷ MySQL Λ্ཱͪ ͍͛ͯΔ • fs.aio-max-nr=65536 Λ 4 ഒʹͯ͠ճආ
106. Jenkins EC2 Plugin • ଞͷΧʔωϧύϥϝʔλ΋νϡʔχϯά • ςετ͸ߴ଎ԽͰ͖ͣ... • MySQL ͷ্ཱͪ͛ʹ͕͔͔࣌ؒΔ • Test::mysqld::Multi • ฒྻ਺ΛNഒͯ͠΋࣮ߦ࣌ؒ͸1/NʹͳΒͳ͍ • CPUΛ࢖͍੾Εͳ͍
107. AWS Ҡߦͷ ϝϦοτɾ σϝϦοτ
108. AWS Ҡߦͷֹۚ໘Ͱͷ ϝϦοτ • ϓϩδΣΫτ͝ͱʹਖ਼֬ͳར༻ֹ͕ग़ͤΔ • ΦϯϓϨͰ͸ɺαʔό(VM)͕༨ͬͨ৔߹શ ࣾඅ༻ʹճ͍ͯͨ͠ • ݟ্͔͚͸҆ՁʹͳΔ͕ɺશࣾతʹ͸අ༻ ͕͔͔͍ͬͯΔ͜ͱΛཧղ͞Εͳ͍
109. AWS Ҡߦͷֹۚ໘Ͱͷ σϝϦοτ • අ༻͸ଟগ૿Ճ͢Δ • େن໛Ͱͳ͚Ε͹ਓ݅අΛؚΊΔͱ҆Ձ(ͳ ͸ͣ) • ࣗલͰαʔόӡ༻͢ΔΑΓָ • దਖ਼ͳඅ༻͕ܭ্͞ΕΔͨΊɺඅ༻͕૿Ճ͠ ͨΑ͏ʹࢥΘΕͯ͠·͏
110. AWS Ҡߦͷػೳ໘Ͱͷ ϝϦοτ • ϚωʔδυαʔϏεͰӡ༻ෛՙͷ௿ݮ • ҆Ձʹαʔόͷަ׵͕Մೳ • ϋʔυ΢ΣΞੑೳͷ޲্ • ΞϓϦͷνϡʔχϯάͳ͠ͰϨεϙϯελ ΠϜվળ
112. AWS Ҡߦͷػೳ໘Ͱͷ σϝϦοτ • ඞཁͳػೳ͕ͳ͍͜ͱ͕͋Δ • ElastiCache ͷ reboot with failover • RDS ͷ λΠϜκʔϯରԠ
113. ஌ݟ
114. ஌ݟ • ҆ఆӡ༻Ͱ͖͍ͯΔͱݱঢ়ҡ࣋ʹͳΓ͕ͪ • αʔόߏ੒ͷݟ௚͕͠Ͱ͖Δνϟϯε • ྑ͘ͳ͔ͬͨ෦෼͕ු͖ூΓʹͳΔ • ؂ࢹ͢΂͖͜ͱ͕Ͱ͖͍ͯͳ͍ • Chef Ͱ؅ཧͰ͖͍ͯΔ΋ͷͱͰ͖͍ͯͳ͍ ΋ͷ͕ࠞࡏ͍ͯ͠Δ
115. ஌ݟ • े਺୆ن໛ͷγεςϜͰ͋Ε͹ 2~3 ໊ఔ౓Ͱ 1 ϲ݄͋Ε͹ҠߦͰ͖Δ • ҙ֎ͱԿͱ͔ͳΔ • ϝϯςφϯε೔ͷௐ੔͕ҙ֎ͱେม
116. ஌ݟ • ࣄલԋशΛԿ౓΋΍Δͱ҆৺ • MySQL ͷσʔλҠߦ͸ݕূ࣌ʹ3ɾ4ճɺલ ೔ʹ 2 ճ࣮ࢪ • ίϚϯυΛίϐϖ࣮ͯ͠ߦ͢Δ͚ͩͰྑ͍ ঢ়ଶʹ͓ͯ͘͠ͱߋʹ҆৺
117. ·ͱΊ • AWS ͷҠߦࣄྫ঺հ • ݱঢ়ҡ࣋ΑΓ΋վળ • ߏ੒Λݟ௚͢ྑ͍ػձ • ػձ͕ͳ͍ͱվળ͠ʹ͍͘