ソーシャルゲームにおける AWS 移行事例

by tkuchiki

Slide 1

Slide 1 text

ιʔγϟϧήʔϜʹ͓͚Δ AWSҠߦࣄྫ YAPC::Asia TOKYO 2015 @tkuchiki

Slide 2

Slide 2 text

WHOAMI(1) • @tkuchiki • KAYAC Inc. 4೥໨ • ΦϖϨʔγϣϯΤϯδχΞ • ओʹιʔγϟϧήʔϜ୲౰ • perl ͸ΞϓϦέʔγϣϯΤϯδχΞͩͬͨ ࠒʹ 3ϲ݄΄Ͳ...

Slide 3

Slide 3 text

Agenda • AWS Ҡߦͷഎܠ • ࣄྫ 1 • ࣄྫ 2, 3 • ؂ࢹʹ͍ͭͯ • Jenkins Slave ͷ׆༻ • AWS ҠߦͷϝϦοτɾσϝϦοτ • ஌ݟ

Slide 4

Slide 4 text

AWS Ҡߦͷഎܠ • 2011 ೥͔Β DC Ͱͷӡ༻։࢝ • 5 ೥ϦʔεͰαʔόػߪೖ • Ϧʔεظؒऴྃޙͷ͜ͱΛߟ͑Δඞཁੑ • 2014 ೥Ҏ߱ʹϦϦʔεͨ͠αʔϏε͸ɺࠓޙ ͷ͜ͱΛݟਾ͑ͯ AWS ʹߏங

Slide 5

Slide 5 text

AWS Ҡߦͷഎܠ • ࣾ಺࠷େن໛ͷαʔϏεͷ AWS Ҡߦͱ਺ݸͷ αʔϏεऴྃ • ࢖༻͠ͳ͍αʔό͕૿Ճ • 2014 ೥ 3 ݄ʹϥοΫ࡟ݮ • ࠓޙ΋༨৒αʔό͕૿Ճ͢ΔՄೳੑΛߟ͑ͯ ࠷దԽ͠΍͍͢ AWS ʹશ໘Ҡߦ͢Δ͜ͱʹ

Slide 6

Slide 6 text

ࣄྫ 1

Slide 7

Slide 7 text

• ๯ݥΫΠζΩϯάμϜ • Ҡߦظؒ 1 ϲ݄ • Ҡߦ࣌ʹ perl ͷϚΠφʔόʔδϣϯΞοϓ • cpan module ͷόʔδϣϯ΋ • CentOS 6ΛϚΠφʔόʔδϣϯΞοϓ ࣄྫ 1

Slide 8

Slide 8 text

• Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ 2 ໊ • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊ • Φϒβʔόʔ 2 ໊ ࣄྫ 1

Slide 9

Slide 9 text

Ҡߦ࡞ۀ • dev αʔόߏங • RDS for MySQL ͷݕূ • ElastiCache for Redis ͷݕূ • Cpanﬁle ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • Mozilla::CA ͷ໰୊ • ຊ൪αʔόߏங • ϝϯςφϯε

Slide 10

Slide 10 text

dev αʔόߏங • dev αʔό͕ 2 ୆͋ͬͨ • ৑௕ߏ੒Ͱ͸ͳ͍ • ؅ཧ͕େมʹͳΔ͚ͩͳͷͰ1୆ʹ·ͱΊΔ • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook Խ

Slide 11

Slide 11 text

RDS for MySQL ͷݕূ • RDS ͸λΠϜκʔϯͷઃఆ͕Ͱ͖ͳ͍ͷͰ઀ ଓ࣌ʹઃఆมߋ͢Δ • MySQL ͷ৔߹ɺinit_connect • SET SESSION time_zone = CASE WHEN POSITION('rds' IN CURRENT_USER()) = 1 THEN 'UTC' ELSE '+9:00' END;

Slide 12

Slide 12 text

RDS for MySQL ͷݕূ • Failover ͯ͠΋໰୊ͳ͍͔ • init_connect ͰΤϥʔ͕ͰΔ৔߹͕͋Δͱ ͍͏৘ใ͕... • reboot with failover ػೳΛ࢖༻ • MySQL ͷ reconnect Λ༗ޮʹ͠ͳ͍

Slide 13

Slide 13 text

ΦϯϓϨ MySQL -> RDS • ΦϯϓϨ - RDS Ͱ Replication ͕૊ΊΔ • RDS ͸λΠϜκʔϯ͕มߋͰ͖ͳ͍ • UTC ͷΈ • JST Ͱӡ༻͍ͯ͠Δ৔߹ 9 ࣌ؒͣΕΔ • ࣌ࠁؔ਺Λ࢖͍ͬͯͳ͍৔߹͸໰୊ͳ͍

Slide 14

Slide 14 text

MySQL ͷσʔλαΠζ • MySQL 5.5 • mysqldump(raw): 14.92GB • mysqldump(zgip): 2.79GB

Slide 15

Slide 15 text

$ time zcat dump.sql.gz | mysql -u ... real 209m18.642s user 6m42.994s sys 0m16.495s MySQL ͷΠϯϙʔτ

Slide 16

Slide 16 text

209෼…

Slide 17

Slide 17 text

Ͳ͏ʹ͔ͯ͠ ߴ଎Խ͍ͨ͠

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

खॱ͕·ͱΊ ΒΕ͍ͯͨ

Slide 20

Slide 20 text

MySQL5.5 ࠷଎σʔλҠߦखॱ • ςʔϒϧ࡞੒ • ϓϥΠϚϦΠϯσοΫε࡞੒ • ֎෦Ωʔ੍໿෇༩ • ֎෦Ωʔ੍໿ແޮԽ • σʔλϩʔυ • ֎෦Ωʔ੍໿༗ޮԽ • ηΧϯμϦΠϯσοΫε࡞੒ http://dbstudy.info/ﬁles/20131007/mysql56_load_r2.pdf

Slide 21

Slide 21 text

mysqldump • ηΧϯμϦΠϯσοΫεൈ͖ͰdumpͰ͖ͳ͍ • --disable-keys(-K) Λ࢖͏ͱߦ͕͢΂ͯૠೖ͞ Εͨ͋ͱʹΠϯσοΫε࡞੒ • INSERT จ͕ ALTER TABLE (DISABLE| ENABLE) KEYS ͰғΘΕΔ • MyISAM ͷΈ...

Slide 22

Slide 22 text

mysqldumpindex • https://github.com/tkuchiki/mysqldumpindex • SHOW TABLES ͱ SHOW INDEXES FROM ͷ݁Ռ ͔ΒɺηΧϯμϦΠϯσοΫεͷ࡞੒ɺ࡟আ SQL Λ࡞੒͢Δ • mysqldump --no-data Ͱ dump ͨ͠ SQL ͱɺη ΧϯμϦΠϯσοΫε࡟আɾ࡞੒ޙʹ dump ͨ͠ SQL ͷ diff ΛͱͬͯҰக͢Δ͜ͱΛ֬ೝ

Slide 23

Slide 23 text

# σʔλΠϯϙʔτ $ time { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo "FOREIGN_KEY_CHECKS=1"; } | mysql -u ... real 54m52.894s user 6m43.438s sys 0m15.711s # ηΧϯμϦΠϯσοΫε࡞੒ $ time mysql -u ... < create_index.sql real 38m26.767s user 0m0.010s sys 0m0.018s ࠷଎खॱͰͷσʔλҠߦ࣌ؒ 209 ෼͔Β 93෼ʹ୹ॖʂ

Slide 24

Slide 24 text

mysqldump(percona-server) • --innodb-optimize-keys • CREATE TABLE จ͔Β UNIQUE ͱ INDEX ΛऔΓআ͘ • INSERT จͷޙΖʹ ALTER TABLE ADD [UNIQUE] KEY Λ௥Ճ

Slide 25

Slide 25 text

mysqldump(percona-server) • --innodb-optimize-keys Λ࢖͏͚ͩͰ࠷଎ͷ खॱʹͳΓͦ͏ • ͱࢥ࣮ͬͯݧͯ͠Έ·͕ͨ͠ૣ͘ͳΒͣ

Slide 26

Slide 26 text

RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • dsn ʹ timeout Λઃఆ • mysql_connect_timeout • mysql_read_timeout • mysql_write_timeout • ઃఆ͠ͳ͍ͱϓϩηε͕࠶ੜ੒͞Εͳ͍ͱ࠶઀ଓ ͠ͳ͍ • DNS ΛҾ͖௚͞ͳ͍

Slide 27

Slide 27 text

RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆͯ͠΋ͩΊͳ৔߹͕͋Δ • ωοτϫʔΫஅ • MySQL server has gone away • Ұ౓ࣦഊ͢Δͱϓϩηε͕࠶ੜ੒͞ΕΔ·Ͱܨ ͕Βͳ͍... • ping Ͱܨ͕Δ͔֬ೝ͢Δ͔ɺDBIx::Connector Λ࢖͏

Slide 28

Slide 28 text

if (!$dbh->ping) { $dbh->disconnect; $dbh = DBI->connect($dsn,...); } RDS Ͱਖ਼ৗʹ Failover ͤ͞Δ

Slide 29

Slide 29 text

ElastiCache for Redis ͷݕূ • RDS ͱҧͬͯɺreboot with failover ػೳ͕࢖ ͑ͳ͍ • 30 ~ 60sec ͔͔ΔॲཧΛͤ͞Δͱ Failover ͢Δ͜ͱ͕͋Δ • iptables ͰύέοτΛམͱ͢ • ࠓͷͱ͜Ζਖ਼֬ʹςετ͢Δ͜ͱ͕Ͱ͖ͳ͍

Slide 30

Slide 30 text

ElastiCache Ͱਖ਼ৗʹ Failover ͤ͞Δ • timeout Λઃఆ • timeout ͯ͠΋ DNS ͸Ҿ͖௚͞ͳ͍ • ping Ͱ֬ೝͯ͠ɺͩΊͳΒ࠶઀ଓ

Slide 31

Slide 31 text

if (!$redis->ping) { $redis->quit; $redis = Redis::Fast->new(...); } ElastiCache Ͱਖ਼ৗʹ Failover ͤ͞Δ

Slide 32

Slide 32 text

Cpanﬁle ࡞੒ / cpan module ͷόʔδϣϯΞοϓ • ؀ڥߏஙΛ༰қʹߦ͑ΔΑ͏ʹ • ࠷৽൛ͷ cpan module ΛೖΕͯςετ • ໰୊ͳ͔ͬͨͷͰ࠷৽൛ʹ • DateTime ͚ͩ͏Δ͏ඵૠೖ·͑ʹόʔδϣ ϯμ΢ϯ • :60 Λࠩ͠ࠐΜͰ͘Δ...

Slide 33

Slide 33 text

Mozilla::CA ͷ໰୊ • Mozilla::CA::SSL_ca_ﬁle() ͕ฦ͢ূ໌ॻ͕ݹ ͍ • ΞϓϦ಺՝ۚͷϨγʔτݕূͰΤϥʔ͕ग़ ΔΑ͏ʹ... • ෇ଐͷ mk-ca-bundle.pl Λ࢖༻

Slide 34

Slide 34 text

$ ./perl5/lib/perl5/Mozilla/mk-ca-bundle.pl --help ... -p list of Mozilla trust purposes and levels for certificates to include in output. Takes the form of a comma separated list of purposes, a colon, and a comma separated list of levels. (default: SERVER_AUTH:TRUSTED_DELEGATOR) Valid purposes are: ALL, DIGITAL_SIGNATURE, NON_REPUDIATION, KEY_ENCIPHERMENT, DATA_ENCIPHERMENT, KEY_AGREEMENT, KEY_CERT_SIGN, CRL_SIGN, SERVER_AUTH, CLIENT_AUTH, CODE_SIGNING, EMAIL_PROTECTION, IPSEC_END_SYSTEM, IPSEC_TUNNEL, IPSEC_USER, TIME_STAMPING, STEP_UP_APPROVED Valid levels are: ALL, TRUSTED_DELEGATOR, NOT_TRUSTED, MUST_VERIFY_TRUST, TRUSTED Mozilla::CA ͷ໰୊

Slide 35

Slide 35 text

Mozilla::CA ͷ໰୊ • -p PURPOSES:LEVELS Ͱ໨తͱϨϕϧ͕ࢦఆ Ͱ͖Δ • -p ALL:ALL ͰશࢦఆՄ • ੜ੒ͨ͠ূ໌ॻΛࢦఆ͢ΔΑ͏ʹมߋ

Slide 36

Slide 36 text

my %args = ( ssl_opts => { SSL_ca_file => '/path/to/ca-bundle.crt', }, ... ); my $furl = Furl->new(%args); ... Mozilla::CA ͷ໰୊

Slide 37

Slide 37 text

ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λେ෯ʹมߋ • web αʔόͱ app αʔόͷ౷߹ • ࣗલͰ MySQL ͱ Redis Λཱͯͳ͍ • RDS ͱ ElastiCache

Slide 38

Slide 38 text

-# /HJOY "QQ 3FEJT NBTUFS 3FEJT TMBWF #BUDI .Z42-NBTUFS .Z42-TMBWF %FQMPZ STZOD "SDIFS 7*1 DIFGTPMP DBQJTUSBOP

Slide 39

Slide 39 text

ຊ൪αʔόߏங • consul ͷಋೖ • ಺෦ DNS • stretcher ͷಋೖ • Fluentd ͷઃఆݟ௚͠

Slide 40

Slide 40 text

Consul • Hashicorp ੡ͷΦʔέετϨʔγϣϯπʔϧ • KVSɺDNSɺHTTP ΠϯλϑΣʔε • ಺෦ DNS ͱͯ͠׆༻

Slide 41

Slide 41 text

Consul DNS • 8600 port Ͱఏڙ • node ͱ service ʹରԠ • nodename.node(.datacenter).consul • servicename.service(.datacenter).consul

Slide 42

Slide 42 text

dnsmasq • DNS forwarder, cache • consul ͷ 8600 port ʹ forward • શαʔόͰՔಇ

Slide 43

Slide 43 text

$ cat /etc/dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1 dnsmasq

Slide 44

Slide 44 text

VPC DHCP Options Set • VPC ͷ DHCP ͷઃఆΛߦ͏ • resolv.conf ͕ੜ੒͞ΕΔ • ݱࡏ͸ Management Console ͔Β΋ઃఆՄೳ

Slide 45

Slide 45 text

$ cat /etc/resolv.conf ; generated by /sbin/dhclient-script search node.consul service.consul nameserver 127.0.0.1 nameserver 10.1.0.2 ; VPC ͷ DNS nameserver 10.1.11.100 ; Unbound VPC DHCP Options Set

Slide 46

Slide 46 text

Unbound • DNS cache, resolver • VPC ͷ DNS ΛΩϟογϡ • VPC ͷ resolver ʹ໰୊͕͋ͬͯ΋໊લղܾ ͕Ͱ͖ΔΑ͏ʹ • deploy αʔόͰՔಇ

Slide 47

Slide 47 text

stretcher ʹΑΔ deploy • https://github.com/fujiwara/stretcher • Go ੡ • S3 ͔Β tarball Λऔಘͯ͠ల։ɺrsync • consulɺserf ͱ࿈ܞ • ৄ͘͠͸ɺhttps://speakerdeck.com/fujiwara3/ consultozi-zuo-osswohuo-yong-sita100tai-gui-mo- falsewebsabisuyun-yong

Slide 48

Slide 48 text

&-# /HJOY "QQ &MBTUJ$BDIF NBTUFS &MBTUJ$BDIF TMBWF #BUDI 3%4NBTUFS 3%4TMBWF CBDLVQ %FQMPZ DPOTVM FWFOU #VJME 6QMPBE 4 DIFGTPMP DBQJTUSBOP

Slide 49

Slide 49 text

ϝϯςφϯε (ࣄલ४උ) • ηΧϯμϦΠϯσοΫεΛ࡟আ͓͍ͯͨ͠ςʔ ϒϧΛҠߦઌͷ DB ʹ༻ҙ • RDS Λ Single-AZ ʹ͢Δ • ύϥϝʔλάϧʔϓͷௐ੔

Slide 50

Slide 50 text

ϝϯςφϯε (ࣄલ४උ) • RDS ΛSingle-AZ ʹ͢Δ • Multi-AZ ͷ৔߹ɺ଱ো֐ੑ͸޲্͢Δ͕ॻ ͖ࠐΈ଎౓͕2ഒʹ

Slide 51

Slide 51 text

ϝϯςφϯε (ࣄલ४උ) • ύϥϝʔλάϧʔϓͷมߋ • innodb_ﬂush_log_at_trx_commit=0 • innodb_support_xa=0 • sync_binlog=0

Slide 52

Slide 52 text

ϝϯςφϯε • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Redis ͷσʔλҠߦ • ಈ࡞֬ೝ • DNS੾Γସ͑ • Multi-AZ Խ & ύϥϝʔλάϧʔϓมߋ

Slide 53

Slide 53 text

MySQL ͷσʔλҠߦ • εΩʔϚ͸༧Ί৽ DB ʹೖΕͯ͋ΔͷͰɺσʔ λ͚ͩ dump (& gzip) • FOREIGN_KEY_CHECKS=0 ͱ 1 Λ zcat ͷલ ޙʹೖΕͯ mysql ʹ౉͢

Slide 54

Slide 54 text

$ { echo "FOREIGN_KEY_CHECKS=0"; zcat data.sql.gz; echo "FOREIGN_KEY_CHECKS=1"; } | mysql -u ... MySQL ͷσʔλҠߦ

Slide 55

Slide 55 text

MySQL ͷσʔλҠߦ • import ͕ऴΘͬͨΒηΧϯμϦΠϯσοΫε Λ࡞੒͢Δ • information_schema.tables Λݟ֤ͯςʔϒϧ ͷߦ਺Λൺֱ

Slide 56

Slide 56 text

mysql -u USER -B -N -e "SELECT TABLE_NAME, TABLE_ROWS FROM information_schema.tables WHERE TABLE_SCHEMA = 'my_db' ORDER BY TABLE_NAME" MySQL ͷσʔλҠߦ

Slide 57

Slide 57 text

Redis ͷσʔλҠߦ • redis-cli CONFIG SET appendonly yes • appendfsync no ʹ͢Δͱߴ଎(fsync͠ͳ͍) • redis-cli BGSAVE • Redis Λఀࢭ • cat appendonly.aof | redis-cli --pipe -h ...

Slide 58

Slide 58 text

Redis ͷσʔλҠߦ • Ҡߦલޙͷ྆؀ڥͰ KEYS “*” ͷ݁ՌΛϑΝΠ ϧʹམͱ͢ • ݁ՌΛ sort • diff Λऔͬͯ֬ೝ • expireͰ࡟আ͞Ε͍ͯΔͱଟগࠩ෼͕ग़Δ

Slide 59

Slide 59 text

ಈ࡞֬ೝ • Ҡߦޙͷ؀ڥʹ઀ଓͰ͖ΔΞϓϦΛ୺຤ʹೖ Ε͓ͯ͘ • υϝΠϯΛઃఆ • εϚʔτϑΥϯͷ DNS ઃఆΛมߋͨ͠Γɺ DNS ͷԠ౴Λِ૷͢ΔΑΓָ

Slide 60

Slide 60 text

Single-AZ -> Multi-AZ ύϥϝʔλάϧʔϓͷมߋ • Multi-AZ ʹมߋ • Single-AZ ͔Β Multi-AZ ΁ͷมߋ͸ΦϯϥΠϯͰ ࣮ߦՄೳ • ύϥϝʔλάϧʔϓͷมߋ • innodb_ﬂush_log_at_trx_commit=1 • innodb_support_xa=1 • sync_binlog=1

Slide 61

Slide 61 text

DNS ͷ੾Γସ͑ • TTL Λ༧Ί୹͓ͯ͘͘͠ • TTL Λ୹ͯ͘͠΋چ؀ڥʹΞΫηε͕དྷΔՄ ೳੑ • DB ΍ KVS ͸ඞͣࢭΊ͓ͯ͘

Slide 62

Slide 62 text

ࣄྫ 2,3

Slide 63

Slide 63 text

ࣄྫ 2,3 • ๻ΒͷߕࢠԂʂ೤ಆฤͷ Mobage ൛ͱ iOS / Android ൛ • Ҡߦظؒ߹Θͤͯ 1 ϲ݄ • ΄ͱΜͲαʔόߏ੒͕ಉ͡ͳͷͰಉ࣌ਐߦ • perlbrew ͰΠϯετʔϧͨ͠ perl ؀ڥΛͦͷ·· ͍࣋ͬͯ͘ • SL6 -> CentOS6 ͳͷͰ

Slide 64

Slide 64 text

ࣄྫ 2,3 • Ҡߦମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ ֤ 1 ໊ • ΦϖϨʔγϣϯΤϯδχΞ 1 ໊ • Φϒβʔόʔ 2 ໊

Slide 65

Slide 65 text

Ҡߦ࡞ۀ • dev αʔόߏங • (RDS for MySQL ͷݕূ) • yrmcds ͷݕূ • ຊ൪αʔόߏங • ϝϯςφϯε • Ҡߦޙʹൃੜͨ͠໰୊

Slide 66

Slide 66 text

dev αʔόߏங • Chef cookbook ͷ੔උ • Chef Ͱ؅ཧ͞Ε͍ͯͳ͍΋ͷΛચ͍ग़ͯ͠ cookbook Խ

Slide 67

Slide 67 text

yrmcds ͷݕূ • session ʹ Kyoto Tycoon Λ࢖༻ • ػೳతʹ໰୊͸ͳ͍ • ٕज़ݕূ΋݉Ͷͯผͷϛυϧ΢ΣΞΛ࢖͏ • Redis ͩͱΞϓϦͷେ෯ͳमਖ਼͕ඞཁ • Memcached ޓ׵ͷ΋ͷ͕ྑ͍ • yrmcds

Slide 68

Slide 68 text

yrmcds • https://github.com/cybozu/yrmcds • Memcached ΄΅׬શޓ׵ • master / slave ϨϓϦέʔγϣϯ • VIP ͕෇͍͍ͯΔαʔό͕ master • ϝϞϦ؅ཧʹ Slab Λ࢖༻͍ͯ͠ͳ͍ • αʔόαΠυϩοΫ

Slide 69

Slide 69 text

EC2 ʹ VIP Λ͚ͭΔ • EC2 ʹ VIP Λ͚ͭͯ΋௨৴Ͱ͖ͳ͍ • ENI ͸σϑΥϧτͰૹ৴ݩ/ૹ৴ઌνΣοΫ ͕༗ޮʹͳ͍ͬͯΔ • ແޮʹ͠ͳ͍ͱࣗ෼ѼͯҎ֎ͷύέοτ Λड͚औΕͳ͍

Slide 70

Slide 70 text

Keepalived • VIP ͷ੾Γସ͑ʹ࢖༻ • VPC Ͱ͸ϚϧνΩϟετ͕࢖͑ͳ͍ • ؤுΕ͹࢖͑ΔΑ͏Ͱ͕͢ɺKeepalived ͷՄ ༻ੑ΋ߟ͑ͳͯ͘͸ͳΒͳ͍ • Consul lock Λ׆༻ͨ͠ Failover γεςϜ

Slide 71

Slide 71 text

Consul lock • ෼ࢄϩοΫػೳ • consul lock service/lock deploy.sh • ෼ࢄηϚϑΥػೳ • consul lock -n 2 service/lock deploy.sh

Slide 72

Slide 72 text

Consul lock Λ׆༻ͨ͠ Failover γεςϜ • lock Λऔͬͨϗετʹ VIP Λ͚ͭΔ • ϩʔΧϧͷ yrmcds ͷࢮ׆؂ࢹ • μ΢ϯͨ͠Β lock Λղ์͢Δ • Slave Ͱ΋ lock ΛऔΓʹߦͬͯ lock ղ์଴ͪͷঢ়ଶʹ ͢Δ • yrmcds ͕μ΢ϯ͢Δͱ Slave ͷ͏ͪҰ͕ͭ lock Λऔͬ ͯ VIP Λ͚ͭΔ(master ʹঢ֨)

Slide 73

Slide 73 text

Consul lock Λ׆༻ͨ͠ Failover γεςϜ • VPC Route Table ͷॻ͖׵͑ • yrmcds Λ svc -o Ͱىಈ • উखʹ࠶ىಈ͠ͳ͍Α͏ʹ • /service/yrmcds ΋ىಈ࣌ʹ࡟আ • αʔό͕ಥવࢮͯ͠ VIP ͕͍ͭͨ··࠶ىಈ ͯ͠΋ master ʹͳΒͳ͍Α͏ʹ

Slide 74

Slide 74 text

ຊ൪αʔόߏங • Chef Ͱ؅ཧ͍ͯ͠ͳ͍΋ͷ͸ͳ͔ͬͨͷͰ cookbook Λվྑ • αʔόߏ੒Λมߋ • web αʔόͱ app αʔόͷ౷߹ • Kyoto Tycoon Λ yrmcds ʹҠߦ

Slide 75

Slide 75 text

-# /HJOY "QQ ,ZPUP5ZDPPO TFTTJPO #BUDI .Z42-NBTUFS .Z42-TMBWF %FQMPZ STZOD "SDIFS DIFGTPMP DBQJTUSBOP ,ZPUP5ZDPPO TFTTJPO CBDLVQ )"1SPYZ .FNDBDIFE DBDIF )"1SPYZ )"1SPYZ

Slide 76

Slide 76 text

ຊ൪αʔόߏங • MySQL Λ Slave ࢀর͠ͳͯ͘΋໰୊ͳ͘ͳͬ ͨͷͰ HAProxy ͕ෆཁʹ • Consul + dnsmasq ಋೖ • ಺෦DNS

Slide 77

Slide 77 text

&-# ZSNDET NBTUFS ZSNDET TMBWF #BUDI %FQMPZ STZOD "SDIFS 7*1 DIFGTPMP DBQJTUSBOP 3%4NBTUFS 3%4TMBWF CBDLVQ /HJOY "QQ

Slide 78

Slide 78 text

Ҡߦ࡞ۀ • ΞϓϦΛϝϯςφϯεϞʔυʹ͢Δ • MySQL ͷσʔλҠߦ • Session ͷσʔλҠߦ • (ಈ࡞֬ೝ) • (DNS੾Γସ͑)

Slide 79

Slide 79 text

MySQL ͷσʔλҠߦ • MySQL ͷσʔλҠߦΛ࠷଎खॱͰ࣮ࢪ • Duplicate entry ͕ग़ࣦͯഊ... • લ೔ͷԋशͰ͸੒ޭ • ະͩʹݪҼෆ໌ • ֎෦Ωʔ੍໿Λ΄ͱΜͲ࢖͍ͬͯͳ͍ͨΊ௨ ৗͷ mysqldump Ͱ΋ 5 ෼͔͕ࠩ͠ͳ͔ͬͨ

Slide 80

Slide 80 text

Session ͷσʔλҠߦ • ͠·ͤΜͰͨ͠ • ϩάΠϯ͠௚͠ • Ϣʔβ਺తʹ໰୊ͳ͠ • ϩάΠϯ͕ूத͢Δͱࡹ͖੾Εͳ͍৔߹΋ ͋ΔͷͰ௨ৗ͸Ҡߦͨ͠΄͏͕ྑ͍

Slide 81

Slide 81 text

Ҡߦޙʹൃੜͨ͠໰୊ • Consul ͸਺ඵ Leader lost ͢Δ͜ͱ͕͋Δ • Leader lost ͢Δͱ lock ͕֎ΕΔ... • Ҡߦޙ 2 ೔࿈ଓͰ Failover • dns_conﬁg.node_ttl, allow_stale, max_stale Λઃఆ͢Δ • Leader lost ࣌΋਺ඵͳΒ DNS ͕Ҿ͚Δ

Slide 82

Slide 82 text

؂ࢹʹ ͍ͭͯ

Slide 83

Slide 83 text

Zabbix • ࣗࣾαʔϏεͷ؂ࢹ • VPC Peering Connection Ͱ internal ௨৴ • Ұ෦ Zabbix Proxy Λར༻

Slide 84

Slide 84 text

Zabbix Proxy • Zabbix Server ͷ୅ΘΓͱͯ͠ಈ࡞ • ෛՙͷܰݮ • ऩूσʔλΛ֤ϗετͰόοϑΝϦϯά • Firewall ͷઃఆ͕༰қ • ֎෦νΣοΫ • ίϚϯυΛ࣮ߦͯ͠ MySQL ͷ QPS Λܭଌ

Slide 85

Slide 85 text

Zabbix Proxy • ศར͕ͩ SPOF ʹͳΔ • ΠϯλʔωοτΛܦ༝ͤͣ؂ࢹ͕Մೳʹͳͬ ͨͷͰ௚઀؂ࢹʹมߋ • Active Agent Check • ผͷ໰୊͕...

Slide 86

Slide 86 text

Zabbix Active Check • ֎෦νΣοΫͷεΫϦϓτʹύεϫʔυΛॻ ͍͍ͯΔ • Zabbix Proxy ͕ೖ͍ͬͯΔϗετʹઃఆ • ௚઀؂ࢹʹͳΓ࢖͑ͳ͘ͳΔ • ؂ࢹର৅͕૿͑Δͨͼʹ Zabbix Server ͷઃ ఆมߋΛߦ͏ͷ͸ආ͚͍ͨ

Slide 87

Slide 87 text

Zabbix Active Check • UserParameter Λઃఆͨ͠ϗετΛొ࿥ • RDS ΍ ElastiCache ͳͲͷ؂ࢹ͕Մೳʹ • ςϯϓϨʔτʹొ࿥ͨ͠ΞΠςϜΛϗετϚ ΫϩͰมߋՄೳʹ • ࠶ར༻͠΍͍͢

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

PUSH ௨஌༻ SSL ূ໌ॻͷ؂ࢹ • PUSH ௨஌ʹ࢖͏ SSL ূ໌ॻͷظݶΛ؂ࢹ • ূ໌ॻͷ PATH ͸αʔό͝ͱʹҧ͏ • ϓϩδΣΫτ͝ͱʹΞΠςϜΛ௥Ճ͢Δͷ͸ େม • Low Level Discovery(LLD) Ͱղܾ

Slide 91

Slide 91 text

Low Level Discovery(LLD) • ΞΠςϜɺτϦΨɺάϥϑΛࣗಈͰੜ੒͢Δ ͨΊͷػೳ • ϑΝΠϧγεςϜɾωοτϫʔΫΠϯλ ϑΣʔεݕग़༻ͷLLD͕૊Έࠐ·Εͨςϯϓ Ϩʔτ΋͋Δ • LLD Λࣗ࡞͢Δ͜ͱ΋Մೳ

Slide 92

Slide 92 text

Low Level Discovery(LLD) • UserParameter ʹ ΞΠςϜऩू༻ͷઃఆ௥Ճ • ΞΠςϜऩू͸ɺܾΊΒΕͨܗࣜͷ JSON σʔλΛฦ͢

Slide 93

Slide 93 text

#!/bin/bash CERTS="${1}" CERTS_DATA="" for CERT in ${CERTS}; do CERTS_DATA="${CERTS_DATA},{\"{#CERT_PATH}\":\"${CERT}\"}" done echo "{\"data\":[$(echo ${CERTS_DATA} | sed -e 's/^,//')]}" Low Level Discovery(LLD)

Slide 94

Slide 94 text

{ "data": [ {"{#CERT_PATH}":"/path/to/cert1"}, {"{#CERT_PATH}":"/path/to/cert2"} ] } Low Level Discovery(LLD)

Slide 95

Slide 95 text

#!/bin/bash CERT="${1}" NOW=$(date +%s) EXPIRE=$(date -d "$(openssl x509 -in ${CERT} -noout -dates | tail -n 1 | sed -e 's/notAfter=//')" +%s) echo $(((${EXPIRE} - ${NOW}) / 60 / 60 / 24)) Low Level Discovery(LLD)

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

εϖʔε۠੾ΓͰূ໌ॻΛࢦఆ

Slide 99

Slide 99 text

RDS ͷ؂ࢹ • ﬂuent-plugin-cloudwatch ͰϝτϦΫεΛऩू • ﬂuent-plugin-zabbix Ͱ Zabbix ʹૹ৴ • MySQL ͷ stats ͸ SQL Λ࣮ߦ • SHOW ENGINE INNODB STATUS • SHOW GLOBAL STATUS

Slide 100

Slide 100 text

ElastiCache ͷ؂ࢹ • redis-cli INFO ͷ stats Λऩू • DNS ΛҾ͍ͯ Primary Cache Cluster ͷ IP Λ؂ࢹ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover • UserParameter ͰίϚϯυ࣮ߦ • net.dns.record ͸ DNS ͕Ҿ͚ͳ͍࣌͸Ξϥʔ τΛඈ͹ͤͳ͍...

Slide 101

Slide 101 text

#!/bin/bash host="/usr/bin/host" DOMAIN=${1} TIMEOUT=10 RECORDS=$($host -s -W $TIMEOUT $DOMAIN | awk 'NR > 1 {print $NF}' | sort) [ "${RECORDS}" = "" ] && echo "fail" && exit 1 echo $RECORDS ElastiCache ͷ؂ࢹ

Slide 102

Slide 102 text

yrmcds ͷ؂ࢹ • stats ͰϝτϦΫεΛऔಘՄೳ • https://github.com/cybozu/yrmcds/issues/47 ͰରԠ͍͖ͯͨͩ͠·ͨ͠ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • VIP ͕෇͍͍ͯΔΠϯελϯεͷ Private IP Λ νΣοΫ • લޙͰ஋͕ҟͳΔ৔߹͸ Failover ͱͯ͠ݕग़

Slide 103

Slide 103 text

Jenkins Slave ͷ׆༻

Slide 104

Slide 104 text

Jenkins EC2 Plugin • Jenkins Slave Λ AMI ͔Βىಈ • ςετ͕࣮ߦ͞Εͳ͘ͳ͔ͬͯΒ 30෼͘Β ͍Ͱ Terminate • ىಈ͕࣌ؒ୹͘ͳΔͷͰ҆ՁʹͳΔ • Spot Instance ΋࢖͑ΔͷͰɺΑΓ҆Ձʹ

Slide 105

Slide 105 text

Jenkins EC2 Plugin • 32 core ͷ CPU ͰίΞ਺෼·Ͱฒྻ਺Λ͋͛ ͯςετΛճ͢ͱ͚͜ΔΑ͏ʹ... • MySQL ͕ EAGAIN ΤϥʔΛग़͢Α͏ʹ • Test::mysqld Ͱෳ਺ݸͷ MySQL Λ্ཱͪ ͍͛ͯΔ • fs.aio-max-nr=65536 Λ 4 ഒʹͯ͠ճආ

Slide 106

Slide 106 text

Jenkins EC2 Plugin • ଞͷΧʔωϧύϥϝʔλ΋νϡʔχϯά • ςετ͸ߴ଎ԽͰ͖ͣ... • MySQL ͷ্ཱͪ͛ʹ͕͔͔࣌ؒΔ • Test::mysqld::Multi • ฒྻ਺ΛNഒͯ͠΋࣮ߦ࣌ؒ͸1/NʹͳΒͳ͍ • CPUΛ࢖͍੾Εͳ͍

Slide 107

Slide 107 text

AWS Ҡߦͷ ϝϦοτɾ σϝϦοτ

Slide 108

Slide 108 text

AWS Ҡߦͷֹۚ໘Ͱͷ ϝϦοτ • ϓϩδΣΫτ͝ͱʹਖ਼֬ͳར༻ֹ͕ग़ͤΔ • ΦϯϓϨͰ͸ɺαʔό(VM)͕༨ͬͨ৔߹શ ࣾඅ༻ʹճ͍ͯͨ͠ • ݟ্͔͚͸҆ՁʹͳΔ͕ɺશࣾతʹ͸අ༻ ͕͔͔͍ͬͯΔ͜ͱΛཧղ͞Εͳ͍

Slide 109

Slide 109 text

AWS Ҡߦͷֹۚ໘Ͱͷ σϝϦοτ • අ༻͸ଟগ૿Ճ͢Δ • େن໛Ͱͳ͚Ε͹ਓ݅අΛؚΊΔͱ҆Ձ(ͳ ͸ͣ) • ࣗલͰαʔόӡ༻͢ΔΑΓָ • దਖ਼ͳඅ༻͕ܭ্͞ΕΔͨΊɺඅ༻͕૿Ճ͠ ͨΑ͏ʹࢥΘΕͯ͠·͏

Slide 110

Slide 110 text

AWS Ҡߦͷػೳ໘Ͱͷ ϝϦοτ • ϚωʔδυαʔϏεͰӡ༻ෛՙͷ௿ݮ • ҆Ձʹαʔόͷަ׵͕Մೳ • ϋʔυ΢ΣΞੑೳͷ޲্ • ΞϓϦͷνϡʔχϯάͳ͠ͰϨεϙϯελ ΠϜվળ

Slide 111

Slide 111 text

No content

Slide 112

Slide 112 text

AWS Ҡߦͷػೳ໘Ͱͷ σϝϦοτ • ඞཁͳػೳ͕ͳ͍͜ͱ͕͋Δ • ElastiCache ͷ reboot with failover • RDS ͷ λΠϜκʔϯରԠ

Slide 113

Slide 113 text

஌ݟ

Slide 114

Slide 114 text

஌ݟ • ҆ఆӡ༻Ͱ͖͍ͯΔͱݱঢ়ҡ࣋ʹͳΓ͕ͪ • αʔόߏ੒ͷݟ௚͕͠Ͱ͖Δνϟϯε • ྑ͘ͳ͔ͬͨ෦෼͕ු͖ூΓʹͳΔ • ؂ࢹ͢΂͖͜ͱ͕Ͱ͖͍ͯͳ͍ • Chef Ͱ؅ཧͰ͖͍ͯΔ΋ͷͱͰ͖͍ͯͳ͍ ΋ͷ͕ࠞࡏ͍ͯ͠Δ

Slide 115

Slide 115 text

஌ݟ • े਺୆ن໛ͷγεςϜͰ͋Ε͹ 2~3 ໊ఔ౓Ͱ 1 ϲ݄͋Ε͹ҠߦͰ͖Δ • ҙ֎ͱԿͱ͔ͳΔ • ϝϯςφϯε೔ͷௐ੔͕ҙ֎ͱେม

Slide 116

Slide 116 text

஌ݟ • ࣄલԋशΛԿ౓΋΍Δͱ҆৺ • MySQL ͷσʔλҠߦ͸ݕূ࣌ʹ3ɾ4ճɺલ ೔ʹ 2 ճ࣮ࢪ • ίϚϯυΛίϐϖ࣮ͯ͠ߦ͢Δ͚ͩͰྑ͍ ঢ়ଶʹ͓ͯ͘͠ͱߋʹ҆৺

Slide 117

Slide 117 text

·ͱΊ • AWS ͷҠߦࣄྫ঺հ • ݱঢ়ҡ࣋ΑΓ΋վળ • ߏ੒Λݟ௚͢ྑ͍ػձ • ػձ͕ͳ͍ͱվળ͠ʹ͍͘