Tweet
Tweet

Slide 1

Slide 1 text

2೥ؒӡ༻͍ͯ͠Δ ιʔγϟϧήʔϜͷ αʔόߏ੒ͷมભ @tkuchiki

Slide 2

Slide 2 text

※ެ։༻ʹҰ෦εϥΠυΛ ࡟আ͓ͯ͠Γ·͢

Slide 3

Slide 3 text

WHOAMI(1) • @tkuchiki • ΠϯϑϥνʔϜ • ୲౰ • ιʔγϟϧήʔϜ • ࣗࣾαʔϏε • Web+DB PRESS Vol.94

Slide 4

Slide 4 text

Agenda • αʔόߏ੒ͷมભ • ໰୊ղܾฤ • Auto Scaling ฤ • Replace ฤ

Slide 5

Slide 5 text

໰୊ղܾฤ

Slide 6

Slide 6 text

ϦϦʔε࣌ͷαʔόߏ੒

Slide 7

Slide 7 text

ϛυϧ΢ΣΞ • શ୆ڞ௨ • Perl 5.16.3 • consul + dnsmasq • ಺෦ DNS • Service Discovery(DNS) • zabbix-agent

Slide 8

Slide 8 text

ϛυϧ΢ΣΞ • Openresty 1.4.6 • MySQL 5.6(RDS) • zabbix-proxy

Slide 9

Slide 9 text

ϛυϧ΢ΣΞ • Redis + Redis Sentinel • master ͷ IP Λ Sentinel ʹ໰͍߹ΘͤΔ • Consul ͷ Service Discovery • gearmand • Job Queue

Slide 10

Slide 10 text

ϛυϧ΢ΣΞ • fluent-agent-lite • Nginx ͷϩάΛసૹ • td-agent • s3, numric_counter, datacounter, sampling_filter, anomalydetect, cloudwatch, zabbix, suppress, redshift

Slide 11

Slide 11 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ

Slide 12

Slide 12 text

fluentd ͕٧·Δ • ॱௐʹΞΫηε͕૿͑Δ • ϩάू໿ଆͷ fluentd ͕٧·Γ࢝ΊΔ • fluentd ͸ 1core ͔͠࢖͑ͳ͍ • Nginx ͷΞΫηεϩάू໿ͷෛՙ͕ߴ͍ • fluent-plugin-multiprocess Ͱ Nginx ͷ ΞΫηεϩάू໿ͱͦΕҎ֎Λ෼͚Δ

Slide 13

Slide 13 text

σΟεΫ༰ྔ͕ᷓΕͦ͏ʹͳΔ • fluentd ͷෛՙ௿ݮ͸੒ޭ • ࠓ౓͸σΟεΫ༰ྔ͕଍Γͳ͘ͳΔ • cron.daily ͰϩάͷѹॖΛͯ͠ɺҰఆظؒա ͗ͨΒ࡟আ͍ͯͨ͠ • σΟεΫΛ௥Ճͯ͠ cron.daily ͰϩάΛୀආ

Slide 14

Slide 14 text

·ͩσΟεΫ༰ྔ͕ᷓΕͦ͏ • 1೔1ճͷϩάୀආͰ͸ؒʹ߹Θͳ͘ͳΔ • ຖ࣌ϩάΛѹॖͱୀආ

Slide 15

Slide 15 text

MySQL ߴෛՙ • MySQL 1 ୆Ͱ͸ࡹ͖͖Εͳ͍ෛՙ • Slave(Read Replica) Λ௥Ճͯ͠ࢀর෼ࢄ • HAProxy ಋೖ • ͍ͭͰʹ؅ཧը໘༻ͷ Slave Λ௥Ճ

Slide 16

Slide 16 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM Log(Backup) Log Asset Bundle 0QFOSFTUZ

Slide 17

Slide 17 text

ࢼ߹ Batch ͷ࣮ߦ͕࣌ؒ௕͍ • Ϣʔβ૿Ճʹ൐͍ࢼ߹਺΋૿Ճ • Batch ͷ࣮ߦ׬ྃ·Ͱͷ͕࣌ؒͲΜͲΜ৳ͼ ͍ͯ͘ • εέʔϧΞοϓͰରԠ • ݱࡏ΋εέʔϧΞ΢τ͠ͳ͍ߏ੒ͳͷͰࠓ ޙͷ՝୊

Slide 18

Slide 18 text

fluent-plugin-redshift͕ࢗ͞Δ • fluentd ͔Β redshift ʹσʔλΛ import ͢Δ • ϦϦʔε࣌͸໰୊ͳ͔͕ͬͨɺΞΫηε͕૿ ͔͑ͯΒ࣌ʑࢗ͞ΔΑ͏ʹͳΔ • plugin-redshift ͕ࢗ͞Δͱ fluentd ࣗମ͕ ࢭ·Δ • ൒೥Ҏ্ଓ͖·ͨ͠

Slide 19

Slide 19 text

rin • https://github.com/fujiwara/rin • S3 ͷΠϕϯτ௨஌Ͱ SQS ʹΩϡʔΛ௥Ճ • SQS ͷΩϡʔΛऔΓग़ͯ͠ redshift ʹ COPY ίϚϯυΛ࣮ߦ

Slide 20

Slide 20 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 Redis x 2 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log 3FEJT 3FEJT 4FOUJOFM UEBHFOU 3FEJT 4FOUJOFM SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ Log( +Backup)

Slide 21

Slide 21 text

ElastiCache for Redis ʹҠߦ • Sentinel ͷӡ༻ίετ͕ߴ͍ • ElastiCache for Redis ͷ Multi-AZ ͕ϦϦʔε • EC2(ΦϯϓϨ)͔ΒͷҠߦͳΒ AOF Λग़ྗ͠ ͯ redis-cli --pipe Ͱྲྀ͚ͩ͢

Slide 22

Slide 22 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup)

Slide 23

Slide 23 text

҆ఆظʹಥೖ • ϦϦʔεޙ1೥͘Β͍Ͱେ͖ͳ໰୊͸ղফ • վળʹऔΓ૊ΊΔΑ͏ʹͳΔ

Slide 24

Slide 24 text

Auto Scaling ฤ

Slide 25

Slide 25 text

αʔόͷखಈ௥Ճ • ΠϕϯτͰߴෛՙ͕ݟࠐ·ΕΔ৔߹ʹ Webɾ ΞϓϦέʔγϣϯαʔόΛखಈͰ௥Ճ • AWS CLI Ͱ EC2 Λىಈͯ͠ ELB ʹ௥ՃͰ͖ ΔΑ͏ʹ͢ΔεΫϦϓτΛ࡞੒ • Management Console ͕৮Εͳͯ͘΋࡞ ۀ͕Ͱ͖Δ

Slide 26

Slide 26 text

αʔόͷखಈ௥Ճ • ࡞ۀ͸Ͱ͖Δ͕... • ෛՙ͕௿͍ͱαʔόϦιʔε͕༨Δ • ༧ظͤ͵ෛՙ૿େʹରԠͰ͖ͳ͍ • EC2 ͕ಥવࢮ͢ΔͱखಈͰ෮׆ͤ͞Δඞཁ͕ ͋ͬͨ • ࠓ͸ CloudWatch Alarm Ͱࣗಈ࠶ىಈͤ͞Δ ͜ͱ͕Ͱ͖·͢

Slide 27

Slide 27 text

Auto Scaling ͷಋೖ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ޲্ • खಈΦϖϨʔγϣϯΛۃྗݮΒ͢ • Πϕϯτ։࢝લʹ minimum capacity Λ૿΍ ͢ඞཁ͕͋ΔͨΊ׬શʹͳ͘͢͜ͱ͸Ͱ͖ ͳ͍

Slide 28

Slide 28 text

Auto Scaling ಋೖͷ໰୊ • Ͳ͏΍ͬͯαʔόΛ࠷৽ͷঢ়ଶʹ͢Δʁ • ΞϓϦέʔγϣϯίʔυɾαʔόઃఆΛम ਖ਼͢Δ౓ʹ AMI Λ࡞੒ʁ • αʔόىಈ࣌ʹ deploy ͢Δʁ • deploy αʔό͔ΒίʔυΛऔͬͯ͘Δʁ

Slide 29

Slide 29 text

Stretcher • https://github.com/fujiwara/stretcher • Consul/Serf ͷ Event Λड͚ͯ manifest Λ औಘ • tarball Λ S3 ͳͲ͔Βऔಘͯ͠ rsync Ͱల։ • ೚ҙͷίϚϯυΛ࣮ߦͰ͖Δ • chef-solo ͷ࣮ߦʹ΋ར༻͍ͯ͠Δ

Slide 30

Slide 30 text

# stretcher $ consul watch -type event -name deploy stretcher # deploy $ consul event -name deploy s3://path/to/manifest.yml # ඪ४ೖྗΛड͚Δ͜ͱ΋Մೳ $ echo s3://path/to/manifest.yml | stretcher Stretcher ͰͷσϓϩΠྫ

Slide 31

Slide 31 text

Auto Scaling ͷ࢓૊Έ • Consul KV ʹ deploy ͞Ε͍ͯΔίʔυͷ manifest ϑΝΠϧͷ path Λอଘ • /etc/rc.local Ͱ stretcher Λ࢖ͬͯɺchef-solo ͱΞϓϦͷ deploy • tarball ʹ deploy id ΛೖΕͯɺConsul KV ͷ஋ͱൺֱ͠ɺঢ়ଶ ͕Ұக͠ͳ͚Ε͹ΞϓϦΛىಈ͠ͳ͍ • Zabbix ʹϗετΛ௥Ճ • autoscale service Λ࡞ͬͯɺshutdown ࣌ʹ Zabbix ϗετΛ࡟ আ & Consul Cluster ͔Β leave

Slide 32

Slide 32 text

Auto Scaling ಋೖͷޮՌ • ౰ॳͷ໨త͸ୡ੒ • αʔόඅͷ࠷దԽ • Մ༻ੑͷ޲্ • Pull ܕ deploy ʹมߋ͢Δ͜ͱͰ deploy ࣌ؒ ୹ॖ

Slide 33

Slide 33 text

Replace ฤ

Slide 34

Slide 34 text

৽ήʔϜαʔόϦϦʔε • 2016೥8݄ʹ৽ήʔϜαʔόͷϦϦʔεܾఆ • طଘήʔϜαʔόͱ͸શ͘ผͷ؀ڥͱͯ͠ߏங • ιʔείʔυ͸ڞ༗ • DB ΋ޓ͍ʹࢀর͋͠Θͳ͍ • ΦϯϥΠϯήʔϜͷήʔϜαʔόͷΑ͏ͳΠϝʔδ • ϦϦʔε࣌ʹ௕࣌ؒϝϯςφϯεΛೖΕΒΕΔ • طଘ؀ڥ΋ Replace ͢Δ͜ͱʹ

Slide 35

Slide 35 text

طଘήʔϜαʔόͷ໰୊ • OS ͕ݹ͍ • Amazon Linux 2014.03 • ΠϯελϯελΠϓ΋ݹ͍ • ߏ੒͕ݹ͍ɾྑ͘ͳ͍ • ϩάू໿αʔό͕εέʔϧ͠ͳ͍ • ϩάͷूܭ͸ Norikra Ͱ... • etc

Slide 36

Slide 36 text

αʔόߏ੒ͷ Replace • OS Λ Amazon Linux ࠷৽൛ʹ • ΠϯελϯελΠϓ΋࠷৽(c4, m4)ʹ • ϩάू໿αʔόΛεέʔϧͰ͖ΔΑ͏ʹ • fluent-agent-lite -> fluent-agent-hydra • RDS for MySQL -> Aurora

Slide 37

Slide 37 text

αʔόߏ੒ͷ Replace • ϛυϧ΢ΣΞͷόʔδϣϯΞοϓ • Openresty • Consul • HAProxy • td-agent Λ΍Ίͯ build ͨ͠࠷৽൛ͷ ruby Ͱ fluentd Λಈ͔͢ • εέʔϧͰ͖ΔΑ͏ʹ͢ΔͨΊ multiprocess Λ࢖Θ ͳ͍Α͏ʹ

Slide 38

Slide 38 text

αʔόߏ੒ͷ Replace • Consul ͷ External Service Λ׆༻ • Consul Cluster ֎ͷ Service Discovery ػೳ • RDS ͱ ElastiCache ͷ endpoint Λ Consul DNS ܦ༝Ͱࢀর • ήʔϜαʔό͝ͱʹઃఆϑΝΠϧͰࢀরઌΛม͑Δඞཁ ͕ͳ͍ • consul-template Λ׆༻ͯ͠ϝϯςφϯεΛೖΕΒΕΔΑ͏ʹ͢ Δ • KV Λมߋͨ͠Β Nginx ͕ 503 Λฦ͢Α͏ʹͳΔ

Slide 39

Slide 39 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 S3 CloudFront RDS for MySQL Redshift Origin 3%4GPS.Z42- 3%4GPS.Z42-Y 0QFOSFTUZ qVFOUBHFOUMJUF UEBHFOU )"1SPYZ qVFOUBHFOUMJUF HFBSNBOE Log UEBHFOU SJO Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log( +Backup) Replace લ

Slide 40

Slide 40 text

ELB Web+App Deploy x 1 Batch x 1 Admin x 1 S3 CloudFront Aurora Redshift Origin "VSPSB "VSPSBY 0QFOSFTUZ qVFOUBHFOUIZESB qVFOUE )"1SPYZ qVFOUBHFOUIZESB HFBSNBOE Log Log( +Backup) Log Asset Bundle 0QFOSFTUZ SQS COPYίϚϯυ ElastiCache for Redis Log x 2 Norikra x 1 qVFOUE SJO qVFOUE /PSJLSB Log Replace ޙ

Slide 41

Slide 41 text

Replace ͷޮՌ • ϩάू໿αʔόͷೖΕସ͕͑(΄΅)ແఀࢭͰߦ ͑Δ • Aurora มߋޙɺEBS ͷϨΠςϯγ૿ʹΑΔΫ ΤϦੑೳͷྼԽ͕ൃੜ͠ͳ͘ͳΔ • Norikra + fluentd + fluent-plugin-slack ͰϦΞ ϧλΠϜͳΤϥʔ௨஌͕Մೳʹ

Slide 42

Slide 42 text

Replace ͷޮՌ • ϩάͷूܭΛ Norikra Ͱߦ͏͜ͱͰϩάू໿ αʔόͷෛՙ௿ݮ • Zabbix ΁ͷϝτϦΫεૹ৴͸ https:// github.com/tkuchiki/norikra-listener-zabbix Ͱ Norikra ͔Β௚઀ૹ৴

Slide 43

Slide 43 text

Replace ͷޮՌ • ϝϯςφϯεঢ়ଶͷ੾Γସ͕͑ߴ଎ʹ • Nginx ͰϑΝΠϧͷ༗ແΛݟͯ 503 Λฦ͠ ͍ͯͨͨΊɺͦͷͨΊ͚ͩʹ deploy ͢Δ͔ શ୆ʹϑΝΠϧΛ഑Δɾফ͢࡞ۀ͕ඞཁͩͬ ͨ

Slide 44

Slide 44 text

Replace ͷޮՌ • ΠϯελϯελΠϓΛมߋͨ͜͠ͱͰ CPU ͷ ॲཧੑೳ͕޲্ • ϨεϙϯελΠϜ͕(ଟগ)ߴ଎ʹ

Slide 45

Slide 45 text

՝୊ • Batch αʔό͕εέʔϧ͠ͳ͍ • gearmand Λ SQS ʹஔ͖׵͑Δ • Norikra ͕ SPOF • Kinesis Streams, Analytics, Firefose + Lambda ʁ