モンストのマルチクラウドについて / sre-lounge-at-xflag

by Tomohiro Koike

Slide 1

Slide 1 text

SRE Lounge at XFLAG XFLAG ։ൃຊ෦ XFLAGࣄۀਪਐࣨ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ

Slide 2

Slide 2 text

ࣗݾ঺հ ‣XFLAGࣄۀਪਐࣨ SREάϧʔϓॴଐ ‣ϛΫγΟʹ͸2007೥຤ʹೖࣾ ‣ۀ຿಺༰ ‣ϞϯελʔετϥΠΫ ‣೔ຊ൛/ւ֎൛ ‣Πϯϑϥؔ࿈ͷۀ຿ ‣ͦͷଞ͍Ζ͍Ζ 2

Slide 3

Slide 3 text

XFLAG ελδΦʹ͓͚ΔSRE 3

Slide 4

Slide 4 text

XFLAG ελδΦʹ͓͚ΔSRE ‣2016೥7݄ʹSREάϧʔϓ૑ઃ ‣ਓ਺͸7໊(2018-05-10ݱࡏ) ‣৽ଔͱத్ೖ͕ࣾ൒ʑ͘Β͍ ‣ϑϧελοΫΛٻΊ͍ͯͳ͍ ‣΍Δ΂͖࢓ࣄ͸ࣗ෼Ͱݟ͚ͭɺೳಈతʹ࣮ߦ͢Δ ‣ίϛϡχέʔγϣϯ͸Slack, Githubͷissueத৺ 4

Slide 5

Slide 5 text

XFLAG ελδΦʹ͓͚ΔSRE ‣Google ͷఏএ͢Δ͍ΘΏΔSRE ͱ͸ҧ͏΋ͷ΋ ‣৽نαʔϏεͷ։ൃΛͨ͠Γ ‣αʔϏε֎Ͱͷ։ൃɾߏஙͨ͠Γ 5

Slide 6

Slide 6 text

XFLAG ελδΦʹ͓͚ΔSRE SREͷϛογϣϯͱ͍͑͹ 6

Slide 7

Slide 7 text

XFLAG ελδΦʹ͓͚ΔSRE αʔϏεͷ҆ఆԽͷͨΊ ίʔυʹΑΔෛՙ࡟ݮɾࣗಈԽ 7

Slide 8

Slide 8 text

XFLAG ελδΦʹ͓͚ΔSRE ͦΕ͚ͩͰ͸ͳ͍ 8

Slide 9

Slide 9 text

XFLAG ελδΦʹ͓͚ΔSRE ຊ೔͸SREͷࣄྫͷҰͭͱͯ͠ ϞϯετͷϚϧνΫϥ΢υӡ༻ͷ͓࿩ 9

Slide 10

Slide 10 text

Ϟϯετͷߏ੒ 10

Slide 11

Slide 11 text

Ϟϯετͷߏ੒(γεςϜ) ‣ΞϓϦέʔγϣϯ ‣Ruby(padrino) + unicorn + Nginx ‣σʔλϕʔε ‣MariaDB 5.5 ‣Batch ‣Rescue + redis ‣Ωϟογϡ ‣memcached ‣ϩά ‣Fluentd 11

Slide 12

Slide 12 text

Ϟϯετͷߏ੒(γεςϜ) 12 Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer

Slide 13

Slide 13 text

Ϟϯετͷߏ੒(਺ࣈ) ‣ΞϓϦέʔγϣϯ ‣ίΞ਺: ໿ 15,000 ‣σʔλϕʔε ‣୆਺: 144୆ ‣େྔͷϝϞϦʹ ioMemoryΛੵΜͩϚγϯ΋͋Δ ‣memcached ‣୆਺: 40୆ ‣૯༰ྔ 1.2TB 13

Slide 14

Slide 14 text

Ϟϯετͷߏ੒(Πϯϑϥ) 14 DataCenter 1 • DB • memcached DataCenter 2 • DB • memcached application application application application Cloud1 Cloud2 Cloud3 Cloud4 Internet

Slide 15

Slide 15 text

͜͏ͳΔ·Ͱ 15

Slide 16

Slide 16 text

Πϯϑϥߏ੒ͷྺ࢙ ‣Ϟϯετ͸ΦϯϓϨͰӡ༻͞Εͯͨ ‣εϚϗήʔϜͷಛੑͱͯ͠ ‣ΠϕϯτͰരൃతʹΞΫηε૿ 16

Slide 17

Slide 17 text

Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ΞϓϦέʔγϣϯαʔόΛҰ࣌తʹ૿ڧ͍ͨ͠ ‣DB͸σʔληϯλʹ͋Δ΋ͷΛࢀর͍ͨ͠ 17

Slide 18

Slide 18 text

Πϯϑϥߏ੒ͷྺ࢙ ‣AWS ͷ EC2 ΠϯελϯεΛར༻ ‣c4.4xlargeλΠϓ 18

Slide 19

Slide 19 text

Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ϩοΫΠϯ͸ආ͚͍ͨ ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 19

Slide 20

Slide 20 text

Πϯϑϥߏ੒ͷྺ࢙ ‣GMO ΞϓϦΫϥ΢υΛར༻ ‣Xeon E5-2690v3 λΠϓ 20

Slide 21

Slide 21 text

Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣प೥΍೥຤೥࢝ͷΠϕϯτ ‣େྔʹΞΫηε͕དྷͦ͏(1೔͚ͩ) ‣ίετύϑΥʔϚϯε ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 21

Slide 22

Slide 22 text

Πϯϑϥߏ੒ͷྺ࢙ ‣IBMΞϓϦΫϥ΢υͷϕΞϝλϧϚγϯ ‣Xeon E5-2690V4 λΠϓ ‣Google Cloud ͷCompute Engine ‣n1-highcpu-64 λΠϓ 22

Slide 23

Slide 23 text

ϚϧνΫϥ΢υӡ༻ 23

Slide 24

Slide 24 text

ϚϧνΫϥ΢υͷӡ༻ ‣γϯϓϧʹ ‣֤Ϋϥ΢υͰΠϝʔδ(AMI౳)Λ࡞੒ ‣ϓϩϏδϣχϯά͸ansible, Chef ͳͲར༻ ‣؂ࢹͳͲ͸ΦϯϓϨɾΫϥ΢υࠞࡏ ‣Nagios ‣CloudForecast ͳͲ 24

Slide 25

Slide 25 text

ϚϧνΫϥ΢υͷӡ༻ ➡ϕʔεͷΠϝʔδ͔Βىಈ ➡ansible Λద༻͢Δ ➡Ϣʔβͷ௥ՃɺωοτϫʔΫͷઃఆetc ➡chef Λద༻ ➡ύοέʔδͷ௥Ճɺconfig ௐ੔ 25

Slide 26

Slide 26 text

ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υຖͷAPIʹ߹Θͤͨࣾ಺CLIπʔϧ ‣Πϯελϯεͷ࡞੒ɾҰཡ ‣Πϯελϯεͷ࡟আ ‣Πϝʔδͷ࡞੒ɾҰཡ 26

Slide 27

Slide 27 text

ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υɾσʔληϯλؒͷωοτϫʔΫ ‣μΠϨΫτίωΫτ઀ଓ ‣10G × n ຊ ‣ωοτϫʔΫνʔϜͷ͓͔͛ 27

Slide 28

Slide 28 text

ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υͰͷ഑෼ɾόϥϯεௐ੔ ‣ҰՕॴ͕ଟ͘ͳΓ͗ͨ͢Γ͸͠ͳ͍Α͏ʹ ‣ఆظతʹόϥϯεௐ੔Λߦ͏ ‣ਓྗͰ ‣ಀ͛ಓ͸༻ҙ͓ͯ͘͠ 28