モンストのマルチクラウドについて / sre-lounge-at-xflag

SRE Lounge #3 での発表資料になります

Transcript

1. SRE Lounge at XFLAG XFLAG ։ൃຊ෦ XFLAGࣄۀਪਐࣨ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ
2. ࣗݾ঺հ ‣XFLAGࣄۀਪਐࣨ SREάϧʔϓॴଐ ‣ϛΫγΟʹ͸2007೥຤ʹೖࣾ ‣ۀ຿಺༰ ‣ϞϯελʔετϥΠΫ ‣೔ຊ൛/ւ֎൛ ‣Πϯϑϥؔ࿈ͷۀ຿ ‣ͦͷଞ͍Ζ͍Ζ 2
3. XFLAG ελδΦʹ͓͚ΔSRE 3
4. XFLAG ελδΦʹ͓͚ΔSRE ‣2016೥7݄ʹSREάϧʔϓ૑ઃ ‣ਓ਺͸7໊(2018-05-10ݱࡏ) ‣৽ଔͱத్ೖ͕ࣾ൒ʑ͘Β͍ ‣ϑϧελοΫΛٻΊ͍ͯͳ͍ ‣΍Δ΂͖࢓ࣄ͸ࣗ෼Ͱݟ͚ͭɺೳಈతʹ࣮ߦ͢Δ ‣ίϛϡχέʔγϣϯ͸Slack, Githubͷissueத৺ 4
5. XFLAG ελδΦʹ͓͚ΔSRE ‣Google ͷఏএ͢Δ͍ΘΏΔSRE ͱ͸ҧ͏΋ͷ΋ ‣৽نαʔϏεͷ։ൃΛͨ͠Γ ‣αʔϏε֎Ͱͷ։ൃɾߏஙͨ͠Γ 5
6. XFLAG ελδΦʹ͓͚ΔSRE SREͷϛογϣϯͱ͍͑͹ 6
7. XFLAG ελδΦʹ͓͚ΔSRE αʔϏεͷ҆ఆԽͷͨΊ ίʔυʹΑΔෛՙ࡟ݮɾࣗಈԽ 7
8. XFLAG ελδΦʹ͓͚ΔSRE ͦΕ͚ͩͰ͸ͳ͍ 8
9. XFLAG ελδΦʹ͓͚ΔSRE ຊ೔͸SREͷࣄྫͷҰͭͱͯ͠ ϞϯετͷϚϧνΫϥ΢υӡ༻ͷ͓࿩ 9
10. Ϟϯετͷߏ੒ 10
11. Ϟϯετͷߏ੒(γεςϜ) ‣ΞϓϦέʔγϣϯ ‣Ruby(padrino) + unicorn + Nginx ‣σʔλϕʔε ‣MariaDB 5.5 ‣Batch ‣Rescue + redis ‣Ωϟογϡ ‣memcached ‣ϩά ‣Fluentd 11
12. Ϟϯετͷߏ੒(γεςϜ) 12 Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer
13. Ϟϯετͷߏ੒(਺ࣈ) ‣ΞϓϦέʔγϣϯ ‣ίΞ਺: ໿ 15,000 ‣σʔλϕʔε ‣୆਺: 144୆ ‣େྔͷϝϞϦʹ ioMemoryΛੵΜͩϚγϯ΋͋Δ ‣memcached ‣୆਺: 40୆ ‣૯༰ྔ 1.2TB 13
14. Ϟϯετͷߏ੒(Πϯϑϥ) 14 DataCenter 1 • DB • memcached DataCenter 2 • DB • memcached application application application application Cloud1 Cloud2 Cloud3 Cloud4 Internet
15. ͜͏ͳΔ·Ͱ 15
16. Πϯϑϥߏ੒ͷྺ࢙ ‣Ϟϯετ͸ΦϯϓϨͰӡ༻͞Εͯͨ ‣εϚϗήʔϜͷಛੑͱͯ͠ ‣ΠϕϯτͰരൃతʹΞΫηε૿ 16
17. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ΞϓϦέʔγϣϯαʔόΛҰ࣌తʹ૿ڧ͍ͨ͠ ‣DB͸σʔληϯλʹ͋Δ΋ͷΛࢀর͍ͨ͠ 17
18. Πϯϑϥߏ੒ͷྺ࢙ ‣AWS ͷ EC2 ΠϯελϯεΛར༻ ‣c4.4xlargeλΠϓ 18
19. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ϩοΫΠϯ͸ආ͚͍ͨ ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 19
20. Πϯϑϥߏ੒ͷྺ࢙ ‣GMO ΞϓϦΫϥ΢υΛར༻ ‣Xeon E5-2690v3 λΠϓ 20
21. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣प೥΍೥຤೥࢝ͷΠϕϯτ ‣େྔʹΞΫηε͕དྷͦ͏(1೔͚ͩ) ‣ίετύϑΥʔϚϯε ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 21
22. Πϯϑϥߏ੒ͷྺ࢙ ‣IBMΞϓϦΫϥ΢υͷϕΞϝλϧϚγϯ ‣Xeon E5-2690V4 λΠϓ ‣Google Cloud ͷCompute Engine ‣n1-highcpu-64 λΠϓ 22
23. ϚϧνΫϥ΢υӡ༻ 23
24. ϚϧνΫϥ΢υͷӡ༻ ‣γϯϓϧʹ ‣֤Ϋϥ΢υͰΠϝʔδ(AMI౳)Λ࡞੒ ‣ϓϩϏδϣχϯά͸ansible, Chef ͳͲར༻ ‣؂ࢹͳͲ͸ΦϯϓϨɾΫϥ΢υࠞࡏ ‣Nagios ‣CloudForecast ͳͲ 24
25. ϚϧνΫϥ΢υͷӡ༻ ➡ϕʔεͷΠϝʔδ͔Βىಈ ➡ansible Λద༻͢Δ ➡Ϣʔβͷ௥ՃɺωοτϫʔΫͷઃఆetc ➡chef Λద༻ ➡ύοέʔδͷ௥Ճɺconfig ௐ੔ 25
26. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υຖͷAPIʹ߹Θͤͨࣾ಺CLIπʔϧ ‣Πϯελϯεͷ࡞੒ɾҰཡ ‣Πϯελϯεͷ࡟আ ‣Πϝʔδͷ࡞੒ɾҰཡ 26
27. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υɾσʔληϯλؒͷωοτϫʔΫ ‣μΠϨΫτίωΫτ઀ଓ ‣10G × n ຊ ‣ωοτϫʔΫνʔϜͷ͓͔͛ 27
28. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υͰͷ഑෼ɾόϥϯεௐ੔ ‣ҰՕॴ͕ଟ͘ͳΓ͗ͨ͢Γ͸͠ͳ͍Α͏ʹ ‣ఆظతʹόϥϯεௐ੔Λߦ͏ ‣ਓྗͰ ‣ಀ͛ಓ͸༻ҙ͓ͯ͘͠ 28
29. ϚϧνΫϥ΢υͷӡ༻ ιϑτ΢ΣΞ(ίʔυ)͚ͩͰ͸ͳ͘ ϋʔυ΢ΣΞͰ΋ 29
30. ϚϧνΫϥ΢υͷӡ༻ ͦΕ͕ Site Reliability ͷͨΊͳΒ 30
31. ·ͱΊ 31
32. ·ͱΊ ‣ XFLAG ͷSRE ͷ͝঺հ ‣ ϞϯετͷϚϧνΫϥ΢υར༻ʹ͍ͭͯ 32
33. ·ͱΊ ࣭໰ɾৄ͘͠͸࠙਌ձͰʂ 33
34. Thank you!