モンストのマルチクラウドについて / sre-lounge-at-xflag

Transcript

1. SRE Lounge at XFLAG XFLAG ։ൃຊ෦ XFLAGࣄۀਪਐࣨ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ

2. ࣗݾ঺հ ‣XFLAGࣄۀਪਐࣨ SREάϧʔϓॴଐ ‣ϛΫγΟʹ͸2007೥຤ʹೖࣾ ‣ۀ຿಺༰ ‣ϞϯελʔετϥΠΫ ‣೔ຊ൛/ւ֎൛ ‣Πϯϑϥؔ࿈ͷۀ຿ ‣ͦͷଞ͍Ζ͍Ζ 2

3. XFLAG ελδΦʹ͓͚ΔSRE 3

4. XFLAG ελδΦʹ͓͚ΔSRE ‣2016೥7݄ʹSREάϧʔϓ૑ઃ ‣ਓ਺͸7໊(2018-05-10ݱࡏ) ‣৽ଔͱத్ೖ͕ࣾ൒ʑ͘Β͍ ‣ϑϧελοΫΛٻΊ͍ͯͳ͍ ‣΍Δ΂͖࢓ࣄ͸ࣗ෼Ͱݟ͚ͭɺೳಈతʹ࣮ߦ͢Δ ‣ίϛϡχέʔγϣϯ͸Slack, Githubͷissueத৺ 4

5. XFLAG ελδΦʹ͓͚ΔSRE ‣Google ͷఏএ͢Δ͍ΘΏΔSRE ͱ͸ҧ͏΋ͷ΋ ‣৽نαʔϏεͷ։ൃΛͨ͠Γ ‣αʔϏε֎Ͱͷ։ൃɾߏஙͨ͠Γ 5

6. XFLAG ελδΦʹ͓͚ΔSRE SREͷϛογϣϯͱ͍͑͹ 6

7. XFLAG ελδΦʹ͓͚ΔSRE αʔϏεͷ҆ఆԽͷͨΊ ίʔυʹΑΔෛՙ࡟ݮɾࣗಈԽ 7

8. XFLAG ελδΦʹ͓͚ΔSRE ͦΕ͚ͩͰ͸ͳ͍ 8

9. XFLAG ελδΦʹ͓͚ΔSRE ຊ೔͸SREͷࣄྫͷҰͭͱͯ͠ ϞϯετͷϚϧνΫϥ΢υӡ༻ͷ͓࿩ 9

10. Ϟϯετͷߏ੒ 10

11. Ϟϯετͷߏ੒(γεςϜ) ‣ΞϓϦέʔγϣϯ ‣Ruby(padrino) + unicorn + Nginx ‣σʔλϕʔε ‣MariaDB 5.5

    ‣Batch ‣Rescue + redis ‣Ωϟογϡ ‣memcached ‣ϩά ‣Fluentd 11

12. Ϟϯετͷߏ੒(γεςϜ) 12 Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer

13. Ϟϯετͷߏ੒(਺ࣈ) ‣ΞϓϦέʔγϣϯ ‣ίΞ਺: ໿ 15,000 ‣σʔλϕʔε ‣୆਺: 144୆ ‣େྔͷϝϞϦʹ ioMemoryΛੵΜͩϚγϯ΋͋Δ

    ‣memcached ‣୆਺: 40୆ ‣૯༰ྔ 1.2TB 13

14. Ϟϯετͷߏ੒(Πϯϑϥ) 14 DataCenter 1 • DB • memcached DataCenter 2

    • DB • memcached application application application application Cloud1 Cloud2 Cloud3 Cloud4 Internet

15. ͜͏ͳΔ·Ͱ 15

16. Πϯϑϥߏ੒ͷྺ࢙ ‣Ϟϯετ͸ΦϯϓϨͰӡ༻͞Εͯͨ ‣εϚϗήʔϜͷಛੑͱͯ͠ ‣ΠϕϯτͰരൃతʹΞΫηε૿ 16

17. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ΞϓϦέʔγϣϯαʔόΛҰ࣌తʹ૿ڧ͍ͨ͠ ‣DB͸σʔληϯλʹ͋Δ΋ͷΛࢀর͍ͨ͠ 17

18. Πϯϑϥߏ੒ͷྺ࢙ ‣AWS ͷ EC2 ΠϯελϯεΛར༻ ‣c4.4xlargeλΠϓ 18

19. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ϩοΫΠϯ͸ආ͚͍ͨ ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 19

20. Πϯϑϥߏ੒ͷྺ࢙ ‣GMO ΞϓϦΫϥ΢υΛར༻ ‣Xeon E5-2690v3 λΠϓ 20

21. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣प೥΍೥຤೥࢝ͷΠϕϯτ ‣େྔʹΞΫηε͕དྷͦ͏(1೔͚ͩ) ‣ίετύϑΥʔϚϯε ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 21

22. Πϯϑϥߏ੒ͷྺ࢙ ‣IBMΞϓϦΫϥ΢υͷϕΞϝλϧϚγϯ ‣Xeon E5-2690V4 λΠϓ ‣Google Cloud ͷCompute Engine ‣n1-highcpu-64

    λΠϓ 22

23. ϚϧνΫϥ΢υӡ༻ 23

24. ϚϧνΫϥ΢υͷӡ༻ ‣γϯϓϧʹ ‣֤Ϋϥ΢υͰΠϝʔδ(AMI౳)Λ࡞੒ ‣ϓϩϏδϣχϯά͸ansible, Chef ͳͲར༻ ‣؂ࢹͳͲ͸ΦϯϓϨɾΫϥ΢υࠞࡏ ‣Nagios ‣CloudForecast ͳͲ

    24

25. ϚϧνΫϥ΢υͷӡ༻ ➡ϕʔεͷΠϝʔδ͔Βىಈ ➡ansible Λద༻͢Δ ➡Ϣʔβͷ௥ՃɺωοτϫʔΫͷઃఆetc ➡chef Λద༻ ➡ύοέʔδͷ௥Ճɺconfig ௐ੔ 25

26. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υຖͷAPIʹ߹Θͤͨࣾ಺CLIπʔϧ ‣Πϯελϯεͷ࡞੒ɾҰཡ ‣Πϯελϯεͷ࡟আ ‣Πϝʔδͷ࡞੒ɾҰཡ 26

27. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υɾσʔληϯλؒͷωοτϫʔΫ ‣μΠϨΫτίωΫτ઀ଓ ‣10G × n ຊ ‣ωοτϫʔΫνʔϜͷ͓͔͛ 27

28. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υͰͷ഑෼ɾόϥϯεௐ੔ ‣ҰՕॴ͕ଟ͘ͳΓ͗ͨ͢Γ͸͠ͳ͍Α͏ʹ ‣ఆظతʹόϥϯεௐ੔Λߦ͏ ‣ਓྗͰ ‣ಀ͛ಓ͸༻ҙ͓ͯ͘͠ 28

29. ϚϧνΫϥ΢υͷӡ༻ ιϑτ΢ΣΞ(ίʔυ)͚ͩͰ͸ͳ͘ ϋʔυ΢ΣΞͰ΋ 29

30. ϚϧνΫϥ΢υͷӡ༻ ͦΕ͕ Site Reliability ͷͨΊͳΒ 30

31. ·ͱΊ 31

32. ·ͱΊ ‣ XFLAG ͷSRE ͷ͝঺հ ‣ ϞϯετͷϚϧνΫϥ΢υར༻ʹ͍ͭͯ 32

33. ·ͱΊ ࣭໰ɾৄ͘͠͸࠙਌ձͰʂ 33

34. Thank you!