モンストのマルチクラウドについて / sre-lounge-at-xflag

モンストのマルチクラウドについて / sre-lounge-at-xflag

SRE Lounge #3 での発表資料になります

B276b9088550bc522536ab9d471aeebe?s=128

Tomohiro Koike

May 18, 2018
Tweet

Transcript

  1. SRE Lounge at XFLAG XFLAG ։ൃຊ෦ XFLAGࣄۀਪਐࣨ SREάϧʔϓ খ஑஌༟ גࣜձࣾϛΫγΟ

  2. ࣗݾ঺հ ‣XFLAGࣄۀਪਐࣨ SREάϧʔϓॴଐ ‣ϛΫγΟʹ͸2007೥຤ʹೖࣾ ‣ۀ຿಺༰ ‣ϞϯελʔετϥΠΫ ‣೔ຊ൛/ւ֎൛ ‣Πϯϑϥؔ࿈ͷۀ຿ ‣ͦͷଞ͍Ζ͍Ζ 2

  3. XFLAG ελδΦʹ͓͚ΔSRE 3

  4. XFLAG ελδΦʹ͓͚ΔSRE ‣2016೥7݄ʹSREάϧʔϓ૑ઃ ‣ਓ਺͸7໊(2018-05-10ݱࡏ) ‣৽ଔͱத్ೖ͕ࣾ൒ʑ͘Β͍ ‣ϑϧελοΫΛٻΊ͍ͯͳ͍ ‣΍Δ΂͖࢓ࣄ͸ࣗ෼Ͱݟ͚ͭɺೳಈతʹ࣮ߦ͢Δ ‣ίϛϡχέʔγϣϯ͸Slack, Githubͷissueத৺ 4

  5. XFLAG ελδΦʹ͓͚ΔSRE ‣Google ͷఏএ͢Δ͍ΘΏΔSRE ͱ͸ҧ͏΋ͷ΋ ‣৽نαʔϏεͷ։ൃΛͨ͠Γ ‣αʔϏε֎Ͱͷ։ൃɾߏஙͨ͠Γ 5

  6. XFLAG ελδΦʹ͓͚ΔSRE SREͷϛογϣϯͱ͍͑͹ 6

  7. XFLAG ελδΦʹ͓͚ΔSRE αʔϏεͷ҆ఆԽͷͨΊ ίʔυʹΑΔෛՙ࡟ݮɾࣗಈԽ 7

  8. XFLAG ελδΦʹ͓͚ΔSRE ͦΕ͚ͩͰ͸ͳ͍ 8

  9. XFLAG ελδΦʹ͓͚ΔSRE ຊ೔͸SREͷࣄྫͷҰͭͱͯ͠ ϞϯετͷϚϧνΫϥ΢υӡ༻ͷ͓࿩ 9

  10. Ϟϯετͷߏ੒ 10

  11. Ϟϯετͷߏ੒(γεςϜ) ‣ΞϓϦέʔγϣϯ ‣Ruby(padrino) + unicorn + Nginx ‣σʔλϕʔε ‣MariaDB 5.5

    ‣Batch ‣Rescue + redis ‣Ωϟογϡ ‣memcached ‣ϩά ‣Fluentd 11
  12. Ϟϯετͷߏ੒(γεςϜ) 12 Unicorn memcached MariaDB Redis Fluetnd resque worker LoadBalancer

  13. Ϟϯετͷߏ੒(਺ࣈ) ‣ΞϓϦέʔγϣϯ ‣ίΞ਺: ໿ 15,000 ‣σʔλϕʔε ‣୆਺: 144୆ ‣େྔͷϝϞϦʹ ioMemoryΛੵΜͩϚγϯ΋͋Δ

    ‣memcached ‣୆਺: 40୆ ‣૯༰ྔ 1.2TB 13
  14. Ϟϯετͷߏ੒(Πϯϑϥ) 14 DataCenter 1 • DB • memcached DataCenter 2

    • DB • memcached application application application application Cloud1 Cloud2 Cloud3 Cloud4 Internet
  15. ͜͏ͳΔ·Ͱ 15

  16. Πϯϑϥߏ੒ͷྺ࢙ ‣Ϟϯετ͸ΦϯϓϨͰӡ༻͞Εͯͨ ‣εϚϗήʔϜͷಛੑͱͯ͠ ‣ΠϕϯτͰരൃతʹΞΫηε૿ 16

  17. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ΞϓϦέʔγϣϯαʔόΛҰ࣌తʹ૿ڧ͍ͨ͠ ‣DB͸σʔληϯλʹ͋Δ΋ͷΛࢀর͍ͨ͠ 17

  18. Πϯϑϥߏ੒ͷྺ࢙ ‣AWS ͷ EC2 ΠϯελϯεΛར༻ ‣c4.4xlargeλΠϓ 18

  19. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣ϩοΫΠϯ͸ආ͚͍ͨ ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 19

  20. Πϯϑϥߏ੒ͷྺ࢙ ‣GMO ΞϓϦΫϥ΢υΛར༻ ‣Xeon E5-2690v3 λΠϓ 20

  21. Πϯϑϥߏ੒ͷྺ࢙ ‣ཁ๬ ‣प೥΍೥຤೥࢝ͷΠϕϯτ ‣େྔʹΞΫηε͕དྷͦ͏(1೔͚ͩ) ‣ίετύϑΥʔϚϯε ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ 21

  22. Πϯϑϥߏ੒ͷྺ࢙ ‣IBMΞϓϦΫϥ΢υͷϕΞϝλϧϚγϯ ‣Xeon E5-2690V4 λΠϓ ‣Google Cloud ͷCompute Engine ‣n1-highcpu-64

    λΠϓ 22
  23. ϚϧνΫϥ΢υӡ༻ 23

  24. ϚϧνΫϥ΢υͷӡ༻ ‣γϯϓϧʹ ‣֤Ϋϥ΢υͰΠϝʔδ(AMI౳)Λ࡞੒ ‣ϓϩϏδϣχϯά͸ansible, Chef ͳͲར༻ ‣؂ࢹͳͲ͸ΦϯϓϨɾΫϥ΢υࠞࡏ ‣Nagios ‣CloudForecast ͳͲ

    24
  25. ϚϧνΫϥ΢υͷӡ༻ ➡ϕʔεͷΠϝʔδ͔Βىಈ ➡ansible Λద༻͢Δ ➡Ϣʔβͷ௥ՃɺωοτϫʔΫͷઃఆetc ➡chef Λద༻ ➡ύοέʔδͷ௥Ճɺconfig ௐ੔ 25

  26. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υຖͷAPIʹ߹Θͤͨࣾ಺CLIπʔϧ ‣Πϯελϯεͷ࡞੒ɾҰཡ ‣Πϯελϯεͷ࡟আ ‣Πϝʔδͷ࡞੒ɾҰཡ 26

  27. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υɾσʔληϯλؒͷωοτϫʔΫ ‣μΠϨΫτίωΫτ઀ଓ ‣10G × n ຊ ‣ωοτϫʔΫνʔϜͷ͓͔͛ 27

  28. ϚϧνΫϥ΢υͷӡ༻ ‣֤Ϋϥ΢υͰͷ഑෼ɾόϥϯεௐ੔ ‣ҰՕॴ͕ଟ͘ͳΓ͗ͨ͢Γ͸͠ͳ͍Α͏ʹ ‣ఆظతʹόϥϯεௐ੔Λߦ͏ ‣ਓྗͰ ‣ಀ͛ಓ͸༻ҙ͓ͯ͘͠ 28

  29. ϚϧνΫϥ΢υͷӡ༻ ιϑτ΢ΣΞ(ίʔυ)͚ͩͰ͸ͳ͘ ϋʔυ΢ΣΞͰ΋ 29

  30. ϚϧνΫϥ΢υͷӡ༻ ͦΕ͕ Site Reliability ͷͨΊͳΒ 30

  31. ·ͱΊ 31

  32. ·ͱΊ ‣ XFLAG ͷSRE ͷ͝঺հ ‣ ϞϯετͷϚϧνΫϥ΢υར༻ʹ͍ͭͯ 32

  33. ·ͱΊ ࣭໰ɾৄ͘͠͸࠙਌ձͰʂ 33

  34. Thank you!