Upgrade to Pro — share decks privately, control downloads, hide ads and more …

モンストのマルチクラウドについて / sre-lounge-at-xflag

モンストのマルチクラウドについて / sre-lounge-at-xflag

SRE Lounge #3 での発表資料になります

Tomohiro Koike

May 18, 2018
Tweet

More Decks by Tomohiro Koike

Other Decks in Technology

Transcript

  1. SRE Lounge at XFLAG
    XFLAG ։ൃຊ෦ XFLAGࣄۀਪਐࣨ SREάϧʔϓ
    খ஑஌༟
    גࣜձࣾϛΫγΟ

    View full-size slide

  2. ࣗݾ঺հ
    ‣XFLAGࣄۀਪਐࣨ SREάϧʔϓॴଐ
    ‣ϛΫγΟʹ͸2007೥຤ʹೖࣾ
    ‣ۀ຿಺༰
    ‣ϞϯελʔετϥΠΫ
    ‣೔ຊ൛/ւ֎൛
    ‣Πϯϑϥؔ࿈ͷۀ຿
    ‣ͦͷଞ͍Ζ͍Ζ
    2

    View full-size slide

  3. XFLAG ελδΦʹ͓͚ΔSRE
    3

    View full-size slide

  4. XFLAG ελδΦʹ͓͚ΔSRE
    ‣2016೥7݄ʹSREάϧʔϓ૑ઃ
    ‣ਓ਺͸7໊(2018-05-10ݱࡏ)
    ‣৽ଔͱத్ೖ͕ࣾ൒ʑ͘Β͍
    ‣ϑϧελοΫΛٻΊ͍ͯͳ͍
    ‣΍Δ΂͖࢓ࣄ͸ࣗ෼Ͱݟ͚ͭɺೳಈతʹ࣮ߦ͢Δ
    ‣ίϛϡχέʔγϣϯ͸Slack, Githubͷissueத৺
    4

    View full-size slide

  5. XFLAG ελδΦʹ͓͚ΔSRE
    ‣Google ͷఏএ͢Δ͍ΘΏΔSRE ͱ͸ҧ͏΋ͷ΋
    ‣৽نαʔϏεͷ։ൃΛͨ͠Γ
    ‣αʔϏε֎Ͱͷ։ൃɾߏஙͨ͠Γ
    5

    View full-size slide

  6. XFLAG ελδΦʹ͓͚ΔSRE
    SREͷϛογϣϯͱ͍͑͹
    6

    View full-size slide

  7. XFLAG ελδΦʹ͓͚ΔSRE
    αʔϏεͷ҆ఆԽͷͨΊ
    ίʔυʹΑΔෛՙ࡟ݮɾࣗಈԽ
    7

    View full-size slide

  8. XFLAG ελδΦʹ͓͚ΔSRE
    ͦΕ͚ͩͰ͸ͳ͍
    8

    View full-size slide

  9. XFLAG ελδΦʹ͓͚ΔSRE
    ຊ೔͸SREͷࣄྫͷҰͭͱͯ͠
    ϞϯετͷϚϧνΫϥ΢υӡ༻ͷ͓࿩
    9

    View full-size slide

  10. Ϟϯετͷߏ੒
    10

    View full-size slide

  11. Ϟϯετͷߏ੒(γεςϜ)
    ‣ΞϓϦέʔγϣϯ
    ‣Ruby(padrino) + unicorn + Nginx
    ‣σʔλϕʔε
    ‣MariaDB 5.5
    ‣Batch
    ‣Rescue + redis
    ‣Ωϟογϡ
    ‣memcached
    ‣ϩά
    ‣Fluentd
    11

    View full-size slide

  12. Ϟϯετͷߏ੒(γεςϜ)
    12
    Unicorn
    memcached
    MariaDB
    Redis
    Fluetnd
    resque worker
    LoadBalancer

    View full-size slide

  13. Ϟϯετͷߏ੒(਺ࣈ)
    ‣ΞϓϦέʔγϣϯ
    ‣ίΞ਺: ໿ 15,000
    ‣σʔλϕʔε
    ‣୆਺: 144୆
    ‣େྔͷϝϞϦʹ ioMemoryΛੵΜͩϚγϯ΋͋Δ
    ‣memcached
    ‣୆਺: 40୆
    ‣૯༰ྔ 1.2TB
    13

    View full-size slide

  14. Ϟϯετͷߏ੒(Πϯϑϥ)
    14
    DataCenter 1
    • DB
    • memcached
    DataCenter 2
    • DB
    • memcached
    application application application application
    Cloud1 Cloud2 Cloud3 Cloud4
    Internet

    View full-size slide

  15. ͜͏ͳΔ·Ͱ
    15

    View full-size slide

  16. Πϯϑϥߏ੒ͷྺ࢙
    ‣Ϟϯετ͸ΦϯϓϨͰӡ༻͞Εͯͨ
    ‣εϚϗήʔϜͷಛੑͱͯ͠
    ‣ΠϕϯτͰരൃతʹΞΫηε૿
    16

    View full-size slide

  17. Πϯϑϥߏ੒ͷྺ࢙
    ‣ཁ๬
    ‣ΞϓϦέʔγϣϯαʔόΛҰ࣌తʹ૿ڧ͍ͨ͠
    ‣DB͸σʔληϯλʹ͋Δ΋ͷΛࢀর͍ͨ͠
    17

    View full-size slide

  18. Πϯϑϥߏ੒ͷྺ࢙
    ‣AWS ͷ EC2 ΠϯελϯεΛར༻
    ‣c4.4xlargeλΠϓ
    18

    View full-size slide

  19. Πϯϑϥߏ੒ͷྺ࢙
    ‣ཁ๬
    ‣ϩοΫΠϯ͸ආ͚͍ͨ
    ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ
    19

    View full-size slide

  20. Πϯϑϥߏ੒ͷྺ࢙
    ‣GMO ΞϓϦΫϥ΢υΛར༻
    ‣Xeon E5-2690v3 λΠϓ
    20

    View full-size slide

  21. Πϯϑϥߏ੒ͷྺ࢙
    ‣ཁ๬
    ‣प೥΍೥຤೥࢝ͷΠϕϯτ
    ‣େྔʹΞΫηε͕དྷͦ͏(1೔͚ͩ)
    ‣ίετύϑΥʔϚϯε
    ‣ϨΠςϯγͷྑ͍΋ͷΛ࢖͍͍ͨ
    21

    View full-size slide

  22. Πϯϑϥߏ੒ͷྺ࢙
    ‣IBMΞϓϦΫϥ΢υͷϕΞϝλϧϚγϯ
    ‣Xeon E5-2690V4 λΠϓ
    ‣Google Cloud ͷCompute Engine
    ‣n1-highcpu-64 λΠϓ
    22

    View full-size slide

  23. ϚϧνΫϥ΢υӡ༻
    23

    View full-size slide

  24. ϚϧνΫϥ΢υͷӡ༻
    ‣γϯϓϧʹ
    ‣֤Ϋϥ΢υͰΠϝʔδ(AMI౳)Λ࡞੒
    ‣ϓϩϏδϣχϯά͸ansible, Chef ͳͲར༻
    ‣؂ࢹͳͲ͸ΦϯϓϨɾΫϥ΢υࠞࡏ
    ‣Nagios
    ‣CloudForecast ͳͲ
    24

    View full-size slide

  25. ϚϧνΫϥ΢υͷӡ༻
    ➡ϕʔεͷΠϝʔδ͔Βىಈ
    ➡ansible Λద༻͢Δ
    ➡Ϣʔβͷ௥ՃɺωοτϫʔΫͷઃఆetc
    ➡chef Λద༻
    ➡ύοέʔδͷ௥Ճɺconfig ௐ੔
    25

    View full-size slide

  26. ϚϧνΫϥ΢υͷӡ༻
    ‣֤Ϋϥ΢υຖͷAPIʹ߹Θͤͨࣾ಺CLIπʔϧ
    ‣Πϯελϯεͷ࡞੒ɾҰཡ
    ‣Πϯελϯεͷ࡟আ
    ‣Πϝʔδͷ࡞੒ɾҰཡ
    26

    View full-size slide

  27. ϚϧνΫϥ΢υͷӡ༻
    ‣֤Ϋϥ΢υɾσʔληϯλؒͷωοτϫʔΫ
    ‣μΠϨΫτίωΫτ઀ଓ
    ‣10G × n ຊ
    ‣ωοτϫʔΫνʔϜͷ͓͔͛
    27

    View full-size slide

  28. ϚϧνΫϥ΢υͷӡ༻
    ‣֤Ϋϥ΢υͰͷ഑෼ɾόϥϯεௐ੔
    ‣ҰՕॴ͕ଟ͘ͳΓ͗ͨ͢Γ͸͠ͳ͍Α͏ʹ
    ‣ఆظతʹόϥϯεௐ੔Λߦ͏
    ‣ਓྗͰ
    ‣ಀ͛ಓ͸༻ҙ͓ͯ͘͠
    28

    View full-size slide

  29. ϚϧνΫϥ΢υͷӡ༻
    ιϑτ΢ΣΞ(ίʔυ)͚ͩͰ͸ͳ͘
    ϋʔυ΢ΣΞͰ΋
    29

    View full-size slide

  30. ϚϧνΫϥ΢υͷӡ༻
    ͦΕ͕ Site Reliability ͷͨΊͳΒ
    30

    View full-size slide

  31. ·ͱΊ
    ‣ XFLAG ͷSRE ͷ͝঺հ
    ‣ ϞϯετͷϚϧνΫϥ΢υར༻ʹ͍ͭͯ
    32

    View full-size slide

  32. ·ͱΊ
    ࣭໰ɾৄ͘͠͸࠙਌ձͰʂ
    33

    View full-size slide