Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWSのオートスケールとなかよく付き合う

 AWSのオートスケールとなかよく付き合う

YAP(achimon)C::Asia Hachioji 2016 mid

FUJIWARA Shunichiro

July 02, 2016
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. AWSͷΦʔτεέʔϧͱ
    ͳ͔Α͘෇͖߹͏
    YAP(achimon)C::Asia Hachioji
    2016 mid
    2016-07-02 @fujiwara

    View Slide

  2. @fujiwara
    github.com/fujiwara
    sfujiwara.hatenablog.com
    ٕज़෦

    View Slide

  3. Agenda
    • ΦʔτεέʔϧͬͯԿ
    • ͳͥΦʔτεέʔϧ͢Δͷ͔
    • ͦ΋ͦ΋Φʔτεέʔϧ͢Δ΂͖?
    • Ͳ͏͍͏৚݅ͰΦʔτεέʔϧΛൃಈ͢Δ͔
    • ຊ౰ʹى͖ͨා͍࿩

    View Slide

  4. Φʔτεέʔϧͱ͸

    View Slide

  5. AWSͷΦʔτεέʔϧ
    ΞϓϦέʔγϣϯͷෛՙΛॲཧ͢ΔͨΊʹ
    ద੾ͳ਺ͷEC2ΠϯελϯεΛىಈ͢Δ࢓૊Έ

    View Slide

  6. ͳͥΦʔτεέʔϧ͢Δͷ͔

    View Slide

  7. ͳͥΦʔτεέʔϧ͢Δͷ͔
    ΞΫηε਺(≒ෛՙ)͸1೔ͷதͰมಈ͢Δ

    View Slide

  8. ͳͥΦʔτεέʔϧ͢Δͷ͔
    min : avg : max = 200 : 1000 : 3000
    ϐʔΫ࣌ͷෛՙ͸1೔ฏۉͷ3ഒ
    ϐʔΫΛ૝ఆͨ͠ΠϯελϯεΛৗ࣌ىಈ͢Δͱ !
    ෛՙʹԠͯ͡Πϯελϯε୆਺Λௐ੔͢Ε͹અ໿Ͱ͖Δ

    View Slide

  9. ͳͥΦʔτεέʔϧ͢Δͷ͔
    ʰಥൃతͳෛՙʹ΋ରԠͰ͖Δʱ
    • ͋Δఔ౓·Ͱԣʹ૿΍ͤΔ΋ͷͰ͋Ε͹ ⭕
    • ྫ͑͹RDS͕ϘτϧωοΫʹͳΔͱ… ❓
    • ෛՙͷ૿େ͕଎͗͢Δͱؒʹ߹Θͳ͍

    View Slide

  10. ͳͥΦʔτεέʔϧ͢Δͷ͔
    ͦͷଞʹ΋خ͍͠(ؾΛ͚ͭΔ)͜ͱ
    • ো֐͕ൃੜͨ͠Πϯελϯε͸ࣗಈͰ࡟আ(terminate)
    • ৽͍͠Πϯελϯε͕ࣗಈͰىಈ
    • ઃఆ͞Εͨ୆਺Λอͱ͏ͱͯ͘͠ΕΔ
    ! Πϯελϯεো֐Ͱ࠶ىಈෆཁ
    " ࡟আ͞ΕͨΠϯελϯεͷEBS͸ফ໓

    View Slide

  11. ͦ΋ͦ΋Φʔτεέʔϧ͢Δ΂͖ʁ

    View Slide

  12. ͦ΋ͦ΋Φʔτεέʔϧ͢Δ΂͖ʁ
    16ίΞϚγϯ͕ϐʔΫͰ30୆ඞཁͳγεςϜ
    ! c4.4xlarge(16 cores) $776.66/month
    • avg: × 10 = $7,766 / month
    • max: x 30 = $23,299 / month
    ฏۉ10୆ʹ཈͑ΒΕΕ͹ֹࠩ͸20୆෼ $15,533 ≒ 150ສԁ
    ͋Δఔ౓ͷن໛͕͋Ε͹࡟ݮֹ͕େ͖͍

    View Slide

  13. ن໛͕ͦΕ΄ͲͰ΋ͳ͍৔߹
    4ίΞϚγϯ͕ϐʔΫͰ6୆ඞཁͳγεςϜ
    ! c4.xlarge(4 cores) $193.98/month
    • avg: × 4 = $776 / month
    • max: x 6 = $1,164 / month
    ࠷௿3୆͸ಈ͔͍ͨ͠ͷͰavg = max / 3 ʹ͸ͳΒͳ͍
    ֹࠩ͸2୆෼ $387 ఔ౓ɺݻఆඅͱൺ΂Δͱ…

    View Slide

  14. View Slide

  15. ͦ΋ͦ΋Φʔτεέʔϧ͢Δ΂͖ʁ·ͱΊ
    ΦʔτεέʔϧΛߟྀ͍ͯ͠ͳ͍γεςϜʹޙ͔Βಋೖ͢Δʹ
    ͸͍Ζ͍Ζඞཁ
    ӡ༻த΋҆ఆ͢Δ·Ͱ͸͍Ζ͍Ζ͋Δ
    ಋೖɺӡ༻ίετ͕࡟ݮίετʹݟ߹͏͔Ͳ͏͔൑அ

    View Slide

  16. Auto Scaling ུ֓

    View Slide

  17. Auto Scaling Group
    Πϯελϯεͷ·ͱ·Γ͝ͱʹ Auto
    Scaling GroupΛఆٛ
    • ୆਺ͷ࠷খ(min)ɺ࠷େ(max)ɺر๬
    (desired)
    • ؔ࿈͚ͮΔ ELB
    • ىಈઃఆ (Launch Configuration)
    • εέʔϦϯάϙϦγʔ

    View Slide

  18. ࠷খ୆਺ (min)
    ❌ ͜ͷ୆਺ҎԼʹ͸ݮΒͳ͍
    ⭕ ͜ͷ୆਺Λ࠷௿ݶҡ࣋͠Α͏ͱ͢Δ
    min: 3, ݱࡏ3୆ͷঢ়ଶͰ1୆͕ϔϧενΣοΫࣦഊ
    → ৽͍͠Πϯελϯε͕ىಈ͞ΕΔ
    → ࣦഊͨ͠Πϯελϯε͕࡟আ͞ΕΔ
    Ұ࣌తʹ͸minΛԼճΔ͜ͱ͕͋Δ

    View Slide

  19. ر๬୆਺ (desired)
    ͜ͷ୆਺Λҡ࣋͠Α͏ͱ͢Δ
    εέʔϦϯάϙϦγʔʹΑͬͯ desired ͕มߋ͞ΕΔ
    → Πϯελϯε͕ىಈ/࡟আ͞ΕΔ
    → desiredΛຬͨͨ͠Β׬ྃ
    ىಈࣦഊͨ͠৔߹͸ɺdesiredΛຬͨ͢·ͰϦτϥΠ
    ͏·͘ىಈ͠ͳ͍AMIΛ࡞ͬͯ͠·͏ͱӬԕʹ…!

    View Slide

  20. ࠷େ୆਺ (max)
    ͜ͷ୆਺Ҏ্͸ىಈ͠ͳ͍
    ઃఆϛεͰͷىಈ͗͢͠Λ๷ࢭͰ͖Δ
    ෛՙ૿େ࣌ʹఱҪʹ౰ͨΔՄೳੑ͕͋ΔͷͰؾΛ͚͓ͭͯ͘

    View Slide

  21. εέʔϦϯάϙϦγʔ
    Ͳ͏͍͏৚݅ͰdesiredΛ૿Ճɺݮগͤ͞Δ͔ɺͱ͍͏ఆٛ
    CloudWatchͷϝτϦΫεΛݩʹͨ͠ΞϥʔϜΛτϦΨʹ͢Δ
    • Auto Scaling Group಺ͷEC2 CPU࢖༻཰
    • ओʹWebΞϓϦέʔγϣϯ޲͚
    • Amazon SQS ͷΩϡʔʹཷ·͍ͬͯΔ਺
    • ඇಉظδϣϒॲཧαʔό(ಈըม׵ͳͲ)޲͚

    View Slide

  22. spot or on demand?
    ΦϯσϚϯυΠϯελϯε͔εϙοτΠϯελϯε͔
    Φʔτεέʔϧʹ͸ͲͪΒͰ΋ར༻Մೳ(LaunchConfigͰࢦఆ)
    ΦϯσϚϯυΠϯελϯε
    ͍ͭͰ΋ಉҰՁ֨ͰىಈͰ͖Δ
    (εϙοτͱൺ΂Δͱ)ߴ͍
    WebΞϓϦέʔγϣϯαʔό޲͖

    View Slide

  23. spot or on demand?
    εϙοτΠϯελϯε
    ಈ࡞தͷεϙοτՁ֨ (धཁͱڙڅͰܾఆ)Ͱֹ͕ܾۚ·Δ
    ͍҆ (͍͍ͨͯͷ৔߹ΦϯσϚϯυͷ1/5ʙ1/10)
    ߴಅ͢Δ͜ͱ͕͋Δ (ΦϯσϚϯυՁ֨ͷ਺ഒ΋)
    ೖࡳՁ֨ΛεϙοτՁ্͕֨ճΔͱΠϯελϯε࡟আ
    → શ໓͢ΔՄೳੑ͕͋Δ
    ଟগͷॲཧ஗Ԇ͕ڐ͞ΕΔδϣϒϫʔΧʔ޲͖

    View Slide

  24. View Slide

  25. γεςϜΛΦʔτεέʔϧʹ
    ରԠͤ͞Δ

    View Slide

  26. γεςϜΛΦʔτεέʔϧʹରԠͤ͞Δ
    ʮAMI͔Βىಈ͢Δ͚ͩͰɺਖ਼ৗʹՔಇ͢Δঢ়ଶʹͳΔʯ
    ͜ͷঢ়ଶΛ࡞Βͳ͍ͱΦʔτεέʔϧʹରԠͰ͖ͳ͍
    • ϓϩϏδϣχϯά/σϓϩΠ
    • ؂ࢹ
    • ϩάసૹ

    View Slide

  27. ϓϩϏδϣχϯάɺσϓϩΠख๏
    ❌ σϓϩΠ͍ͨ͠λΠϛϯάͰ push
    ⭕ ىಈͨ͠Πϯελϯε͕ pull
    ΠϯελϯεΛىಈͨ࣌͠఺Ͱͷਓؒͷ࡞ۀΛෆཁʹ͢Δ

    View Slide

  28. AMIΛຖճ࡞Δ
    ϓϩϏδϣχϯάɺΞϓϦέʔγϣϯͷσϓϩΠ͕׬ྃͨ͠ঢ়
    ଶͰAMIΛ࡞Γ௚͢ˠ ىಈ͢Δ͚ͩͰOK
    ϝϦοτ
    • ىಈ͢Δ͚ͩɺىಈޙʹ͸Կ΋ඞཁͳ͍
    σϝϦοτ
    • ઃఆมߋɺΞϓϦέʔγϣϯमਖ਼ͰຖճAMIΛ࡞Γ௚͢खؒ
    • มߋ͝ͱʹΠϯελϯεΛશ෦ೖΕସ͑

    View Slide

  29. AMIΛຖճ࡞Δ
    • σϓϩΠස౓͕ଟ͘ͳ͍αʔϏεͳΒՄೳ
    (10ճ/೔͸೉͍͔͠΋…)
    • ݻఆͷΠϯελϯε͕͋Δ৔߹͸Ͳ͏͢Δ…ʁ
    (σʔλετΞͳͲ͕ϚωʔδυαʔϏεͰ͸ͳ͍৔߹)

    View Slide

  30. ϓϧܕͷϓϩϏδϣχϯάɺσϓϩΠख๏
    S3ͳͲʹΞʔΧΠϒΛอଘɺ֤ΠϯελϯεͰऔಘɺల։͢Δ
    AWS CodeDeploy
    Mamiya github.com/sorah/mamiya
    Stretcher github.com/fujiwara/stretcher

    View Slide

  31. ϓϧܕͷϓϩϏδϣχϯάɺσϓϩΠख๏
    Πϯελϯε͕ىಈ࣌ʹࣗΒ࠷৽ͷ΋ͷΛऔಘɺ࣮ߦ
    S3ΛόοΫΤϯυʹ͢Ε͹Մ༻ੑ΋ ⭕
    σϓϩΠ΋଎͍

    View Slide

  32. ⾠ rsyncͳͲͰϓϧܕσϓϩΠΛࣗ࡞͢Δ
    build్தͷத్൒୺ͳঢ়ଶ͕औಘ͞Εͳ͍Α͏஫ҙ
    → Πϯελϯεىಈத͸σϓϩΠ࡞ۀΛ͠ͳ͍(ͱ͔)
    औಘݩΛߴՄ༻/ߴੑೳʹ͢Δඞཁ͕͋Δ

    View Slide

  33. ⾠ Git͔Βऔಘͯ͠σϓϩΠ͢Δ
    ʮGit͸σϓϩΠπʔϧͰ͸͋Γ·ͤΜʯ
    খن໛ͳΒ໰୊ͳ͍
    ن໛͕େ͖͘ͳΔͱGitαʔόͷෛՙ͕ਂࠁ
    ৄ͘͠͸ YAPC::Asia 2015
    ੈքల։͢Δେن໛΢ΣϒαʔϏεͷσϓϩΠΛࢧ͑Δٕज़
    https://speakerdeck.com/hatena/yapc-asia-tokyo-2015

    View Slide

  34. γεςϜΛΦʔτεέʔϧʹରԠͤ͞Δ ·ͱΊ
    ىಈͨ͠Πϯελϯε͕ࣗ཯͢Δ
    ͋Δϗετ͔Βશ୆ʹ˓˓(ssh)ɺͱ͍͏ൃ૝Λ΍ΊΔ

    View Slide

  35. ؂ࢹ

    View Slide

  36. ؂ࢹͰඞཁͳ͜ͱ
    ಈతʹ؂ࢹର৅Λ૿ݮͰ͖ΔπʔϧΛ࢖͏
    agentΛىಈ͢Δ͚ͩͰ؂ࢹର৅ʹ௥Ճ
    agentىಈΛܖػʹࣗಈొ࿥ઃఆ͕Մೳ
    APIͰϗετొ࿥΋Մೳ

    View Slide

  37. ؂ࢹͰඞཁͳ͜ͱ
    ΦʔτεέʔϦϯάάϧʔϓશ୆Λू໿ͨ͠ϝτϦΫεΛΈΔ
    • CPU࢖༻཰ͳͲ

    View Slide

  38. ؂ࢹͰඞཁͳ͜ͱ
    1୆མ͙ͪͨΒ͍Ͱ͘Α͘Α͠ͳ͍
    େྔʹىಈ͢Ε͹མͪΔ΍ͭ΋͍Δ
    Πϯελϯεఀࢭ࣌ʹ؂ࢹఀࢭʹࣦഊ͢Δ͜ͱ΋͋Δ
    ZabbixͰͷτϦΨͷྫ
    grpavg["AutoScale","icmpping[]",last,0].last(0)<0.7
    ʮϗετάϧʔϓ಺Ͱping੒ޭׂͨ͠߹͕70%ΛԼճͬͨΒʯ

    View Slide

  39. ؂ࢹͰඞཁͳ͜ͱ
    ELBͷ UnHealthyHostCount ͕Ͳ͏ͯ͠΋্͕Δ
    ʰʮUnHealthyHostCount > 0ʯ͕7෼ؒܧଓͨ͠Βܯࠂʱ

    View Slide

  40. View Slide

  41. ϩάΛ଎΍͔ʹΠϯελϯε֎ʹసૹ͢Δ
    ΦʔτεέʔϧͷΠϯελϯε͕࡟আ(terminate)͞ΕΔͱ
    ʮEBS΋ফ͑·͢ʯ

    View Slide

  42. ؂ࢹͱϩάʹ͍ͭͯ ·ͱΊ
    γεςϜશମͱ݈ͯ͠શ͔Ͳ͏͔ΛΈΔ
    ݸผͷΠϯεϯλϯεʹͩ͜ΘΓ͗͢ͳ͍ → ϓϩηεΈ͍ͨͳ
    ΋ͷ
    ΠϯελϯεʹӬଓ෺Λ࢒͞ͳ͍ → ϝϞϦΈ͍ͨͳ΋ͷ

    View Slide

  43. Ͳ͏͍͏৚݅Ͱ
    ΦʔτεέʔϧΛൃಈ͢Δ͔

    View Slide

  44. εέʔϦϯάϙϦγʔ
    ΞϓϦέʔγϣϯαʔόͷ৔߹ͷྫ
    • ௚ۙ5෼ؒͷʮฏۉʯCPU࢖༻཰͕50%Λ௒͑ͨΒ
    → ݱࡏͷ୆਺ͷ40%ͷ୆਺Λىಈ
    → 7෼ͳʹ΋͠ͳ͍(ΠϯελϯεىಈΛ଴ͭ)
    • ௚ۙ1࣌ؒͷʮ࠷ߴʯCPU࢖༻཰͕25%Λ੾ͬͨΒ
    → 1୆ఀࢭ
    → 3෼ͳʹ΋͠ͳ͍(Ұؾʹམͪ͗͢ͳ͍Α͏ʹ)

    View Slide

  45. CPU࢖༻཰(ੵΈ্͛), CPU idle(ฏۉ), ϨεϙϯελΠϜ෼෍

    View Slide

  46. ΠϯελϯεΛ૿΍͢ͱ͖ͷίπ
    CPU 40ʙ50%͙Β͍Ͱىಈ͢Δ
    • CPU࢖༻཰্͕͕Δͱॲཧೳྗ͕ઢܗൺྫ͠ͳ͘ͳΔ
    100%Ͱ50%ͷ2ഒॲཧͰ͖ΔΘ͚Ͱ͸ͳ͍
    • Πϯελϯεىಈ·ͰͷϥάΛߟྀ͢Δ
    3ʙ5෼ఔ౓͸଱͑ΒΕΔ͏ͪʹൃಈ͓ͯ͘͠

    View Slide

  47. ΠϯελϯεΛ૿΍͢ͱ͖ͷίπ
    ݱࡏͷ40%ͷ୆਺ɺ࠷௿2୆ىಈ͢Δ
    • 10୆͋Δͱ͖ʹ1୆૿΍ͯ͠΋ॲཧೳྗ͸ʴ10%͚ͩ
    +40%͕2ճൃಈ͢Ε͹໿2ഒʹͳΔ
    • Πϯελϯε͸ৗʹىಈ੒ޭ͢Δͱ͸ݶΒͳ͍
    AWSଆͷ໰୊ɺϓϩϏδϣχϯά΍σϓϩΠࣦഊ

    View Slide

  48. ΠϯελϯεΛݮΒ͢ͱ͖ͷίπ
    ௚ۙ1࣌ؒͷʮ࠷ߴʯCPU࢖༻཰͕25%ҎԼ
    1୆ͣͭམͱͯ͠3෼଴ͭ
    • ʮ୹͍࣌ؒʯͷʮฏۉʯͰམͱ͞ͳ͍
    ૿͑Δͱ༨༟͕Ͱ͖ΔˠམͪΔˠෛՙ্͕Δˠىಈ!
    ༨༟͕े෼͋Δঢ়ଶ͕௕࣌ؒଓ͍ͨΒམͱ͢
    • Ұؾʹམͱ͢ˠෛՙ্͕͕Δˠىಈ!

    View Slide

  49. View Slide

  50. εέʔϦϯάϙϦγʔ·ͱΊ
    ૿΍͢ͱ͖͸େ୾ʹ → ଍Γͳ͚Ε͹ো֐
    ݮΒ͢ͱ͖͸ண࣮ʹ → ݮΒ͗ͨ͢͠Βো֐
    ᮢ஋Λ߈Ί͗͢ͳ͍

    View Slide

  51. ϔϧενΣοΫ

    View Slide

  52. ELBʹ૊ΈࠐΉͨΊʹ͸ϔϧενΣοΫ͕ඞཁ
    1. EC2Πϯελϯε͕ੜ͖͍ͯΔ
    2. (ELB͔Β)TCPͰ઀ଓͰ͖Δ
    3. (ELB͔Β)HTTPͰԠ౴Λ200Ͱฦ͢
    ԿΛ࣋ͬͯਖ਼ৗͱ൑அ͢Δ͔

    View Slide

  53. ELBʹ૊ΈࠐΉͨΊʹ͸ϔϧενΣοΫ͕ඞཁ
    1. ❌ EC2Πϯελϯε͕ੜ͖͍ͯΔ
    → Πϯελϯεىಈ ≠ ΞϓϦέʔγϣϯͷ४උ
    2. ❌ TCPͰ઀ଓͰ͖Δ
    → Webαʔόىಈ ≠ ΞϓϦέʔγϣϯαʔόͷ४උ
    3. ⭕ HTTPͰԠ౴Λ200Ͱฦ͢ GET /ping HTTP/1.0
    ΞϓϦέʔγϣϯαʔόͰDB઀ଓͳͲ΋νΣοΫ͢Δ

    View Slide

  54. ELBʹ૊ΈࠐΉͨΊʹ͸ϔϧενΣοΫ͕ඞཁ
    Πϯελϯε͸ىಈ࣌ͷࣄނ͕ଟ͍ !
    ΞϓϦέʔγϣϯαʔό͕DB΍ͦͷଞʹ઀ଓͰ͖Δ
    →ΞϓϦέʔγϣϯΑΓԼͷ૚ͷ໰୊͸ͳ͍͸ͣ !

    View Slide

  55. ϔϧενΣοΫ·ͱΊ
    ࣮֬ʹΞϓϦέʔγϣϯͷ४උ͕Ͱ͖͔ͯΒ੒ޭͤ͞Δ
    ΞϓϦέʔγϣϯαʔόʹࢮ׆؂ࢹΤϯυϙΠϯτΛ༻ҙ

    View Slide

  56. ຊ౰ʹ͋ͬͨා͍࿩

    View Slide

  57. ຊ౰ʹ͋ͬͨා͍࿩(1)
    ىಈޙʹϔϧενΣοΫ͕௨Βͳ͍
    Πϯελϯε

    View Slide

  58. ϔϧενΣοΫ͕௨Βͳ͍Πϯελϯε͕ൃੜ
    શ୆Ͱ͸ͳ͍
    ىಈޙͷϓϩϏδϣχϯάɺσϓϩΠ͸໰୊ͳ͍
    ͳ͔ͥElasticacheRedisʹܨ͕Βͳ͍
    →ΞϓϦέʔγϣϯϨϕϧͰͷϔϧενΣοΫʹࣦഊ

    View Slide

  59. ϔϧενΣοΫ͕௨Βͳ͍Πϯελϯε͕ൃੜ
    ىಈˠ࡟আˠىಈˠ࡟আ !!!
    ͩΜͩΜࣦഊ֬཰্͕͕ΔΑ͏ͳ…

    View Slide

  60. ϔϧενΣοΫ͕௨Βͳ͍Πϯελϯε͕ൃੜ
    ͡ʔͬͱϩάΛݟΔ
    ʮΉΉɺ͜ͷIPΞυϨε͸ݟ͕֮͑…ʯ

    View Slide

  61. ϔϧενΣοΫ͕௨Βͳ͍Πϯελϯε͕ൃੜ
    աڈʹىಈ͍ͯͨ͠ΠϯελϯεͷIPΞυϨεΛϩά͔Βऩू
    10୆͙Β͍Ұؾʹىಈ͢Δ
    ࣦഊͨ͠΍ͭͷIPΞυϨεΛݟΔ

    View Slide

  62. Ṗ͸͢΂ͯ...

    View Slide

  63. աڈʹ࢖༻͞Εͨ͜ͱ͕
    ͋ΔIPΞυϨε͕࠶ׂΓ
    ౰ͯ͞ΕͨΠϯελϯε
    ͕ElastcacheRedisʹܨ
    ͕Βͳ͍

    View Slide

  64. աڈʹ࢖༻͞Εͨ͜ͱ͕͋ΔIPΞυϨε͕࠶ׂΓ
    ౰ͯ͞ΕͨΠϯελϯε͕ElastcacheRedisʹ
    ܨ͕Βͳ͍
    ElasticacheRedisͷARPςʔϒϧϦϑϨογϡʹ໰୊͕͋ͬͨ
    (2015೥6݄౰࣌)
    αϙʔτͰݸผରԠ
    (2015೥8݄2िҎ߱ʹ৽نىಈ͞Εͨnode͸໰୊ͳ͍ͦ͏Ͱ͢)

    View Slide

  65. ΞϓϦέʔγϣϯϨϕϧͷ
    ϔϧενΣοΫΛ͍ͯͯ͠Α͔ͬͨ

    View Slide

  66. ຊ౰ʹ͋ͬͨා͍࿩(2)
    ϝϯςφϯεฤ

    View Slide

  67. MHAͰDB master੾Γସ͑
    ΞϓϦέʔγϣϯΛ͢΂ͯϝϯςϞʔυʹ
    MHAͰmaster switch

    View Slide

  68. ϝϯς໌͚Α͏ͱͨ͠Β
    ΞϓϦέʔγϣϯαʔό
    ͕མ͍ͪͯͨ

    View Slide

  69. ϔϧενΣοΫͰDB΁ͷ઀ଓΛΈ͍ͯͨ
    DB master͕switch
    DB΁ͷӬଓ઀ଓ͕੾அ͞ΕΔˠ࣍ͷϦΫΤετͰ࠶઀ଓ(༧ఆ)
    App͸Plack͕80ϓϩηε
    ֤ϓϩηε͕੾அͰΤϥʔΛฦ͢
    →ͳ͔ͳ͔ϔϧενΣοΫ͕௨Βͳ͍
    →Πϯελϯεterminate

    View Slide

  70. ܭըϝϯςͰͳͯ͘΋DB͕ࢮ͵͜ͱ͸͋Δ
    ϔϧενΣοΫ͕௨Βͳ͍ͱΠϯελϯε͕མͪΔ
    RDS΍Elasticache͸ͨ·ʹࢮ͵(ͷ͕લఏ)

    View Slide

  71. ʮΠϯελϯεͷࣄނ͸΄΅ىಈ࣌ʯ
    Ұ౓ਖ਼ৗʹىಈͨ͠Πϯελϯε͕ࢮ͵ͷ͸Πϯελϯεো֐
    ͕΄ͱΜͲ
    ઃఆϛε΍ϛυϧ΢ΣΞͷ໰୊Ͱશ໓͢Δͷ͸·͍ͣ
    → Ұ౓ਖ਼ৗʹ૊Έࠐ·ΕͨΒΞϓϦέʔγϣϯϨϕϧͷϔϧε
    νΣοΫ͸͠ͳ͍ʂ

    View Slide

  72. Ұ౓ਖ਼ৗʹ૊Έࠐ·ΕͨΒ…
    at ίϚϯυΛ࢖͏
    ࣗಈىಈͷ࠷ޙʹ
    at 'now + 5 min' -f /usr/local/sbin/confirm-pass-healthcheck
    ىಈޙ5෼ͷ࣌఺ͰELBͰInServiceͩͬͨΒ
    touch /dev/shm/pass_healthcheck
    ͱ͍͏ίϚϯυΛ࣮ߦ

    View Slide

  73. nginxͷϨϕϧͰϔϧενΣοΫΛฦ͢
    location /ping {
    if (-f "/dev/shm/pass_healthcheck") {
    return 200;
    }
    # ϑΝΠϧ͕ͳ͚Ε͹ΞϓϦέʔγϣϯͷϔϧενΣοΫ΁
    proxy_pass http://app;
    }
    tmpfsʹஔ͍ͯ͋ΔͷͰෆҙͷ࠶ىಈ͕͋ͬͯ΋ىಈ࣌ͱಉ͡

    View Slide

  74. ೖΔ΂͖ELBͰInServiceʹͳ͍ͬͯΔ͔Λ൑அ
    #!/bin/bash
    instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id)
    for elb in $(jq -r ".[]" /etc/elb.conf)
    do
    health=$(aws elb describe-instance-health \
    --load-balancer-name "${elb}" --instances "${instance_id}")
    state=$(echo "$health" | jq -r '.InstanceStates[].State')
    if [[ $state != "InService" ]];
    then
    exit 1
    fi
    done
    touch /dev/shm/pass_healthcheck

    View Slide

  75. ຊ౰ʹ͋ͬͨා͍࿩ ·ͱΊ
    ੈؒ͸ݫ͍͠

    View Slide

  76. ·ͱΊ
    Φʔτεέʔϧ͸ίετ࡟ݮʹ༗༻ (ن໛ʹΑΔ)
    ߟ͑ํͷมֵΛഭΒΕΔ (ָ͍͠)
    ͪΌΜͱ੍ޚ͢Δͷ͸େม (େม)

    View Slide