Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KARTEを支えるAutoScaling~SpotInstance編

tik-son
July 06, 2016

 KARTEを支えるAutoScaling~SpotInstance編

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

で話したこと。
spotInstanceを頑張って使っている話とか、Autoscaling周りのお話とか

tik-son

July 06, 2016
Tweet

More Decks by tik-son

Other Decks in Technology

Transcript

  1. KARTEΛࢧ͑ΔAutoScaling ~SpotInstanceฤ
    ʲτϨλ×ϓϨΠυʳTechBlog Deep Dive Meetup #1 ʙΠϯϑϥฤʙ
    2016/07/05
    @tik-son

    View full-size slide

  2. Profile
    ஛ଜঘ඙ a.k.a @tik-son
    ϓϨΠυʹ͸2014/11ʹJoin
    ओʹΠϯϑϥΤϯδχΞ
    ि຤ΪλϦετ

    View full-size slide

  3. ΞΫηεΛҰਓͷਓؒͱͯ͠
    ϦΞϧλΠϜͰՄࢹԽ͠
    ౰ͨΓલͷ઀٬Λ8FC্ͰՄೳʹ

    View full-size slide

  4. http://tech.plaid.co.jp/
    2016/04 start
    ݄ҰϖʔεͰߋ৽༧ఆ

    View full-size slide

  5. ࿩͢͜ͱ
    AutoscalingΛ࢖͏Ϟνϕʔγϣϯ
    AutoscalingͱSpotInstance
    (AutoscalingͰࠔͬͨ͜ͱ/೰Έ)
    ※ࠓ೔࿩͢Autoscaling͸AWSͷAutoscaling

    View full-size slide

  6. AutoscalingΛ࢖͏Ϟνϕʔγϣϯ

    View full-size slide

  7. ̍೔ͷෛՙਪҠ
    ϐʔΫ࣌ͱඇϐʔΫ࣌Ͱ࠷େ10ഒఔ౓ͷࠩ
    1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ
    Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏

    View full-size slide

  8. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ
    Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏
    2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠
    new

    View full-size slide

  9. ෆ݈શͳΠϯελϯεͷ࿩
    • ෆ݈શͳΠϯελϯε(ύϑΥʔϚϯε͕Ͱͳ͘ͳΔɺΞ
    ϓϦέʔγϣϯͷঢ়ଶ͕͓͔͘͠ͳΔ)͕֬཰తʹগͳ͍
    ͕ग़ݱ

    • ෆ݈શͳΠϯελϯε͕࢒Γଓ͚ΔͱKARTEͷύϑΥʔ
    Ϛϯε͕(෦෼తʹ)Լ͕ͬͯ͠·͏͜ͱʹͳΔ

    • ਖ਼ৗԽ͠ͳ͍ͱ͍͚ͳ͍͚ͲɺखಈOperation͸ΊΜͲ͘
    ͍͞

    View full-size slide

  10. ᶃΞϓϦέʔγϣϯͷhealthcheck༻ͷport/pathʹELB
    Ͱhealthcheck
    ᶄAutoscalingͰhealthcheck͕ࣦഊͨ͠ΠϯελϯεΛམͱ͢
    ᶅAutoscalingͰ৽ͨͳΠϯελϯεΛ্ཱͪ͛Δ
    ※WebαʔόͰͳͯ͘΋ɺhealthcheckϙʔτΛ࡞͓͚ͬͯ͹࣮ݱՄೳ



    new
    ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚Δ࢓૊Έ
    terminate

    View full-size slide

  11. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ
    Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏
    2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠
    AutoscalingΛ࢖͏Ϟνϕʔγϣϯ͸2ͭ

    View full-size slide

  12. KARTEར༻ऀ༷ KARTEར༻ऀ༷ͷ
    ͓٬༷
    ղੳ
    ؅ཧը໘ τϥοΫ
    ղੳDB
    τϥοΫDB
    ؅ཧը໘DB
    εςʔτϨεͳαʔό(EC2)શͯ
    Autoscalingར༻Օॴ

    View full-size slide

  13. AutoscalingͱSpotInstance

    View full-size slide

  14. • onedmandͷ1/3 ʙ 1/5 ͙Β͍ͷՁ֨ͰΠϯελϯεΛར༻Ͱ͖Δ
    • ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ →ɹSpotΠϯελϯεΛಈ͔ͤΔ
    • ೖࡳՁ֨ < ࢢ৔Ձ֨ → spotΠϯελϯεར༻Ͱ͖ͳ͍ɻՔಇதͷΠϯελϯεΛམͱ͞
    ΕΔ
    • ىಈઃఆʹೖࡳՁ֨Λઃఆ͢Δ͚ͩͰAutoscalingܦ༝Ͱ؆୯ʹར༻Ͱ͖Δ
    Spot Instance
    ೖࡳՁ֨

    View full-size slide

  15. SpotInstanceͷಋೖʹ͍ͭͯߟ͑ͯΈͨ

    View full-size slide

  16. KARTE͸αʔόϦιʔεΛଟ͘࢖͏αʔϏεͳͷ
    ͰɺSpotInstance࢖͑ͨΒخ͍͠
    Ͱ΋ແ࣊൵ʹམͱ͞ΕΔͱ൵͍͜͠ͱʹɾɾ

    View full-size slide

  17. ೖࡳֹۚΛondemandΑΓ΋͔ͳΓߴ͓͚ͯ͘͠͹མ
    ͱ͞ΕΔ͜ͱ͸ͳ͍͚Ͳɾɾ
    ondemandͷ໿10ഒͷ஋ஈ × 3೔ = ondemandͷ1ϲ݄
    UnControllableͳͷ͸ා͍ɻΠϯελϯε୆਺΋݁ߏ
    ͋Δ

    View full-size slide

  18. spotͷՁ͕֨ondemandͷՁ֨Λ௒͑ͨλΠϛϯάͰ
    ্ख͘ɺಉ͡ΠϯελϯελΠϓ͕ىಈ͢Δondemandͷ
    AutoscalingGroupʹ੾Γସ͑ΕͨΒ޾ͤ
    Autoscaling 㱻 SpotFleet ͷ࿈ܞ΋͠͹Β͘͞Εͳͦ͞͏
    ͩͳɾɾ

    View full-size slide

  19. ࢓૊Έ͕ͳ͍ΜͩͬͨΒɺ
    ෳ਺ͷAutoscalingGroupΛ༻ҙ͠ɺ੾Γସ͑ͯ࢖͏
    ࢓૊ΈΛ࡞ͬͯΈΑ͏
    ͰɺspotInstanceΛੵۃతʹ࢖ͬͯ·͢

    View full-size slide

  20. spot Group
    • ϝΠϯͰ࢖͏
    • SpotInstance͕ىಈ
    • ೖࡳՁ֨͸ondemand
    ͷՁ֨
    • αϒͰ࢖͏
    • ondemandInstance
    ͕ىಈ
    ondemand Group
    ੾Γସ͑
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ
    ༻ҙ͢Δͷ͸3छྨͷAutoscalingGroup

    View full-size slide

  21. ondemand Group
    spot Group
    • αϒͰ࢖͏
    • ondemandInstance
    ͕ىಈ
    ීஈ
    ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ͷ࣌
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ
    • ϝΠϯͰ࢖͏
    • SpotInstance͕ىಈ
    • ೖࡳՁ֨͸ondemand
    ͷՁ֨

    View full-size slide

  22. ondemand Group
    spot Group
    • αϒͰ࢖͏
    • ondemandInstance
    ͕ىಈ
    ࢢ৔Ձ্֨ঢ࣌
    ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ
    ੾Γସ͑
    • ϝΠϯͰ࢖͏
    • SpotInstance͕ىಈ
    • ೖࡳՁ֨͸ondemand
    ͷՁ֨

    View full-size slide

  23. ondemand Group
    spot Group
    • αϒͰ࢖͏
    • ondemandInstance
    ͕ىಈ
    ࢢ৔Ձ֨௿Լ࣌
    ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ
    ੾Γସ͑
    • ϝΠϯͰ࢖͏
    • SpotInstance͕ىಈ
    • ೖࡳՁ֨͸ondemand
    ͷՁ֨

    View full-size slide

  24. ੾Γସ͑ํ๏ʹ͍ͭͯ
    ΋͏গ͠ৄ͘͠ʹ͍ͭͯ࿩͍͖ͯ͠·͢

    View full-size slide

  25. ͦΕͧΕͷGroupʹॴଐ͢ΔEC2Πϯελϯε͕੾Γ
    ସ͑ઌͷASGΛૢ࡞
    ondemand Group
    spot Group
    جຊతͳߟ͑ํ

    View full-size slide

  26. ࢢ৔Ձ্֨ঢ࣌ͷ੾Γସ͑खॱ
    (ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨)
    ondemand Group
    spot Group

    View full-size slide

  27. ondemand Group
    spot Group

    View full-size slide

  28. - ೖࡳՁ֨ < ࢢ৔Ձ֨ →TerminationΛ໿̎෼ؒ଴ͬͯ͘ΕΔ
    - Πϯελϯε಺͔Βࣗ෼ࣗ਎͕Ձ্֨ঢͰམͱ͞ΕΔ͔Ͳ͏͔͕෼͔Δ
    - http://169.254.169.254/latest/meta-data/spot/termination-time
    →ɹ2016-07-05T17:00:00Z or 404 Not Found
    - ݁ߏૣ͘ݕ஌Ͱ͖Δ
    ೖࡳՁ֨

    View full-size slide

  29. ondemand Group
    spot Group
    ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌

    View full-size slide

  30. ondemand Group
    spot Group
    ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌
    ᶄonDemandGroupͷSizeΛ૿΍͢

    View full-size slide

  31. ondemand Group
    spot Group
    ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌
    ᶄonDemandGroupͷSizeΛ૿΍͢
    ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ

    View full-size slide

  32. ೖࡳՁ֨
    • SpotPrice͕ߴಅ͢Δ࣌͸ࢢ৔Ձ͕֨ෆ҆ఆͳ͕࣌ଟ͍

    • ୹࣌ؒͰͷՁ֨ͷ௿ԼͰSpotInstanceͷىಈΛ൑அ͢Δͱɺ͙͢ʹՁ֨ߴಅ͙ͯ͢͠
    ʹམͱ͞Εͯ͠·͏orz

    • SpotPriceߴಅ࣌͸ࢢ৔Ձ͕֨҆ఆ͢Δ·ͰSpotInstance্͕͕Βͳ͍Α͏ʹ͓ͯ͘͠

    View full-size slide

  33. ondemand Group
    spot Group
    ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌
    ᶄonDemandGroupͷSizeΛ૿΍͢
    ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ
    ᶆ spot→ondemandʹ੾ΓସΘΔ

    View full-size slide

  34. ࢢ৔Ձ֨௿Լ࣌
    (ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨)
    ͷ੾Γସ͑
    ondemand Group
    spot Group

    View full-size slide

  35. ondemand Group
    spot Group

    View full-size slide

  36. ondemand Group
    spot Group
    ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck

    View full-size slide

  37. ̍ճࢢ৔Ձ͕֨ߴಅ͢Δͱɺෆ҆ఆͳঢ়ଶ͕࢑͘ଓ͘͜ͱ͕ଟ͍
    ୹࣌ؒͰ൑அͯ͠͠·͏ͱ͙͢ʹΠϯελϯε͕མͱ͞ΕΔϜμʹ
    ೖࡳՁ֨

    View full-size slide

  38. ondemand Group
    spot Group
    ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck
    ᶄSpotGroupͷSizeΛ૿΍͢

    View full-size slide

  39. ondemand Group
    spot Group
    ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck
    ᶄSpotGroupͷSizeΛ૿΍͢
    ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ

    View full-size slide

  40. ondemand Group
    spot Group
    ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck
    ᶄSpotGroupͷSizeΛ૿΍͢
    ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ
    ᶆondemand→spotʹ੾ΓସΘΔ

    View full-size slide

  41. ݁ہstaticGroupͬͯʁ
    static Group

    View full-size slide

  42. ͻͱ͜ͱͰݴ͏ͱอݥ༻
    spot→ondemandͷ੾Γସ͑ͷࡍʹɺΠϯελϯεͷىಈ࣌ؒʹΑͬ
    ͯ͸ɺActiveͳΠϯελϯε͕গͳ͘ͳΓఏڙαʔϏεͷSLAʹ͙ͦ
    Θͳ͘ͳͬͯ͠·͏͜ͱ΋ߟ͑ΒΕΔͷͰ
    ඞͣҰఆ୆਺͸ಈ͖ଓ͚ΔStaticGroupΛ༻ҙͯ͠όϥϯεΛऔͬͯ
    ͍Δ
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ

    View full-size slide

  43. on demand Group
    spot Group
    • αϒͰ࢖͏
    • ondemandInstance
    ͕ىಈ
    static Group
    • ৗʹҰఆ୆਺͕ىಈ
    • ondemandInstance͕
    ىಈ
    3छྨͷάϧʔϓΛ࢖͍෼͚ͯ·͢
    ੾Γସ͑
    • ϝΠϯͰ࢖͏
    • SpotInstance͕ىಈ
    • ೖࡳՁ֨͸ondemand
    ͷՁ֨

    View full-size slide

  44. AutoscalingͰࠔͬͨ͜ͱ/೰Έ

    View full-size slide

  45. ᶃ:RateExceeded
    GUI͔ΒAutoscalingΛશ͘ૢ࡞Ͱ͖ͳ͘ͳͬͯ͠·ͬ
    ͨɻΠϯελϯεΛ૿΍͍ͨ͠ͷʹ૿΍ͤͳ͍ɾɾɾ
    why!? αϙʔτʹฉ͍ͯΈͨɻ

    View full-size slide

  46. A. APIΛୟ͖͗͢
    • spot㱻ondemandͷ੾Γସ͑ͷͨΊͷ؂ࢹ
    • deployલޙͷॲཧ
    • instanceىಈ࣌
    • spotΠϯελϯεͷࢢ৔Ձ֨ͷDatadog΁ͷૹ෇࣌
    • terraform࣮ߦ࣌
    • DatadogͷAutoscaling Integration ɾɾɾɾetc

    View full-size slide

  47. API Request੍͕ݶ͞ΕΔࡍͷɺ۩ମతͳ্ݶճ਺͸ڭ͑ͯ΋Β͑ͳ
    ͔ͬͨͷͰ
    • ෆཁͳIntegrationɺॲཧΛݟ௚͠
    • ϦτϥΠ͸ΤΫεϙωϯγϟϧόοΫΦϑ
    ͪͳΈʹCloudWatchʹCustomMetricΛૹΔ࣌΋ಉ͡

    View full-size slide

  48. AutoscalingͷτϦΨʔʹҾ͔͔ͬΔˠEC2Λىಈˠ࠷৽ͷCodeΛpullͯ͠
    setup(CodeDeploy)→ΞϓϦىಈ
    ىಈ·ͰͲ͏ͯ͠΋਺෼͔͔ͬͯ͠·͏
    - EC2ͷىಈ͕࣌ؒͦΜͳʹૣ͘ͳ͍(SpotInstanceͷىಈ࣌ؒ͸࠷ۙૣ͘ͳ͚ͬͨͲ)
    - εφοϓγϣοτ͔Β෮ݩ͞ΕͨEBSϘϦϡʔϜ΁ͷॳճΞΫηε࣌͸ύϑΥʔϚϯ
    ε͕Լ͕Δ
    ݱࡏ͸ɺ͋Δఔ౓༨༟Λ΋ͨͤͨcapacityઃܭ
    ᶄಥൃతͳෛՙͱ͸૬ੑѱ͍

    View full-size slide

  49. ಥൃతͳෛՙʹ΋໰୊ͳ͘εέʔϧ͢Δͤ͞Δʹ͸ʁ(໛ࡧத)
    ᶃΠϯελϯεϨϕϧͰͷAutoscalingΛ࢖Θͳ͍
    • ཪଆͰΑ͠ͳʹ΍ͬͯ͘ΕΔαʔόϨεͳαʔϏεͷGAEͱ͔Lambdaͱ͔ͦͷลΓ
    Λ্ख͔ͭ͘͏ʁ
    • αʔόϦιʔεΛ͔ͬ͠Γ࢖͏ॲཧ͸૬ੑ߹Θͳͦ͞͏
    • ϑϧϚωʔδυαʔϏεͷϦεΫ
    • vs ୆਺ʹ༨༟Λ࣋ͨͤͨSpotInstanceͰͷӡ༻?
    ᶄΠϯελϯεͷىಈ࣌ؒͱcodeͷ൓ө࣌ؒΛ୹͘͢Δ
    - GCE?
    - ίϯςφ?

    View full-size slide

  50. ·ͱΊ
    • AutoscalingͷԸܙ͸ड͚ͯΔ
    • ؤுͬͯspotInstanceΛ࢖͍ͬͯΔ
    • AutoscalingAPIͷ͝ར༻͸ܭըతʹ
    • ·ͩ·ͩAutoscalingपΓͰվળͷ༨஍͋Γɻߋ
    ͳΔਐԽΛ໨ࢦͯ͠·͢

    View full-size slide

  51. Termination Reason:Server.InternalError
    ͋Δ೔Πϯελϯε͕ىಈ͞Εͳ͘ͳͬͨ
    ͜ͷmessage͕ίϯιʔϧͷΞΫςΟϏςΟཤྺʹ
    Launching a new EC2 instance: i-xxxxxxx.
    Status Reason: Instance became unhealthy
    while waiting for instance to be in InService
    state. Termination Reason: Server.InternalError
    why!? αϙʔτʹฉ͍ͯΈͨɻ

    View full-size slide

  52. A. EBSͷ߹ܭ࢖༻ྔ͕ΞΧ΢ϯτͷ্ݶ௒͑

    View full-size slide

  53. • AutoscalingͰཱͯΔΠϯελϯε͕࣋ͭϦιʔ
    εͷ্ݶ஋ʹୡͨ࣌͠΋ɺىಈʹࣦഊ͢Δ
    • ࣦͨͩ͠ഊཧ༝͸Console͔Β෼͔Γʹ͍͘
    (Termination Reason: Server.InternalError)

    View full-size slide