Save 37% off PRO during our Black Friday Sale! »

KARTEを支えるAutoScaling~SpotInstance編

Fc099d9851ad91908e0c16a70aa2ddc1?s=47 tik-son
July 06, 2016

 KARTEを支えるAutoScaling~SpotInstance編

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

で話したこと。
spotInstanceを頑張って使っている話とか、Autoscaling周りのお話とか

Fc099d9851ad91908e0c16a70aa2ddc1?s=128

tik-son

July 06, 2016
Tweet

Transcript

  1. KARTEΛࢧ͑ΔAutoScaling ~SpotInstanceฤ ʲτϨλ×ϓϨΠυʳTechBlog Deep Dive Meetup #1 ʙΠϯϑϥฤʙ 2016/07/05 @tik-son

  2. Profile ஛ଜঘ඙ a.k.a @tik-son ϓϨΠυʹ͸2014/11ʹJoin ओʹΠϯϑϥΤϯδχΞ ि຤ΪλϦετ

  3. ΞΫηεΛҰਓͷਓؒͱͯ͠ ϦΞϧλΠϜͰՄࢹԽ͠ ౰ͨΓલͷ઀٬Λ8FC্ͰՄೳʹ

  4. http://tech.plaid.co.jp/ 2016/04 start ݄ҰϖʔεͰߋ৽༧ఆ

  5. ࿩͢͜ͱ AutoscalingΛ࢖͏Ϟνϕʔγϣϯ AutoscalingͱSpotInstance (AutoscalingͰࠔͬͨ͜ͱ/೰Έ) ※ࠓ೔࿩͢Autoscaling͸AWSͷAutoscaling

  6. AutoscalingΛ࢖͏Ϟνϕʔγϣϯ

  7. ̍೔ͷෛՙਪҠ ϐʔΫ࣌ͱඇϐʔΫ࣌Ͱ࠷େ10ഒఔ౓ͷࠩ 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏

  8. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏ 2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠ new

  9. ෆ݈શͳΠϯελϯεͷ࿩ • ෆ݈શͳΠϯελϯε(ύϑΥʔϚϯε͕Ͱͳ͘ͳΔɺΞ ϓϦέʔγϣϯͷঢ়ଶ͕͓͔͘͠ͳΔ)͕֬཰తʹগͳ͍ ͕ग़ݱ • ෆ݈શͳΠϯελϯε͕࢒Γଓ͚ΔͱKARTEͷύϑΥʔ Ϛϯε͕(෦෼తʹ)Լ͕ͬͯ͠·͏͜ͱʹͳΔ • ਖ਼ৗԽ͠ͳ͍ͱ͍͚ͳ͍͚ͲɺखಈOperation͸ΊΜͲ͘

    ͍͞
  10. ᶃΞϓϦέʔγϣϯͷhealthcheck༻ͷport/pathʹELB Ͱhealthcheck ᶄAutoscalingͰhealthcheck͕ࣦഊͨ͠ΠϯελϯεΛམͱ͢ ᶅAutoscalingͰ৽ͨͳΠϯελϯεΛ্ཱͪ͛Δ ※WebαʔόͰͳͯ͘΋ɺhealthcheckϙʔτΛ࡞͓͚ͬͯ͹࣮ݱՄೳ ᶃ ᶄ ᶅ new ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚Δ࢓૊Έ

    terminate
  11. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏ 2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠ AutoscalingΛ࢖͏Ϟνϕʔγϣϯ͸2ͭ

  12. KARTEར༻ऀ༷ KARTEར༻ऀ༷ͷ ͓٬༷ ղੳ ؅ཧը໘ τϥοΫ ղੳDB τϥοΫDB ؅ཧը໘DB εςʔτϨεͳαʔό(EC2)શͯ

    Autoscalingར༻Օॴ
  13. AutoscalingͱSpotInstance

  14. • onedmandͷ1/3 ʙ 1/5 ͙Β͍ͷՁ֨ͰΠϯελϯεΛར༻Ͱ͖Δ • ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ →ɹSpotΠϯελϯεΛಈ͔ͤΔ

    • ೖࡳՁ֨ < ࢢ৔Ձ֨ → spotΠϯελϯεར༻Ͱ͖ͳ͍ɻՔಇதͷΠϯελϯεΛམͱ͞ ΕΔ • ىಈઃఆʹೖࡳՁ֨Λઃఆ͢Δ͚ͩͰAutoscalingܦ༝Ͱ؆୯ʹར༻Ͱ͖Δ Spot Instance ೖࡳՁ֨
  15. SpotInstanceͷಋೖʹ͍ͭͯߟ͑ͯΈͨ

  16. KARTE͸αʔόϦιʔεΛଟ͘࢖͏αʔϏεͳͷ ͰɺSpotInstance࢖͑ͨΒخ͍͠ Ͱ΋ແ࣊൵ʹམͱ͞ΕΔͱ൵͍͜͠ͱʹɾɾ

  17. ೖࡳֹۚΛondemandΑΓ΋͔ͳΓߴ͓͚ͯ͘͠͹མ ͱ͞ΕΔ͜ͱ͸ͳ͍͚Ͳɾɾ ondemandͷ໿10ഒͷ஋ஈ × 3೔ = ondemandͷ1ϲ݄ UnControllableͳͷ͸ා͍ɻΠϯελϯε୆਺΋݁ߏ ͋Δ

  18. spotͷՁ͕֨ondemandͷՁ֨Λ௒͑ͨλΠϛϯάͰ ্ख͘ɺಉ͡ΠϯελϯελΠϓ͕ىಈ͢Δondemandͷ AutoscalingGroupʹ੾Γସ͑ΕͨΒ޾ͤ Autoscaling 㱻 SpotFleet ͷ࿈ܞ΋͠͹Β͘͞Εͳͦ͞͏ ͩͳɾɾ

  19. ࢓૊Έ͕ͳ͍ΜͩͬͨΒɺ ෳ਺ͷAutoscalingGroupΛ༻ҙ͠ɺ੾Γସ͑ͯ࢖͏ ࢓૊ΈΛ࡞ͬͯΈΑ͏ ͰɺspotInstanceΛੵۃతʹ࢖ͬͯ·͢

  20. spot Group • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨ •

    αϒͰ࢖͏ • ondemandInstance ͕ىಈ ondemand Group ੾Γସ͑ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ༻ҙ͢Δͷ͸3छྨͷAutoscalingGroup
  21. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ීஈ

    ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ͷ࣌ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
  22. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ࢢ৔Ձ্֨ঢ࣌

    ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
  23. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ࢢ৔Ձ֨௿Լ࣌

    ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
  24. ੾Γସ͑ํ๏ʹ͍ͭͯ ΋͏গ͠ৄ͘͠ʹ͍ͭͯ࿩͍͖ͯ͠·͢

  25. ͦΕͧΕͷGroupʹॴଐ͢ΔEC2Πϯελϯε͕੾Γ ସ͑ઌͷASGΛૢ࡞ ondemand Group spot Group جຊతͳߟ͑ํ

  26. ࢢ৔Ձ্֨ঢ࣌ͷ੾Γସ͑खॱ (ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨) ondemand Group

    spot Group
  27. ondemand Group spot Group

  28. - ೖࡳՁ֨ < ࢢ৔Ձ֨ →TerminationΛ໿̎෼ؒ଴ͬͯ͘ΕΔ - Πϯελϯε಺͔Βࣗ෼ࣗ਎͕Ձ্֨ঢͰམͱ͞ΕΔ͔Ͳ͏͔͕෼͔Δ - http://169.254.169.254/latest/meta-data/spot/termination-time →ɹ2016-07-05T17:00:00Z

    or 404 Not Found - ݁ߏૣ͘ݕ஌Ͱ͖Δ ೖࡳՁ֨
  29. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶃ

  30. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶄ

  31. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ ᶅ

  32. ೖࡳՁ֨ • SpotPrice͕ߴಅ͢Δ࣌͸ࢢ৔Ձ͕֨ෆ҆ఆͳ͕࣌ଟ͍ • ୹࣌ؒͰͷՁ֨ͷ௿ԼͰSpotInstanceͷىಈΛ൑அ͢Δͱɺ͙͢ʹՁ֨ߴಅ͙ͯ͢͠ ʹམͱ͞Εͯ͠·͏orz • SpotPriceߴಅ࣌͸ࢢ৔Ձ͕֨҆ఆ͢Δ·ͰSpotInstance্͕͕Βͳ͍Α͏ʹ͓ͯ͘͠

  33. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ ᶆ

    spot→ondemandʹ੾ΓସΘΔ
  34. ࢢ৔Ձ֨௿Լ࣌ (ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨) ͷ੾Γସ͑ ondemand

    Group spot Group
  35. ondemand Group spot Group

  36. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶃ

  37. ̍ճࢢ৔Ձ͕֨ߴಅ͢Δͱɺෆ҆ఆͳঢ়ଶ͕࢑͘ଓ͘͜ͱ͕ଟ͍ ୹࣌ؒͰ൑அͯ͠͠·͏ͱ͙͢ʹΠϯελϯε͕མͱ͞ΕΔϜμʹ ೖࡳՁ֨

  38. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶄ

  39. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ ᶅ

  40. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ ᶆondemand→spotʹ੾ΓସΘΔ

  41. ݁ہstaticGroupͬͯʁ static Group

  42. ͻͱ͜ͱͰݴ͏ͱอݥ༻ spot→ondemandͷ੾Γସ͑ͷࡍʹɺΠϯελϯεͷىಈ࣌ؒʹΑͬ ͯ͸ɺActiveͳΠϯελϯε͕গͳ͘ͳΓఏڙαʔϏεͷSLAʹ͙ͦ Θͳ͘ͳͬͯ͠·͏͜ͱ΋ߟ͑ΒΕΔͷͰ ඞͣҰఆ୆਺͸ಈ͖ଓ͚ΔStaticGroupΛ༻ҙͯ͠όϥϯεΛऔͬͯ ͍Δ static Group • ৗʹҰఆ୆਺͕ىಈ

    • ondemandInstance͕ ىಈ
  43. on demand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ

    static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ 3छྨͷάϧʔϓΛ࢖͍෼͚ͯ·͢ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
  44. AutoscalingͰࠔͬͨ͜ͱ/೰Έ

  45. ᶃ:RateExceeded GUI͔ΒAutoscalingΛશ͘ૢ࡞Ͱ͖ͳ͘ͳͬͯ͠·ͬ ͨɻΠϯελϯεΛ૿΍͍ͨ͠ͷʹ૿΍ͤͳ͍ɾɾɾ why!? αϙʔτʹฉ͍ͯΈͨɻ

  46. A. APIΛୟ͖͗͢ • spot㱻ondemandͷ੾Γସ͑ͷͨΊͷ؂ࢹ • deployલޙͷॲཧ • instanceىಈ࣌ • spotΠϯελϯεͷࢢ৔Ձ֨ͷDatadog΁ͷૹ෇࣌

    • terraform࣮ߦ࣌ • DatadogͷAutoscaling Integration ɾɾɾɾetc
  47. API Request੍͕ݶ͞ΕΔࡍͷɺ۩ମతͳ্ݶճ਺͸ڭ͑ͯ΋Β͑ͳ ͔ͬͨͷͰ • ෆཁͳIntegrationɺॲཧΛݟ௚͠ • ϦτϥΠ͸ΤΫεϙωϯγϟϧόοΫΦϑ ͪͳΈʹCloudWatchʹCustomMetricΛૹΔ࣌΋ಉ͡

  48. AutoscalingͷτϦΨʔʹҾ͔͔ͬΔˠEC2Λىಈˠ࠷৽ͷCodeΛpullͯ͠ setup(CodeDeploy)→ΞϓϦىಈ ىಈ·ͰͲ͏ͯ͠΋਺෼͔͔ͬͯ͠·͏ - EC2ͷىಈ͕࣌ؒͦΜͳʹૣ͘ͳ͍(SpotInstanceͷىಈ࣌ؒ͸࠷ۙૣ͘ͳ͚ͬͨͲ) - εφοϓγϣοτ͔Β෮ݩ͞ΕͨEBSϘϦϡʔϜ΁ͷॳճΞΫηε࣌͸ύϑΥʔϚϯ ε͕Լ͕Δ ݱࡏ͸ɺ͋Δఔ౓༨༟Λ΋ͨͤͨcapacityઃܭ ᶄಥൃతͳෛՙͱ͸૬ੑѱ͍

  49. ಥൃతͳෛՙʹ΋໰୊ͳ͘εέʔϧ͢Δͤ͞Δʹ͸ʁ(໛ࡧத) ᶃΠϯελϯεϨϕϧͰͷAutoscalingΛ࢖Θͳ͍ • ཪଆͰΑ͠ͳʹ΍ͬͯ͘ΕΔαʔόϨεͳαʔϏεͷGAEͱ͔Lambdaͱ͔ͦͷลΓ Λ্ख͔ͭ͘͏ʁ • αʔόϦιʔεΛ͔ͬ͠Γ࢖͏ॲཧ͸૬ੑ߹Θͳͦ͞͏ • ϑϧϚωʔδυαʔϏεͷϦεΫ •

    vs ୆਺ʹ༨༟Λ࣋ͨͤͨSpotInstanceͰͷӡ༻? ᶄΠϯελϯεͷىಈ࣌ؒͱcodeͷ൓ө࣌ؒΛ୹͘͢Δ - GCE? - ίϯςφ?
  50. ·ͱΊ • AutoscalingͷԸܙ͸ड͚ͯΔ • ؤுͬͯspotInstanceΛ࢖͍ͬͯΔ • AutoscalingAPIͷ͝ར༻͸ܭըతʹ • ·ͩ·ͩAutoscalingपΓͰվળͷ༨஍͋Γɻߋ ͳΔਐԽΛ໨ࢦͯ͠·͢

  51. ͓·͚

  52. Termination Reason:Server.InternalError ͋Δ೔Πϯελϯε͕ىಈ͞Εͳ͘ͳͬͨ ͜ͷmessage͕ίϯιʔϧͷΞΫςΟϏςΟཤྺʹ Launching a new EC2 instance: i-xxxxxxx.

    Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Server.InternalError why!? αϙʔτʹฉ͍ͯΈͨɻ
  53. A. EBSͷ߹ܭ࢖༻ྔ͕ΞΧ΢ϯτͷ্ݶ௒͑

  54. • AutoscalingͰཱͯΔΠϯελϯε͕࣋ͭϦιʔ εͷ্ݶ஋ʹୡͨ࣌͠΋ɺىಈʹࣦഊ͢Δ • ࣦͨͩ͠ഊཧ༝͸Console͔Β෼͔Γʹ͍͘ (Termination Reason: Server.InternalError)