KARTEを支えるAutoScaling~SpotInstance編

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

で話したこと。
spotInstanceを頑張って使っている話とか、Autoscaling周りのお話とか

Transcript

1. KARTEΛࢧ͑ΔAutoScaling ~SpotInstanceฤ ʲτϨλ×ϓϨΠυʳTechBlog Deep Dive Meetup #1 ʙΠϯϑϥฤʙ 2016/07/05 @tik-son
2. Profile ஛ଜঘ඙ a.k.a @tik-son ϓϨΠυʹ͸2014/11ʹJoin ओʹΠϯϑϥΤϯδχΞ ि຤ΪλϦετ
3. ΞΫηεΛҰਓͷਓؒͱͯ͠
   ϦΞϧλΠϜͰՄࢹԽ͠
   ౰ͨΓલͷ઀٬Λ8FC্ͰՄೳʹ
4. http://tech.plaid.co.jp/ 2016/04 start ݄ҰϖʔεͰߋ৽༧ఆ
5. ࿩͢͜ͱ AutoscalingΛ࢖͏Ϟνϕʔγϣϯ AutoscalingͱSpotInstance (AutoscalingͰࠔͬͨ͜ͱ/೰Έ) ※ࠓ೔࿩͢Autoscaling͸AWSͷAutoscaling
6. AutoscalingΛ࢖͏Ϟνϕʔγϣϯ
7. ̍೔ͷෛՙਪҠ ϐʔΫ࣌ͱඇϐʔΫ࣌Ͱ࠷େ10ഒఔ౓ͷࠩ 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏
8. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏ 2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠ new
9. ෆ݈શͳΠϯελϯεͷ࿩ • ෆ݈શͳΠϯελϯε(ύϑΥʔϚϯε͕Ͱͳ͘ͳΔɺΞ ϓϦέʔγϣϯͷঢ়ଶ͕͓͔͘͠ͳΔ)͕֬཰తʹগͳ͍ ͕ग़ݱ • ෆ݈શͳΠϯελϯε͕࢒Γଓ͚ΔͱKARTEͷύϑΥʔ Ϛϯε͕(෦෼తʹ)Լ͕ͬͯ͠·͏͜ͱʹͳΔ • ਖ਼ৗԽ͠ͳ͍ͱ͍͚ͳ͍͚ͲɺखಈOperation͸ΊΜͲ͘ ͍͞
10. ᶃΞϓϦέʔγϣϯͷhealthcheck༻ͷport/pathʹELB Ͱhealthcheck ᶄAutoscalingͰhealthcheck͕ࣦഊͨ͠ΠϯελϯεΛམͱ͢ ᶅAutoscalingͰ৽ͨͳΠϯελϯεΛ্ཱͪ͛Δ ※WebαʔόͰͳͯ͘΋ɺhealthcheckϙʔτΛ࡞͓͚ͬͯ͹࣮ݱՄೳ ᶃ ᶄ ᶅ new ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚Δ࢓૊Έ terminate
11. 1. ࣌ؒଳʹΑͬͯେ͖͘ෛՙ͕มΘΔͨΊɺϦιʔεΛෛՙͷϐʔ Ϋʹ͋Θͤͯ༻ҙ͓ͯ͘͠ͱແବʹίετ͕͔͔ͬͯ͠·͏ 2. ݈શͳΠϯελϯεͷΈ͕ੜ͖ଓ͚ΔΑ͏ʹ͍ͨ͠ AutoscalingΛ࢖͏Ϟνϕʔγϣϯ͸2ͭ
12. KARTEར༻ऀ༷ KARTEར༻ऀ༷ͷ ͓٬༷ ղੳ ؅ཧը໘ τϥοΫ ղੳDB τϥοΫDB ؅ཧը໘DB εςʔτϨεͳαʔό(EC2)શͯ Autoscalingར༻Օॴ
13. AutoscalingͱSpotInstance
14. • onedmandͷ1/3 ʙ 1/5 ͙Β͍ͷՁ֨ͰΠϯελϯεΛར༻Ͱ͖Δ • ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ →ɹSpotΠϯελϯεΛಈ͔ͤΔ • ೖࡳՁ֨ < ࢢ৔Ձ֨ → spotΠϯελϯεར༻Ͱ͖ͳ͍ɻՔಇதͷΠϯελϯεΛམͱ͞ ΕΔ • ىಈઃఆʹೖࡳՁ֨Λઃఆ͢Δ͚ͩͰAutoscalingܦ༝Ͱ؆୯ʹར༻Ͱ͖Δ Spot Instance ೖࡳՁ֨
15. SpotInstanceͷಋೖʹ͍ͭͯߟ͑ͯΈͨ
16. KARTE͸αʔόϦιʔεΛଟ͘࢖͏αʔϏεͳͷ ͰɺSpotInstance࢖͑ͨΒخ͍͠ Ͱ΋ແ࣊൵ʹམͱ͞ΕΔͱ൵͍͜͠ͱʹɾɾ
17. ೖࡳֹۚΛondemandΑΓ΋͔ͳΓߴ͓͚ͯ͘͠͹མ ͱ͞ΕΔ͜ͱ͸ͳ͍͚Ͳɾɾ ondemandͷ໿10ഒͷ஋ஈ × 3೔ = ondemandͷ1ϲ݄ UnControllableͳͷ͸ා͍ɻΠϯελϯε୆਺΋݁ߏ ͋Δ
18. spotͷՁ͕֨ondemandͷՁ֨Λ௒͑ͨλΠϛϯάͰ ্ख͘ɺಉ͡ΠϯελϯελΠϓ͕ىಈ͢Δondemandͷ AutoscalingGroupʹ੾Γସ͑ΕͨΒ޾ͤ Autoscaling 㱻 SpotFleet ͷ࿈ܞ΋͠͹Β͘͞Εͳͦ͞͏ ͩͳɾɾ
19. ࢓૊Έ͕ͳ͍ΜͩͬͨΒɺ ෳ਺ͷAutoscalingGroupΛ༻ҙ͠ɺ੾Γସ͑ͯ࢖͏ ࢓૊ΈΛ࡞ͬͯΈΑ͏ ͰɺspotInstanceΛੵۃతʹ࢖ͬͯ·͢
20. spot Group • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨ • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ondemand Group ੾Γସ͑ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ༻ҙ͢Δͷ͸3छྨͷAutoscalingGroup
21. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ීஈ ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ͷ࣌ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
22. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ࢢ৔Ձ্֨ঢ࣌ ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
23. ondemand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ ࢢ৔Ձ֨௿Լ࣌ ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
24. ੾Γସ͑ํ๏ʹ͍ͭͯ ΋͏গ͠ৄ͘͠ʹ͍ͭͯ࿩͍͖ͯ͠·͢
25. ͦΕͧΕͷGroupʹॴଐ͢ΔEC2Πϯελϯε͕੾Γ ସ͑ઌͷASGΛૢ࡞ ondemand Group spot Group جຊతͳߟ͑ํ
26. ࢢ৔Ձ্֨ঢ࣌ͷ੾Γସ͑खॱ (ೖࡳՁ֨ ≧ ࢢ৔Ձ֨ → ೖࡳՁ֨ < ࢢ৔Ձ֨) ondemand Group spot Group
27. ondemand Group spot Group
28. - ೖࡳՁ֨ < ࢢ৔Ձ֨ →TerminationΛ໿̎෼ؒ଴ͬͯ͘ΕΔ - Πϯελϯε಺͔Βࣗ෼ࣗ਎͕Ձ্֨ঢͰམͱ͞ΕΔ͔Ͳ͏͔͕෼͔Δ - http://169.254.169.254/latest/meta-data/spot/termination-time →ɹ2016-07-05T17:00:00Z or 404 Not Found - ݁ߏૣ͘ݕ஌Ͱ͖Δ ೖࡳՁ֨
29. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶃ
30. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶄ
31. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ ᶅ
32. ೖࡳՁ֨ • SpotPrice͕ߴಅ͢Δ࣌͸ࢢ৔Ձ͕֨ෆ҆ఆͳ͕࣌ଟ͍ • ୹࣌ؒͰͷՁ֨ͷ௿ԼͰSpotInstanceͷىಈΛ൑அ͢Δͱɺ͙͢ʹՁ֨ߴಅ͙ͯ͢͠ ʹམͱ͞Εͯ͠·͏orz • SpotPriceߴಅ࣌͸ࢢ৔Ձ͕֨҆ఆ͢Δ·ͰSpotInstance্͕͕Βͳ͍Α͏ʹ͓ͯ͘͠
33. ondemand Group spot Group ᶃhttp://169.254.169.254/latest/meta-data/spot/termination-time Λ࢖ͬͯݕ஌ ᶄonDemandGroupͷSizeΛ૿΍͢ ᶅ spotInstance্ཱ͕͕ͪΒͳ͍Α͏ʹ͢Δ ᶆ spot→ondemandʹ੾ΓସΘΔ
34. ࢢ৔Ձ֨௿Լ࣌ (ೖࡳՁ֨ < ࢢ৔Ձ֨ → ೖࡳՁ֨ ≧ ࢢ৔Ձ֨) ͷ੾Γସ͑ ondemand Group spot Group
35. ondemand Group spot Group
36. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶃ
37. ̍ճࢢ৔Ձ͕֨ߴಅ͢Δͱɺෆ҆ఆͳঢ়ଶ͕࢑͘ଓ͘͜ͱ͕ଟ͍ ୹࣌ؒͰ൑அͯ͠͠·͏ͱ͙͢ʹΠϯελϯε͕མͱ͞ΕΔϜμʹ ೖࡳՁ֨
38. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶄ
39. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ ᶅ
40. ondemand Group spot Group ᶃೖࡳՁ͕֨ࢢ৔Ձ֨ΛҰఆظ্ؒճΔ͔Λఆظతʹcheck ᶄSpotGroupͷSizeΛ૿΍͢ ᶅSpotGroup͕ਖ਼ৗʹىಈͨ͠ͷΛ֬ೝͯ͠ɺondemandGroupͷsizeΛมߋ͢Δ ᶆondemand→spotʹ੾ΓସΘΔ
41. ݁ہstaticGroupͬͯʁ static Group
42. ͻͱ͜ͱͰݴ͏ͱอݥ༻ spot→ondemandͷ੾Γସ͑ͷࡍʹɺΠϯελϯεͷىಈ࣌ؒʹΑͬ ͯ͸ɺActiveͳΠϯελϯε͕গͳ͘ͳΓఏڙαʔϏεͷSLAʹ͙ͦ Θͳ͘ͳͬͯ͠·͏͜ͱ΋ߟ͑ΒΕΔͷͰ ඞͣҰఆ୆਺͸ಈ͖ଓ͚ΔStaticGroupΛ༻ҙͯ͠όϥϯεΛऔͬͯ ͍Δ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ
43. on demand Group spot Group • αϒͰ࢖͏ • ondemandInstance ͕ىಈ static Group • ৗʹҰఆ୆਺͕ىಈ • ondemandInstance͕ ىಈ 3छྨͷάϧʔϓΛ࢖͍෼͚ͯ·͢ ੾Γସ͑ • ϝΠϯͰ࢖͏ • SpotInstance͕ىಈ • ೖࡳՁ֨͸ondemand ͷՁ֨
44. AutoscalingͰࠔͬͨ͜ͱ/೰Έ
45. ᶃ:RateExceeded GUI͔ΒAutoscalingΛશ͘ૢ࡞Ͱ͖ͳ͘ͳͬͯ͠·ͬ ͨɻΠϯελϯεΛ૿΍͍ͨ͠ͷʹ૿΍ͤͳ͍ɾɾɾ why!? αϙʔτʹฉ͍ͯΈͨɻ
46. A. APIΛୟ͖͗͢ • spot㱻ondemandͷ੾Γସ͑ͷͨΊͷ؂ࢹ • deployલޙͷॲཧ • instanceىಈ࣌ • spotΠϯελϯεͷࢢ৔Ձ֨ͷDatadog΁ͷૹ෇࣌ • terraform࣮ߦ࣌ • DatadogͷAutoscaling Integration ɾɾɾɾetc
47. API Request੍͕ݶ͞ΕΔࡍͷɺ۩ମతͳ্ݶճ਺͸ڭ͑ͯ΋Β͑ͳ ͔ͬͨͷͰ • ෆཁͳIntegrationɺॲཧΛݟ௚͠ • ϦτϥΠ͸ΤΫεϙωϯγϟϧόοΫΦϑ ͪͳΈʹCloudWatchʹCustomMetricΛૹΔ࣌΋ಉ͡
48. AutoscalingͷτϦΨʔʹҾ͔͔ͬΔˠEC2Λىಈˠ࠷৽ͷCodeΛpullͯ͠ setup(CodeDeploy)→ΞϓϦىಈ ىಈ·ͰͲ͏ͯ͠΋਺෼͔͔ͬͯ͠·͏ - EC2ͷىಈ͕࣌ؒͦΜͳʹૣ͘ͳ͍(SpotInstanceͷىಈ࣌ؒ͸࠷ۙૣ͘ͳ͚ͬͨͲ) - εφοϓγϣοτ͔Β෮ݩ͞ΕͨEBSϘϦϡʔϜ΁ͷॳճΞΫηε࣌͸ύϑΥʔϚϯ ε͕Լ͕Δ ݱࡏ͸ɺ͋Δఔ౓༨༟Λ΋ͨͤͨcapacityઃܭ ᶄಥൃతͳෛՙͱ͸૬ੑѱ͍
49. ಥൃతͳෛՙʹ΋໰୊ͳ͘εέʔϧ͢Δͤ͞Δʹ͸ʁ(໛ࡧத) ᶃΠϯελϯεϨϕϧͰͷAutoscalingΛ࢖Θͳ͍ • ཪଆͰΑ͠ͳʹ΍ͬͯ͘ΕΔαʔόϨεͳαʔϏεͷGAEͱ͔Lambdaͱ͔ͦͷลΓ Λ্ख͔ͭ͘͏ʁ • αʔόϦιʔεΛ͔ͬ͠Γ࢖͏ॲཧ͸૬ੑ߹Θͳͦ͞͏ • ϑϧϚωʔδυαʔϏεͷϦεΫ • vs ୆਺ʹ༨༟Λ࣋ͨͤͨSpotInstanceͰͷӡ༻? ᶄΠϯελϯεͷىಈ࣌ؒͱcodeͷ൓ө࣌ؒΛ୹͘͢Δ - GCE? - ίϯςφ?
50. ·ͱΊ • AutoscalingͷԸܙ͸ड͚ͯΔ • ؤுͬͯspotInstanceΛ࢖͍ͬͯΔ • AutoscalingAPIͷ͝ར༻͸ܭըతʹ • ·ͩ·ͩAutoscalingपΓͰվળͷ༨஍͋Γɻߋ ͳΔਐԽΛ໨ࢦͯ͠·͢
51. ͓·͚
52. Termination Reason:Server.InternalError ͋Δ೔Πϯελϯε͕ىಈ͞Εͳ͘ͳͬͨ ͜ͷmessage͕ίϯιʔϧͷΞΫςΟϏςΟཤྺʹ Launching a new EC2 instance: i-xxxxxxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Server.InternalError why!? αϙʔτʹฉ͍ͯΈͨɻ
53. A. EBSͷ߹ܭ࢖༻ྔ͕ΞΧ΢ϯτͷ্ݶ௒͑
54. • AutoscalingͰཱͯΔΠϯελϯε͕࣋ͭϦιʔ εͷ্ݶ஋ʹୡͨ࣌͠΋ɺىಈʹࣦഊ͢Δ • ࣦͨͩ͠ഊཧ༝͸Console͔Β෼͔Γʹ͍͘ (Termination Reason: Server.InternalError)