Kubernetes拡張を利用した自作Autoscalerで実現するストレスフリーな運用の世界

,VCFSOFUFT֦ுΛར༻ͨࣗ͠࡞"VUPTDBMFSͰ ࣮ݱ͢ΔετϨεϑϦʔͳӡ༻ͷੈք .BZV:BNBNPUPc$ZCFSBHFOU *OD $MPVE/BUJWF%BZT5PLZP +VMZ

c1SPpMF ‣ ࢁຊ ·Ώ(@mayusy) ‣ גࣜձࣾCyberagent ΞυςΫຊ෦(2018.06~) ‣ Software Engineer
ओʹಈը޿ࠂͷ഑৴ϓϥοτϑΥʔϜ౳Λ։ൃ ‣ Go/ C#/ Kubernetes/ alcohol /PX d $ZCFSBHFOUΞυςΫຊ෦ʹ+PJO d ৽ଔͰ๭4ࣾʹ+PJO ࣗ࡞"VUPTDBMFS։ൃ

w ฎνʔϜͷΞʔΩςΫνϟ্ɺ͔ͳΓ($1ͷٕज़ʹدͬͨӡ༻ࣄྫʹ ͳΓ·͢ w ($1ͷར༻͸ฎνʔϜʹ͓͚Δ࿩Ͱ͋Γɺ$ZCFSBHFOUʹ͸($1Ҏ֎ ͷํ๏ͰΠϯϑϥΛߏங͍ͯ͠Δ૊৫΋ଘࡏ͠·͢ c͸͡Ίʹ

‣νʔϜͷϓϩμΫτͷ͝঺հ ‣ࣗ࡞"VUPTDBMFS͕ඞཁʹͳͬͨഎܠ ‣(FNJOJͷ֓ཁͱ࣮૷ ‣࣮ࡍͷӡ༻݁Ռ ‣·ͱΊ c"HFOEB

νʔϜͷϓϩμΫτ ͷ͝঺հ

ಈը޿ࠂͷ ഑৴ϓϥοτϑΥʔϜ ޿ࠂ΁ͷಋઢ ഑৴׬ྃϖʔδ cϓϩμΫτ֓ཁ

‣$MPVE1VC4VC w ($1Ͱఏڙ͞Ε͍ͯΔϝοηʔδϯάαʔϏε w 5PQJDɿ1VCMJTIFS͔ΒͷϝοηʔδΛड৴ w 4VCTDSJQUJPOɿ4VCTDSJCFS΁ϝοηʔδΛૹ৴

‣$MPVE#JHUBCMF w($1ͷετϨʔδαʔϏεͷҰͭɽ wεέʔϥϒϧͰߴ͍εϧʔϓοτΛތΔ,74 w/PEF୯ҐͰεέʔϧ͍ͯ͘͠ɽ ,VCFSOFUFTͷ/PEFͱ͸ผ෺ w"VUPTDBMFػೳ͸༻ҙ͞Ε͍ͯͳ͍

Cloud Pub/Sub Publisher Cloud Pub/SubΛ࢖ͬͯඇಉظ ʹϝοηʔδͷॲཧΛߦ͏. Serviceؒͷίϛϡχέʔγϣϯ ͸gRPCͰߦ͏ cΞʔΩςΫνϟ֓ཁ Service
Subscriber Cloud Bigtable Subscriber͸Bigtableʹσʔλͷ ಡΈॻ͖΋ߦͳ͍ͬͯΔ

ࣗ࡞Autoscaler͕ ඞཁʹͳͬͨഎܠ

͢Έ·ͤΜɼ͋Γ͕ͱ͏͍͟͝·͢ɽ ͳΜ͔4VCTDSJCFS͕ґଘ͍ͯ͠Δ#JHUBCMFͷ஗ԆͰϝοηʔδ͕ ٧·ͬͯΔΈ͍ͨͰ͢Ͷɽ #JHUBCMFͷ/PEF਺·Ͱ૿΍͠·ͨ͠ɽ #JHUBCMFͷ஗ԆʹΑΓ4VCTDSJCFSͷMBUFODZ্͕͕ͬͨࢫͷΤϥʔ͕ग़Δ

͢Έ·ͤΜɼ͋Γ͕ͱ͏͍͟͝·͢ɽ εέʔϧͯ͠΋/PEF਺଍Γͯͳ͍Έ͍ͨͳͷͰɼ ߋʹʹ/PEF਺૿΍͠·ͨ͠ɽ #JHUBCMFͷεέʔϧޙ΋4VCTDSJCFSͷMBUFODZ͸ࢭ·Βͣ

#JHUBCMFͷ஗Ԇ΋ऩ·ͬͨͷͰ/PEF਺ΛݮΒ͓͖ͯ͠·ͨ͠ɽ εέʔϧͰ͖ͳ͍͜ͱ͚ͩ͡Όͳͯ͘ɼ /PEF਺͕ଟ͗͢Δࡍʹແବͳίετ͕͔͔ͬͯ͠·͏͜ͱ΋໰୊Ͱ͢Ͷɽ #JHUBCMFͷ஗Ԇ͕ऩଋͨ͠ޙ

HPA Subscriber ᶃpublisherΛ௨ͯ͠Pub/Sub ʹϝοηʔδ͕དྷΔ Subscriber Subscriber ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ Publisher c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊ $MPVE1VC4VC

HPA Subscriber Subscriber Subscriber ᶄSubscriberʹେྔͷ ϝοηʔδ͕ૹ৴͞ΕΔ Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊

ᶅBigtable΁ͷϦΫΤετ ΋େྔʹൃੜ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ
c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊

$MPVE#JHUBCMF ᶆSubscriber͔ΒͷϦΫΤετ૿Ճ ʹΑΓBigtableͷෛՙ͕૿Ճ HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

ᶇSubscriberͷະॲཧͷ ϝοηʔδ͕૿Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ

ᶇHuman Bigtable Scaler (ਓྗ)ʹΑͬͯ  NodeΛखಈͰ௥Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber
Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊

ᶇNodeΛखಈͰ௥Ճ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ c͜ͷͱ͖ϓϩμΫτͰى͖͍ͯͨ໰୊
ਓྗͷ࡞ۀ͕සൟʹൃੜ͢ΔΑ͏ʹͳͬͯ͠·ͬͨͨΊɼ ΍͸Γ#JHUBCMFͷ"VUPTDBMFS͸ඞཁ

cϓϩάϥϜ͔Β#JHUBCMFΛTDBMF͢Δํ๏ͷެࣜ৘ใ ϓϩάϥϜ্͔Β4DBMF͢Δํ๏͸͋Δ͕ɼBVUPTDBMFػೳ͸ࣗલͰ࣮૷͢Δඞཁ͕͋Δ  ࢀߟɿIUUQTDMPVEHPPHMFDPNCJHUBCMFEPDTTDBMJOH

w#JHUBCMFͷ$16࢖༻཰ΛτϦΨʔͱͯ͠#JHUBCMFΛBVUPTDBMF͢Δɽ w,VCFSOFUFTͷϦιʔεঢ়گΛτϦΨʔͱͯ͠ɼ#JHUBCMFΛBVUPTDBMF͢Δɽ c#JHUBCMFͷࣗ࡞"VUPTDBMFSͰ࣮ݱ͍ͨ͜͠ͱ ӡ༻ෛՙͷܰݮˍ༨෼ͳϦιʔεͷ࢖༻ʹΑΔແବͳίετͷ࡟ݮ

Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Subscriber Publisher $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ
Subscriberෆ଍ʹΑΓ ϝοηʔδͷॲཧ͕஗Ԇ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ CPU/RAMෛՙ͕ߴ͍Θ͚Ͱ͸ͳ͍ͷͰ σϑΥϧτͰ͸Φʔτεέʔϧ͠ͳ͍

Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Publisher Subscriber $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ
Human Pod Autoscaler (ਓྗ)ʹΑͬͯ PodΛεέʔϧ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ Subscriber ϦΫΤετ਺౳ΛτϦΨʔ ͱͯ͠TDBMF͍ͨ͠

Bigtableʹ͸Ϧιʔεత ͳ༨༟͕͋Δ $MPVE#JHUBCMF HPA Subscriber Subscriber Publisher Subscriber $MPVE1VC4VC ‣֎෦ʹϦΫΤετΛૹ৴͢ΔॲཧΛ൐͏Subscriberʹͯ
Human Pod Autoscaler (HPA)ʹΑͬͯ PodΛεέʔϧ c೚ҙͷϝτϦΫεىҼͰ1PEΛεέʔϧ͍ͨ͠ Subscriber ͜͜Ͱ΋ਓྗͷ࡞ۀ͕සൟʹൃੜ͢ΔΑ͏ʹͳͬͯ͠·ͬͨͨΊɼ ೚ҙͷϝτϦΫεͰͷ1PEͷ"VUPTDBMFS͕ඞཁͱͳͬͨ

Geminiͷ ֓ཁͱ։ൃͷৄࡉ

KubernetesͷCRDͱ͍͏ػೳΛར༻ͨ͠Custom Autoscaler ‣ػೳ • ೚ҙͷϝτϦΫεىҼͰͷBigtableͷautoscale • ೚ҙͷϝτϦΫεىҼͰͷPod autoscale c(FNJOJͱ͸

,VCFSOFUFTͷ"1*ΛΧελϚΠζͰ͖Δػೳͷ͏ͪͷҰͭɽ ‣"1*&YUFOTJPO w $VTUPN3FTPVSDF%FpOJUJPO $VTUPN$POUSPMMFS w "VUIFOUJDBUJPO 5PLFO3FWJFX"1* w
"VUIPSJ[BUJPO 4VCKFDU"DDFTT3FWJFX"1* w "ENJTTJPO$POUSPM c$3%ͷ֓ཁ

w Ϣʔβʔಠࣗͷ3FTPVSDF5ZQFΛఆٛͰ͖Δػೳ w ಠࣗͷϚχϑΣετϑΝΠϧΛѻ͏͜ͱ͕ՄೳʹͳΔɽ c$VTUPN3FTPVSDF%FpOJUJPO

w Ϣʔβʔ͕ಠࣗʹߏஙͨ͠$POUSPMMFSɽ w ಛఆͷλεΫΛࣗಈԽͰ͖Δɽ w $VTUPN3FTPVSDFͱ૊Έ߹Θͤͯɼଞ$POUSPMMFSͱಉ༷ʹ$POUSPM MPPQ 3FDPODJMJBUJPO-PPQ Λ࣮૷͢Δ͜ͱ͕Մೳɽ c$VTUPN$POUSPMMFS

Gemini Resource 1 Gemini … Gemini Resource 2 Gemini Resource
3 ‣TDBMFઃఆͷऔಘ w$VTUPN$POUSPMMFSͷ$POUSPMMPPQͷதͰܧଓతʹऔಘ w֤(FNJOJ3FTPVSDF͝ͱʹεέʔϧର৅΍NJONBY஋ͷઃఆ͕Մೳ

‣ϝτϦΫεͷ࣌ܥྻσʔλͷऩू w4UBDLESJWFS.POJUPSJOHͱ࿈ܞ wΞϥʔτϙϦγʔ͕ࢀর͢ΔϝτϦΫεͷ࣌ܥྻσʔλΛऔಘ w ྫɿ#JHUBCMFDQVVTBHF #JHUBCMFͷϝτϦΫε w ྫɿະॲཧͷϝοηʔδ਺ 1VC4VCͷϝτϦΫε
wεέʔϧ൑ఆʹ࢖༻͢ΔΞϥʔτϙϦγʔ͸ઃఆϑΝΠϧͰࢦఆՄೳ

‣ $VTUPNSFTPVSDF΁ͷओͳઃఆ߲໨ #JHUBCMFTDBMFฤ w ؂ࢹର৅ͷJOTUBODFOBNF DMVTUFSOBNF w /PEF਺ͷNJONBY஋ w
TDBMFͷτϦΨʔͱ͍ͨ͠ΞϥʔτͷϙϦγʔ*% BQJ7FSTJPOBVUPTDBMFSDZCFSBHFOUDPKQWCFUB LJOEHFNJOJDVTUPNSFTPVSDF໊ NFUBEBUB MBCFMT DPOUSPMMFSUPPMTLTJP OBNFHFNJOJTBNQMF TQFD DMVTUFS/BNFYYYYYYYYYY JOTUBODF/BNFYYYYYYYYYY NBY/PEFT NJO/PEFT QPMJDZ*EYYYYYYYYYYTUBDLESJWFS্ͰϙϦγʔ͝ͱʹৼΒΕΔ*% ઃఆྫ

‣ $VTUPNSFTPVSDF΁ͷओͳઃఆ߲໨ 1PETDBMFฤ w ؂ࢹର৅ͷEFQMPZNFOUOBNF w 3FQMJDB਺ͷNJONBY஋ w TDBMFͷτϦΨʔͱ͍ͨ͠ΞϥʔτͷϙϦγʔ*%
BQJ7FSTJPOBVUPTDBMFSDZCFSBHFOUDPKQWCFUB LJOEHFNJOJDVTUPNSFTPVSDF໊ NFUBEBUB MBCFMT DPOUSPMMFSUPPMTLTJP OBNFHFNJOJTBNQMF TQFD EFQMPZNFOU/BNFYYYYYYYYYY NBY3FQMJDBT NJO3FQMJDBT QPMJDZ*EYYYYYYYYYYTUBDLESJWFS্ͰϙϦγʔ͝ͱʹৼΒΕΔ*% ઃఆྫ

‣εέʔϧϩδοΫ w աڈͷ࣌ܥྻσʔλ͕ܧଓͯ͠ᮢ஋ ΞϥʔτϙϦγʔʹઃఆ͞Ε ͍ͯΔᮢ஋ Λ௒͑Δ͔൱͔Ͱ൑ఆ w εέʔϧΞ΢τ͸ૣ͘ɼεέʔϧΠϯ͸Ώͬ͘Γͱߦ͏ w εύΠΫ൑ఆϩδοΫɼલճεέʔϧ͔ΒҰఆ࣌ؒ͸εέʔϧ͠ͳ
͍Α͏ʹ͢ΔϩδοΫ౳΋ಠࣗͰಋೖ #JHUBCMFΛTDBMFͨ͠ࡍͷ/PEF਺ͷมԽਤ εέʔϧΠϯ εέʔϧΞ΢τ

‣࣌ܥྻσʔλͷղੳ w ੜͷ࣌ܥྻσʔλͰ͸ϊΠζ͕ଟ͘ɼεέʔϧͷޡ൑ఆ͕ൃੜ͠΍͘͢ͳΔ ͨΊɼҠಈฏۉ 4." Λ͔͚ͯεϜʔδϯάͨ͠σʔλΛར༻ɽ w 4."ͷXJOEPXTJ[F ϓϩοτִؒ աڈԿ෼ͷσʔλΛ൑ఆʹ࢖༻͢Δ͔౳
ͷύϥϝʔλ͸ࡉ͔͍ௐ੔͕ඞཁɽ εύΠΫ෦෼ͷϊΠζ͕ ܰݮ͞Ε͍ͯΔ

GKE Gemini ᶃϝτϦΫεͷ࣌ܥྻσʔλͱ ొ࿥ࡁΈͷΞϥʔτϙϦγʔΛऔಘ #JHUBCMF ᶄϝτϦΫε͕ϙϦγʔͷᮢ஋Λ Ұఆ࣌ؒ࣋ଓͯ͠௒͑Δ͔൱͔Λ൑ఆ ᶅBigtableͷAPIΛୟ͍ͯ  Node਺Λߋ৽ c"VUPTDBMF࣌ͷॲཧϑϩʔ
#JHUBCMFฤ

kube-apiserver ᶃϝτϦΫεͷ࣌ܥྻσʔλͱ ొ࿥ࡁΈͷΞϥʔτϙϦγʔΛऔಘ Node Pod Pod Gemini c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ

ᶄϝτϦΫε͕ϙϦγʔͷᮢ஋Λ Ұఆ࣌ؒ࣋ଓͯ͠௒͑Δ͔൱͔Λ൑ఆ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini kube-apiserver

ᶅscale out/ inͷ൑ఆΛͨ͠৔߹͸ kube-apiserver্ͷdeploymentΛߋ৽ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini
kube-apiserver

ᶆScheduler͕Deploymentͷ৘ใ ʹ߹ΘͤͯPodΛNodeʹׂΓ౰ͯ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod Pod Gemini kube-apiserver

ᶇPod͕৽ͨʹ࡞ΒΕΔ c"VUPTDBMF࣌ͷॲཧϑϩʔ 1PEฤ Node Pod New Pod Gemini Pod kube-apiserver

w $VTUPN$POUSPMMFS͸LVCFCVJMEFS౳ͷ4%,Λ࢖༻ͤͣϑϧεΫϥονͰߏஙɽ w 4%,ͷ৘ใ͕গͳ͘ɼ·ͩར༻ࣄྫ΋গͳ͍࣌ظʹ࡞ͬͨͨΊɽ ࣮૷౰࣌ w (PͰ࣮૷ɽ w ओʹ࢖༻ͨ͠ϥΠϒϥϦ
w ࣌ܥྻσʔλ΍ϙϦγʔऔಘɿ4UBDLESJWFS.POJUPSJOH"1*$MJFOU-JCSBSJFT w EFQMPZNFOUߋ৽ɿLVCFSOFUFTDMJFOUHP c։ൃͷৄࡉ

1PEͷ4DBMFʹؔͯ͠͸ɼ౰ॳҎԼͷ࣮૷ํ๏΋ݕ౼͍ͯͨ͠ɽ ‣HPAͷCustom Metrics • Prometheus΍StackdriverͳͲͷϞχλϦϯάπʔϧ͔Β೚ҙͷϝτϦΫεΛऔಘͯ͠ɺ ᮢ஋ΛઃఆͰ͖Δɻ • ͭ·Γ೚ҙͷϝτϦΫεىҼͰΦʔτεέʔϧͰ͖ΔΑ͏ʹͳΔɻˠGeminiͱಉ͡ػೳ cٕज़બఆ

‣ԿނCustom MetricsͰ͸ͳ͘CRDΛ࠾༻ͨ͠ͷ͔ • BigtableͷNodeΛεέʔϧ͍ͤͨ͞ཉٻ΋͋ͬͨͷͰɺPodͷεέʔϧτϦΨʔ Λ૿΍ͤΔ͚ͩͰ͸ෆे෼ɽ • ୯७ʹCRDΛ࢖ͬͯΈ͔ͨͬͨɾCRDͷӡ༻ࣄྫΛ࡞Γ͔ͨͬͨɽ

࣮ࡍͷӡ༻݁Ռ

4DBMFPVU ͷᮢ஋ ˞ॾࣄ৘ʹΑΓ1PEͷTDBMFػೳͷํͷӡ༻݁Ռͷσʔλ͕͓ݟͤͰ͖ͳ͍ͷͰɼ  ࠓճ͸CJHUBCMFͷTDBMFͷํͷӡ༻݁ՌͷΈͷ͝঺հͱͳΓ·͢ɽ

Human ** Scalerͷग़ಈ͸ ௥Ճ Pod ௥Ճ

Human ** Scalerͷग़ಈ͸ ௥Ճ Pod ௥Ճ ΄΅θϩʹ

‣ ྑ͔ͬͨ఺ w εύΠΫ൑ఆ౳ɼಠࣗͷεέʔϧ൑ఆج४ͷಋೖ͕Մೳɽ w εέʔϧτϦΨʔΛॊೈʹ௥ՃͰ͖Δɽ w ྫɿؔ࿈͢ΔଞαʔϏεͷϦΫΤετ૿ՃΛࣗαʔϏεͷTDBMFPVUτϦΨʔʹ͢Δɽ ‣ ۤ࿑ͨ͠఺
w τϦΨʔΛ૿΍͢͜ͱͰɼ$POUSPMMFSͷεέʔϧϩδοΫ͕ෳࡶʹͳΔɽ w ҆ఆͨ͠εέʔϧϩδοΫʹ͢ΔͨΊʹௐ੔ͱվળΛ܁Γฦ͢ඞཁ͕͋ΔͨΊɼ҆ఆՔಈ Ͱ͖ΔϨϕϧʹͳΔ·Ͱʹ͸खؒͱ͕͔͔࣌ؒΔɽ cࣗ࡞"VUPTDBMFSͷ։ൃΛऴ͑ͯ

·ͱΊ

• ӡ༻ෆՄܰݮͱίετ࡟ݮͷͨΊʹɼCRDΛ࢖ͬͯPod΍BigtableΛscaleͰ͖ ΔAutoscalerΛࣗ࡞ͨ͠ɽ • ͜Ε·ͰखಈͰߦ͍͍͔ͬͯͨͭ͘ͷӡ༻͕΄΅ࣗಈԽͰ͖ΔΑ͏ʹͳͬͨɽ • AutoscalerΛࣗ࡞͢ΔͱεέʔϧϩδοΫ΍τϦΨʔʹॊೈੑΛ࣋ͨͤΒΕΔ ҰํͰɼ҆ఆՔಈ·Ͱʹ࣌ؒͱख͕͔͔ؒΔɽ • ࠓޙOSSԽ౳΋ࢹ໺ʹ͍ΕͯܧଓతͳվળΛଓ͚͍͖͍ͯͨɽ
c·ͱΊ

Thank you!

Kubernetes拡張を利用した自作Autoscalerで実現するストレスフリーな運用の世界

Kubernetes拡張を利用した自作Autoscalerで実現するストレスフリーな運用の世界

Other Decks in Technology

Featured

Transcript