HPA with custom metrics

4cb5b950cfd0eabf5d6b828c951d4549?s=47 loftkun
November 27, 2019

HPA with custom metrics

2019/11/27 CloudNative Days Kansai 2019 Meetup
https://cloudnativedays.connpass.com/event/152321/

4cb5b950cfd0eabf5d6b828c951d4549?s=128

loftkun

November 27, 2019
Tweet

Transcript

  1. DVTUPNNFUSJDTʹΑΔ )1"ʹֶͿ 0QFSBUPSͱ"EBQUFS CloudNative Days Kansai 2019 Meetup 2019/11/27 @loftkun

  2. ࣗݾ঺հ !MPGULVO ߕ൹ ৽ޛ Ϡϑʔגࣜձࣾ αʔϏεϓϥοτϑΥʔϜຊ෦ ࡢ೔-5͖ͯ͠·ͨ͠ʂ

  3. .PUJWBUJPO • )1"ͷυΩϡϝϯτΛݟͯ ɾ$16࢖༻཰ʹΑΔ)1" ɾ ಈ͔ͤͨ ɾDVTUPNNFUSJDTʹΑΔ)1" ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ

  4. .PUJWBUJPO • )1"ͷυΩϡϝϯτΛݟͯ ɾ$16࢖༻཰ʹΑΔ)1" ɾ ಈ͔ͤͨ ɾDVTUPNNFUSJDTʹΑΔ)1" ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ

  5. .PUJWBUJPO • )1"ͷυΩϡϝϯτΛݟͯ ɾ$16࢖༻཰ʹΑΔ)1" ɾ ಈ͔ͤͨ ɾDVTUPNNFUSJDTʹΑΔ)1" ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ

    • 0QFSBUPS΍"EBQUFSύλʔϯͷ஌ݟΛಘΔ͜ͱ ͕Ͱ͖ͨͷͰڞ༗͠·͢
  6. &OWJSPONFOU .JOJLVCF

  7. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

  8. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI FOBCMFNFUSJDTTFSWFS

  9. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI SVOFYQPTFQIQBQBDIFTFSWFS

  10. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI DSFBUF)1"

  11. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI JODSFBTFMPBE

  12. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI DIFDLSFQMJDBT εέʔϧͨ͠Θ

  13. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI DVTUPNNFUSJDT͸ʁ ۩ମతͳखॱ͸ͳ͍Θ ࣗલͰಠࣗͷϝτϦΫεΛ༻ҙ͠ͳ͍ͱ ͔͋Μ΍Ζ͜Ε ༻ҙͨ͠ϝτϦΫε͸Ͳ͏΍ͬͯ)1"ί ϯτϩʔϥʹࢀরͤ͞ΔΜ΍ʁ

  14. MVYBTLVCFBENXPSLTIPQ https://github.com/luxas/kubeadm-workshop

  15. MVYBTLVCFBENXPSLTIPQ • k8sクラスタを構築するチュートリアル • DVTUPNNFUSJDTʹΑΔ)1"ͷνϡʔτϦΞϧ ͋Γʂ – l%FQMPZJOHUIF1SPNFUIFVT0QFSBUPSGPSNPOJUPSJOH4FSWJDFTJOUIF DMVTUFSz –

    l%FQMPZJOHBDVTUPNNFUSJDT"1*4FSWFSBOEBTBNQMFBQQz
  16. TBNQMFNFUSJDTBQQZBNM ͜ͷϝτϦΫεͰ)1"͕ಈ࡞͢ΔΑ͏ ༻ҙͯ͘͠ΕͯΔΜ΍ͳɺ ͋Γ͕͍ͨΘʙ

  17. EFNPTNPOJUPSJOH

  18. EFNPTNPOJUPSJOH

  19. EFNPTNPOJUPSJOH DVTUPNNFUSJDTͰ)1"ಈ͍ͯΔΘ ͑͑΍Μ

  20. &01ɾɾɾɾʁ

  21. QSPNFUIFVTPQFSBUPSZBNM Operator?

  22. TBNQMFNFUSJDTBQQZBNM kind : ServiceMonitor ?

  23. TBNQMFQSPNFUIFVTJOTUBODFZBNM kind : Prometheus ?

  24. DVTUPNNFUSJDTZBNM Adapter?

  25. ཧղ͍ͨ͠ͱ͜Ζ HPA contorller Deployment Pod 1 Pod 2 Pod N

    ɾɾɾ Custom metrics Server HFUNFUSJDT DPMMFDUNFUSJDT TDBMFSFQMJDBT
  26. ཧղ͍ͨ͠ͱ͜Ζ HPA contorller Deployment Pod 1 Pod 2 Pod N

    ɾɾɾ Custom metrics Server How to ? HFUNFUSJDT TDBMFSFQMJDBT DPMMFDUNFUSJDT
  27. ཧղ͍ͨ͠ͱ͜Ζ HPA contorller Deployment Pod 1 Pod 2 Pod N

    ɾɾɾ Custom metrics Server How to ? How to ? HFUNFUSJDT TDBMFSFQMJDBT DPMMFDUNFUSJDT
  28. ੔ཧͯ͠Έͨ ϚχϑΣετZBNM ˣ &YDFMͷςΩετϘοΫε ؔ܎ੑͷ੔ཧ͸ࠓͷͱ͜Ζਓྗ͕ඞཁ ͳΜ΍ ࣗಈԽ͍ͨ͠

  29. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ s

    e r v i c e
  30. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator s e r v i c e
  31. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator kind: Prometheus kind: Service Monitor s e r v i c e
  32. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator kind: Service Monitor s e r v i c e monitoring kind: Prometheus
  33. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator kind: Prometheus kind: Service Monitor s e r v i c e check
  34. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator kind: Prometheus kind: Service Monitor s e r v i c e create config
  35. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Prometheus

    Operator kind: Prometheus kind: Service Monitor s e r v i c e
  36. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e s e r v i c e get metrics
  37. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e s e r v i c e JNBHFDirectXMan12/k8s-prometheus-adapter
  38. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e s e r v i c e Kind: APIService
  39. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e kube-api server aggregation layer s e r v i c e Kind: APIService
  40. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e kube-api server aggregation layer s e r v i c e Kind: APIService
  41. )1"XJUIDVTUPNNFUSJDT Deployment Pod 1 Pod 2 Pod N ɾɾɾ Custom

    metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e kube-api server aggregation layer s e r v i c e Kind: APIService HPA contorller
  42. ཧղ͍ͨ͠ͱ͜Ζ HPA contorller Deployment Pod 1 Pod 2 Pod N

    ɾɾɾ Custom metrics Server How to ? How to ? HFUNFUSJDT TDBMFSFQMJDBT DPMMFDUNFUSJDT
  43. )1"XJUIDVTUPNNFUSJDT TDBMFSFQMJDBT Deployment Pod 1 Pod 2 Pod N ɾɾɾ

    Custom metrics Server Prometheus Operator kind: Prometheus kind: Service Monitor s e r v i c e s e r v i c e kube-api server s e r v i c e Kind: APIService HPA contorller DPMMFDUNFUSJDT HFUNFUSJDT aggregation layer
  44. 0QFSBUPSʁ 5XJUUFS࣮گ͕ࢭ·Δ΍ͭ

  45. 0QFSBUPSΛσϓϩΠ͢Δͱ $3%Λ࡞ͬͯ͘ΕΔ

  46. զʑ͸$3%ΦϒδΣΫτΛ DSFBUFͰ͖ΔΑ͏ʹͳΔ

  47. $3%ΦϒδΣΫτΛ ӡ༻ͯ͘͠ΕΔͷ͕0QFSBUPS Prometheus Operator kind: Prometheus kind: Service Monitor s

    e r v i c e Deployment 1SPNFUIFVT 0QFSBUPSͷ৔߹ 4FSWJDF.POJUPSʹ ର৅TFSWJDFͱFOEQPJOUΛઃఆ͓ͯ͘͠ͱ 0QFSBUPS͕ QSPNFUIFVTZBNMΛॻ͍ͯ͘ΕΔʂ ӡ༻͕ίʔυͰ࣮૷͞Ε͍ͯΔΜͩʂ
  48. "EBQUFSʁ

  49. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS ޓ׵ੑ͕ͳ͍ Prometheus kube-api server aggregation layer

  50. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS 1SPNFUIFVT"EBQUFS Custom metrics Server Prometheus kube-api server aggregation layer

     Adapterパターンだ!
  51. ·ͱΊ • 0QFSBUPSͷྫͱͯ͠ɺ1SPNFUIFVT 0QFSBUPSʹΑΔϝτϦΫεͷऩूํ๏͕෼ ͔ͬͨΜ΍ • "EBQUFSύλʔϯͷ۩ମతͳࣄྫΛ஌ͬͨΜ΍ • DVTUPNNFUSJDTʹΑΔ)1"Λಈ͔͢ํ๏͕෼ ͔ͬͨΜ΍

  52. 0QFSBUPSʁ ࣮گ͕੝Γ্͕Ε͹޾͍Ͱ͢ ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ $/%,ͷ-5Ͱݟͨ΍ͭ΍ʂ $/%,Ͱ׬શʹཧղͨ͠Θ

  53. "QQFOEJY

  54. 1PE"VUPTDBMFS

  55. 1PE"VUPTDBMFS SFTPVSDF Pod 1 DQV NFNPSZ

  56. 1PE"VUPTDBMFS Pod 1 SFTPVSDF 7FSUJDBM DQV NFNPSZ

  57. 1PE"VUPTDBMFS SFQMJDBT Pod 1

  58. 1PE"VUPTDBMFS Horizontal SFQMJDBT Pod 1 Pod 2 Pod N ɾɾɾ