HPA with custom metrics

2019/11/27 CloudNative Days Kansai 2019 Meetup
https://cloudnativedays.connpass.com/event/152321/

Transcript

1. DVTUPNNFUSJDTʹΑΔ
   )1"ʹֶͿ
   0QFSBUPSͱ"EBQUFS
   CloudNative Days Kansai 2019 Meetup
   2019/11/27 @loftkun
   

   View Slide

2. ࣗݾ঺հ
   !MPGULVO ߕ൹ ৽ޛ
   
   Ϡϑʔגࣜձࣾ αʔϏεϓϥοτϑΥʔϜຊ෦
   ࡢ೔-5͖ͯ͠·ͨ͠ʂ
   

   View Slide

3. .PUJWBUJPO
   • )1"ͷυΩϡϝϯτΛݟͯ
   ɾ$16࢖༻཰ʹΑΔ)1"
   ɾ ಈ͔ͤͨ
   ɾDVTUPNNFUSJDTʹΑΔ)1"
   ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ
   

   View Slide

4. .PUJWBUJPO
   • )1"ͷυΩϡϝϯτΛݟͯ
   ɾ$16࢖༻཰ʹΑΔ)1"
   ɾ ಈ͔ͤͨ
   ɾDVTUPNNFUSJDTʹΑΔ)1"
   ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ
   

   View Slide

5. .PUJWBUJPO
   • )1"ͷυΩϡϝϯτΛݟͯ
   ɾ$16࢖༻཰ʹΑΔ)1"
   ɾ ಈ͔ͤͨ
   ɾDVTUPNNFUSJDTʹΑΔ)1"
   ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ
   • 0QFSBUPS΍"EBQUFSύλʔϯͷ஌ݟΛಘΔ͜ͱ
   ͕Ͱ͖ͨͷͰڞ༗͠·͢
   

   View Slide

6. &OWJSPONFOU .JOJLVCF
   

   View Slide

7. )PSJ[POUBM1PE"VUPTDBMFS
   8BMLUISPVHI
   https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
   

   View Slide

8. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
   FOBCMFNFUSJDTTFSWFS
   

   View Slide

9. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
   SVOFYQPTFQIQBQBDIFTFSWFS
   

   View Slide

10. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DSFBUF)1"
    

    View Slide

11. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    JODSFBTFMPBE
    

    View Slide

12. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DIFDLSFQMJDBT
    
    εέʔϧͨ͠Θ
    

    View Slide

13. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DVTUPNNFUSJDT͸ʁ
    
    ۩ମతͳखॱ͸ͳ͍Θ
    ࣗલͰಠࣗͷϝτϦΫεΛ༻ҙ͠ͳ͍ͱ
    ͔͋Μ΍Ζ͜Ε
    ༻ҙͨ͠ϝτϦΫε͸Ͳ͏΍ͬͯ)1"ί
    ϯτϩʔϥʹࢀরͤ͞ΔΜ΍ʁ
    

    View Slide

14. MVYBTLVCFBENXPSLTIPQ
    https://github.com/luxas/kubeadm-workshop
    

    View Slide

15. MVYBTLVCFBENXPSLTIPQ
    • k8sクラスタを構築するチュートリアル
    • DVTUPNNFUSJDTʹΑΔ)1"ͷνϡʔτϦΞϧ
    ͋Γʂ
    – l%FQMPZJOHUIF1SPNFUIFVT0QFSBUPSGPSNPOJUPSJOH4FSWJDFTJOUIF
    DMVTUFSz
    – l%FQMPZJOHBDVTUPNNFUSJDT"1*4FSWFSBOEBTBNQMFBQQz
    

    View Slide

16. TBNQMFNFUSJDTBQQZBNM
    
    ͜ͷϝτϦΫεͰ)1"͕ಈ࡞͢ΔΑ͏
    ༻ҙͯ͘͠ΕͯΔΜ΍ͳɺ
    ͋Γ͕͍ͨΘʙ
    

    View Slide

17. EFNPTNPOJUPSJOH
    

    View Slide

18. EFNPTNPOJUPSJOH
    

    View Slide

19. EFNPTNPOJUPSJOH
    DVTUPNNFUSJDTͰ)1"ಈ͍ͯΔΘ
    ͑͑΍Μ
    
    

    View Slide

20. &01ɾɾɾɾʁ
    

    View Slide

21. QSPNFUIFVTPQFSBUPSZBNM
    
    Operator？
    

    View Slide

22. TBNQMFNFUSJDTBQQZBNM
    kind : ServiceMonitor ？
    

    View Slide

23. TBNQMFQSPNFUIFVTJOTUBODFZBNM
    kind : Prometheus ？
    

    View Slide

24. DVTUPNNFUSJDTZBNM
    
    Adapter？
    

    View Slide

25. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    HFUNFUSJDT
    DPMMFDUNFUSJDT
    TDBMFSFQMJDBT
    

    View Slide

26. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    
    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT
    

    View Slide

27. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    
    
    How to ?
    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT
    

    View Slide

28. ੔ཧͯ͠Έͨ
    
    ϚχϑΣετZBNM
    ˣ
    &YDFMͷςΩετϘοΫε
    ؔ܎ੑͷ੔ཧ͸ࠓͷͱ͜Ζਓྗ͕ඞཁ
    ͳΜ΍ ࣗಈԽ͍ͨ͠
    
    

    View Slide

29. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    s
    e
    r
    v
    i
    c
    e
    

    View Slide

30. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    s
    e
    r
    v
    i
    c
    e
    

    View Slide

31. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    

    View Slide

32. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    monitoring
    kind:
    Prometheus
    

    View Slide

33. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    check
    

    View Slide

34. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    create config
    

    View Slide

35. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    

    View Slide

36. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    get metrics
    

    View Slide

37. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    JNBHFDirectXMan12/k8s-prometheus-adapter
    

    View Slide

38. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService
    

    View Slide

39. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService
    

    View Slide

40. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService
    

    View Slide

41. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService HPA
    contorller
    

    View Slide

42. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    
    
    How to ?
    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT
    

    View Slide

43. )1"XJUIDVTUPNNFUSJDT
    TDBMFSFQMJDBT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService HPA
    contorller
    DPMMFDUNFUSJDT
    HFUNFUSJDT
    aggregation layer
    

    View Slide

44. 0QFSBUPSʁ
    5XJUUFS࣮گ͕ࢭ·Δ΍ͭ
    

    View Slide

45. 0QFSBUPSΛσϓϩΠ͢Δͱ
    $3%Λ࡞ͬͯ͘ΕΔ
    

    View Slide

46. զʑ͸$3%ΦϒδΣΫτΛ
    DSFBUFͰ͖ΔΑ͏ʹͳΔ
    

    View Slide

47. $3%ΦϒδΣΫτΛ
    ӡ༻ͯ͘͠ΕΔͷ͕0QFSBUPS
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    Deployment
    1SPNFUIFVT 0QFSBUPSͷ৔߹
    4FSWJDF.POJUPSʹ
    ର৅TFSWJDFͱFOEQPJOUΛઃఆ͓ͯ͘͠ͱ
    0QFSBUPS͕
    QSPNFUIFVTZBNMΛॻ͍ͯ͘ΕΔʂ
    ӡ༻͕ίʔυͰ࣮૷͞Ε͍ͯΔΜͩʂ
    

    View Slide

48. "EBQUFSʁ
    

    View Slide

49. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS
    ޓ׵ੑ͕ͳ͍
    Prometheus
    kube-api server
    aggregation layer
    

    View Slide

50. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS
    1SPNFUIFVT"EBQUFS
    Custom
    metrics
    Server
    Prometheus
    kube-api server
    aggregation layer
    
    Adapterパターンだ！
    
    

    View Slide

51. ·ͱΊ
    • 0QFSBUPSͷྫͱͯ͠ɺ1SPNFUIFVT
    0QFSBUPSʹΑΔϝτϦΫεͷऩूํ๏͕෼
    ͔ͬͨΜ΍
    • "EBQUFSύλʔϯͷ۩ମతͳࣄྫΛ஌ͬͨΜ΍
    • DVTUPNNFUSJDTʹΑΔ)1"Λಈ͔͢ํ๏͕෼
    ͔ͬͨΜ΍
    

    View Slide

52. 0QFSBUPSʁ
    ࣮گ͕੝Γ্͕Ε͹޾͍Ͱ͢
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    
    $/%,ͷ-5Ͱݟͨ΍ͭ΍ʂ
    
    $/%,Ͱ׬શʹཧղͨ͠Θ
    

    View Slide

53. "QQFOEJY
    

    View Slide

54. 1PE"VUPTDBMFS
    

    View Slide

55. 1PE"VUPTDBMFS
    SFTPVSDF
    Pod 1
    DQV NFNPSZ
    

    View Slide

56. 1PE"VUPTDBMFS
    Pod 1
    SFTPVSDF
    7FSUJDBM
    DQV NFNPSZ
    

    View Slide

57. 1PE"VUPTDBMFS
    SFQMJDBT
    Pod 1
    

    View Slide

58. 1PE"VUPTDBMFS
    Horizontal
    SFQMJDBT
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    

    View Slide