$30 off During Our Annual Pro Sale. View Details »

HPA with custom metrics

loftkun
November 27, 2019

HPA with custom metrics

2019/11/27 CloudNative Days Kansai 2019 Meetup
https://cloudnativedays.connpass.com/event/152321/

loftkun

November 27, 2019
Tweet

More Decks by loftkun

Other Decks in Technology

Transcript

  1. DVTUPNNFUSJDTʹΑΔ
    )1"ʹֶͿ
    0QFSBUPSͱ"EBQUFS
    CloudNative Days Kansai 2019 Meetup
    2019/11/27 @loftkun

    View Slide

  2. ࣗݾ঺հ
    !MPGULVO ߕ൹ ৽ޛ

    Ϡϑʔגࣜձࣾ αʔϏεϓϥοτϑΥʔϜຊ෦
    ࡢ೔-5͖ͯ͠·ͨ͠ʂ

    View Slide

  3. .PUJWBUJPO
    • )1"ͷυΩϡϝϯτΛݟͯ
    ɾ$16࢖༻཰ʹΑΔ)1"
    ɾ ಈ͔ͤͨ
    ɾDVTUPNNFUSJDTʹΑΔ)1"
    ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ

    View Slide

  4. .PUJWBUJPO
    • )1"ͷυΩϡϝϯτΛݟͯ
    ɾ$16࢖༻཰ʹΑΔ)1"
    ɾ ಈ͔ͤͨ
    ɾDVTUPNNFUSJDTʹΑΔ)1"
    ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ

    View Slide

  5. .PUJWBUJPO
    • )1"ͷυΩϡϝϯτΛݟͯ
    ɾ$16࢖༻཰ʹΑΔ)1"
    ɾ ಈ͔ͤͨ
    ɾDVTUPNNFUSJDTʹΑΔ)1"
    ɾ ۩ମతͳखॱ͕ͳ͔ͬͨ ௐ΂ͯΈͨ
    • 0QFSBUPS΍"EBQUFSύλʔϯͷ஌ݟΛಘΔ͜ͱ
    ͕Ͱ͖ͨͷͰڞ༗͠·͢

    View Slide

  6. &OWJSPONFOU .JOJLVCF

    View Slide

  7. )PSJ[POUBM1PE"VUPTDBMFS
    8BMLUISPVHI
    https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

    View Slide

  8. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    FOBCMFNFUSJDTTFSWFS

    View Slide

  9. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    SVOFYQPTFQIQBQBDIFTFSWFS

    View Slide

  10. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DSFBUF)1"

    View Slide

  11. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    JODSFBTFMPBE

    View Slide

  12. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DIFDLSFQMJDBT

    εέʔϧͨ͠Θ

    View Slide

  13. )PSJ[POUBM1PE"VUPTDBMFS 8BMLUISPVHI
    DVTUPNNFUSJDT͸ʁ

    ۩ମతͳखॱ͸ͳ͍Θ
    ࣗલͰಠࣗͷϝτϦΫεΛ༻ҙ͠ͳ͍ͱ
    ͔͋Μ΍Ζ͜Ε
    ༻ҙͨ͠ϝτϦΫε͸Ͳ͏΍ͬͯ)1"ί
    ϯτϩʔϥʹࢀরͤ͞ΔΜ΍ʁ

    View Slide

  14. MVYBTLVCFBENXPSLTIPQ
    https://github.com/luxas/kubeadm-workshop

    View Slide

  15. MVYBTLVCFBENXPSLTIPQ
    • k8sクラスタを構築するチュートリアル
    • DVTUPNNFUSJDTʹΑΔ)1"ͷνϡʔτϦΞϧ
    ͋Γʂ
    – l%FQMPZJOHUIF1SPNFUIFVT0QFSBUPSGPSNPOJUPSJOH4FSWJDFTJOUIF
    DMVTUFSz
    – l%FQMPZJOHBDVTUPNNFUSJDT"1*4FSWFSBOEBTBNQMFBQQz

    View Slide

  16. TBNQMFNFUSJDTBQQZBNM

    ͜ͷϝτϦΫεͰ)1"͕ಈ࡞͢ΔΑ͏
    ༻ҙͯ͘͠ΕͯΔΜ΍ͳɺ
    ͋Γ͕͍ͨΘʙ

    View Slide

  17. EFNPTNPOJUPSJOH

    View Slide

  18. EFNPTNPOJUPSJOH

    View Slide

  19. EFNPTNPOJUPSJOH
    DVTUPNNFUSJDTͰ)1"ಈ͍ͯΔΘ
    ͑͑΍Μ

    View Slide

  20. &01ɾɾɾɾʁ

    View Slide

  21. QSPNFUIFVTPQFSBUPSZBNM

    Operator?

    View Slide

  22. TBNQMFNFUSJDTBQQZBNM
    kind : ServiceMonitor ?

    View Slide

  23. TBNQMFQSPNFUIFVTJOTUBODFZBNM
    kind : Prometheus ?

    View Slide

  24. DVTUPNNFUSJDTZBNM

    Adapter?

    View Slide

  25. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    HFUNFUSJDT
    DPMMFDUNFUSJDT
    TDBMFSFQMJDBT

    View Slide

  26. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server

    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT

    View Slide

  27. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server


    How to ?
    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT

    View Slide

  28. ੔ཧͯ͠Έͨ

    ϚχϑΣετZBNM
    ˣ
    &YDFMͷςΩετϘοΫε
    ؔ܎ੑͷ੔ཧ͸ࠓͷͱ͜Ζਓྗ͕ඞཁ
    ͳΜ΍ ࣗಈԽ͍ͨ͠

    View Slide

  29. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    s
    e
    r
    v
    i
    c
    e

    View Slide

  30. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    s
    e
    r
    v
    i
    c
    e

    View Slide

  31. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e

    View Slide

  32. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    monitoring
    kind:
    Prometheus

    View Slide

  33. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    check

    View Slide

  34. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    create config

    View Slide

  35. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e

    View Slide

  36. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    get metrics

    View Slide

  37. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    JNBHFDirectXMan12/k8s-prometheus-adapter

    View Slide

  38. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService

    View Slide

  39. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService

    View Slide

  40. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService

    View Slide

  41. )1"XJUIDVTUPNNFUSJDT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    aggregation layer
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService HPA
    contorller

    View Slide

  42. ཧղ͍ͨ͠ͱ͜Ζ
    HPA
    contorller
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server


    How to ?
    How to ?
    HFUNFUSJDT
    TDBMFSFQMJDBT
    DPMMFDUNFUSJDT

    View Slide

  43. )1"XJUIDVTUPNNFUSJDT
    TDBMFSFQMJDBT
    Deployment
    Pod 1 Pod 2 Pod N
    ɾɾɾ
    Custom
    metrics
    Server
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    s
    e
    r
    v
    i
    c
    e
    kube-api server
    s
    e
    r
    v
    i
    c
    e
    Kind:
    APIService HPA
    contorller
    DPMMFDUNFUSJDT
    HFUNFUSJDT
    aggregation layer

    View Slide

  44. 0QFSBUPSʁ
    5XJUUFS࣮گ͕ࢭ·Δ΍ͭ

    View Slide

  45. 0QFSBUPSΛσϓϩΠ͢Δͱ
    $3%Λ࡞ͬͯ͘ΕΔ

    View Slide

  46. զʑ͸$3%ΦϒδΣΫτΛ
    DSFBUFͰ͖ΔΑ͏ʹͳΔ

    View Slide

  47. $3%ΦϒδΣΫτΛ
    ӡ༻ͯ͘͠ΕΔͷ͕0QFSBUPS
    Prometheus
    Operator
    kind:
    Prometheus
    kind:
    Service
    Monitor
    s
    e
    r
    v
    i
    c
    e
    Deployment
    1SPNFUIFVT 0QFSBUPSͷ৔߹
    4FSWJDF.POJUPSʹ
    ର৅TFSWJDFͱFOEQPJOUΛઃఆ͓ͯ͘͠ͱ
    0QFSBUPS͕
    QSPNFUIFVTZBNMΛॻ͍ͯ͘ΕΔʂ
    ӡ༻͕ίʔυͰ࣮૷͞Ε͍ͯΔΜͩʂ

    View Slide

  48. "EBQUFSʁ

    View Slide

  49. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS
    ޓ׵ੑ͕ͳ͍
    Prometheus
    kube-api server
    aggregation layer

    View Slide

  50. ޓ׵ੑΛఏڙ͢Δͷ͕"EBQUFS
    1SPNFUIFVT"EBQUFS
    Custom
    metrics
    Server
    Prometheus
    kube-api server
    aggregation layer

    Adapterパターンだ!

    View Slide

  51. ·ͱΊ
    • 0QFSBUPSͷྫͱͯ͠ɺ1SPNFUIFVT
    0QFSBUPSʹΑΔϝτϦΫεͷऩूํ๏͕෼
    ͔ͬͨΜ΍
    • "EBQUFSύλʔϯͷ۩ମతͳࣄྫΛ஌ͬͨΜ΍
    • DVTUPNNFUSJDTʹΑΔ)1"Λಈ͔͢ํ๏͕෼
    ͔ͬͨΜ΍

    View Slide

  52. 0QFSBUPSʁ
    ࣮گ͕੝Γ্͕Ε͹޾͍Ͱ͢
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ

    $/%,ͷ-5Ͱݟͨ΍ͭ΍ʂ

    $/%,Ͱ׬શʹཧղͨ͠Θ

    View Slide

  53. "QQFOEJY

    View Slide

  54. 1PE"VUPTDBMFS

    View Slide

  55. 1PE"VUPTDBMFS
    SFTPVSDF
    Pod 1
    DQV NFNPSZ

    View Slide

  56. 1PE"VUPTDBMFS
    Pod 1
    SFTPVSDF
    7FSUJDBM
    DQV NFNPSZ

    View Slide

  57. 1PE"VUPTDBMFS
    SFQMJDBT
    Pod 1

    View Slide

  58. 1PE"VUPTDBMFS
    Horizontal
    SFQMJDBT
    Pod 1 Pod 2 Pod N
    ɾɾɾ

    View Slide