PrometheusでKubernetesの高速オートスケールを実現した話

Transcript

1. ©2018 Wantedly, Inc. PrometheusͰKubernetesͷߴ଎ΦʔτεέʔϧΛ࣮ݱͨ͠࿩ Cloud Native Meetup Tokyo #2 LT 1.Jun.2018 - Atsushi Tanaka @bgpat
2. ©2018 Wantedly, Inc. Atsushi Tanaka • @bgpat • Infrastructure Engineer
 at Wantedly, Inc. • Favorites: Kubernetes, Docker, Golang
 JavaScript, Vue.js
3. ©2018 Wantedly, Inc. w Φʔτεέʔϧ͕஗͍
   w ଎͘εέʔϧ͢Δํ๏
   w ϓϩμΫγϣϯͰ࢖ͬͯΈͨ
   ࿩͢͜ͱ
4. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ Horizontal Pod Autoscaler (HPA) + CPU $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ
5. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ ϝϦοτ
   w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍
   w ݹ͍όʔδϣϯͰ΋࢖͑Δ Horizontal Pod Autoscaler (HPA) + CPU
6. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ ϝϦοτ
   w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍
   w ݹ͍όʔδϣϯͰ΋࢖͑Δ σϝϦοτ
   w ॠؒతͳεύΠΫʹऑ͍ Horizontal Pod Autoscaler (HPA) + CPU
7. ©2018 Wantedly, Inc.
8. ©2018 Wantedly, Inc.
9. ©2018 Wantedly, Inc.
10. ©2018 Wantedly, Inc. શવؒʹ߹͍ͬͯͳ͍
11. ©2018 Wantedly, Inc. ΋ͬͱૣ͘εέʔϧͯ͠΄͍͠
12. ©2018 Wantedly, Inc. εύΠΫʹڧ͍Φʔτεέʔϧ HPA + Custom Metrics ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ w W͔Β࢖͑Δ
    w ྫ͑͹ʜ
    ϦΫΤετ਺
    ίωΫγϣϯ਺
    w Ξμϓλ͕ඞཁ
13. ©2018 Wantedly, Inc. εύΠΫʹڧ͍Φʔτεέʔϧ HPA + Custom Metrics ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ w W͔Β࢖͑Δ
    w ྫ͑͹ʜ
    ϦΫΤετ਺
    ίωΫγϣϯ਺
    w Ξμϓλ͕ඞཁ
    ˠ
    ,VCFSOFUFT
    ʹϝτϦΫεΛૹΔͨΊͷαʔόʔ
14. ©2018 Wantedly, Inc. HPA + Custom Metrics k8s-prometheus-adapter 1SPNFUIFVT͕ऩूͨ͠ϝτϦΫεΛ,VCFSOFUFTʹૹͬͯ͘ΕΔαʔόʔ
15. ©2018 Wantedly, Inc. ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ
 Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ
16. ©2018 Wantedly, Inc. ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ
 Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ Ͱ͖Ε͹ΠϯϑϥϨΠϠʔͰղܾ͍ͨ͠
17. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ /HJOY
    ͷ
    TUVC@TUBUVT location /basic_status { stub_status; } Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106
18. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ /HJOY
    ͷ
    TUVC@TUBUVT location /basic_status { stub_status; } Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏
19. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ TUVC@TUBUVT
    Λ
    FYQPSUFS
    ͷܗࣜʹ͍ͨ͠ # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
20. ©2018 Wantedly, Inc. ,VCFSOFUFST
 "1*
    4FSWFS LTQSPNFUIFVTBEBQUFS 1SPNFUIFVT )PSJ[POUBM
    1PE
    "VUPTDBMFS %FQMPZNFOU 3FQMJDB4FU 1PE 1PE 1PE OHJOY@FYQPSUFS /HJOY
 TUVC@TUBUVT
21. ©2018 Wantedly, Inc. ݁Ռ w සൟʹεέʔϧ͢ΔΑ͏ʹͳͬͨ
    w $16ෛՙ͕҆ఆͨ͠
    w Ξϥʔτ͕໐Βͳ͘ͳͬͨ
    
22. ©2018 Wantedly, Inc. ߴ଎Φʔτεέʔϧ͕࣮ݱͰ͖ͨ
23. ©2018 Wantedly, Inc. https://github.com/bgpat/k8s-hpa-example