PrometheusでKubernetesの高速オートスケールを実現した話

C3b68d6841dcd343be9685f1cdaf8ac9?s=47 Atsushi Tanaka
June 01, 2018
1k

 PrometheusでKubernetesの高速オートスケールを実現した話

C3b68d6841dcd343be9685f1cdaf8ac9?s=128

Atsushi Tanaka

June 01, 2018
Tweet

Transcript

  1. ©2018 Wantedly, Inc. PrometheusͰKubernetesͷߴ଎ΦʔτεέʔϧΛ࣮ݱͨ͠࿩ Cloud Native Meetup Tokyo #2 LT

    1.Jun.2018 - Atsushi Tanaka @bgpat
  2. ©2018 Wantedly, Inc. Atsushi Tanaka • @bgpat • Infrastructure Engineer


    at Wantedly, Inc. • Favorites: Kubernetes, Docker, Golang
 JavaScript, Vue.js
  3. ©2018 Wantedly, Inc. w Φʔτεέʔϧ͕஗͍ w ଎͘εέʔϧ͢Δํ๏ w ϓϩμΫγϣϯͰ࢖ͬͯΈͨ ࿩͢͜ͱ

  4. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ Horizontal Pod Autoscaler (HPA) + CPU

    $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ
  5. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ ϝϦοτ w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍ w ݹ͍όʔδϣϯͰ΋࢖͑Δ

    Horizontal Pod Autoscaler (HPA) + CPU
  6. ©2018 Wantedly, Inc. ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ ϝϦοτ w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍ w ݹ͍όʔδϣϯͰ΋࢖͑Δ

    σϝϦοτ w ॠؒతͳεύΠΫʹऑ͍ Horizontal Pod Autoscaler (HPA) + CPU
  7. ©2018 Wantedly, Inc.

  8. ©2018 Wantedly, Inc.

  9. ©2018 Wantedly, Inc.

  10. ©2018 Wantedly, Inc. શવؒʹ߹͍ͬͯͳ͍

  11. ©2018 Wantedly, Inc. ΋ͬͱૣ͘εέʔϧͯ͠΄͍͠

  12. ©2018 Wantedly, Inc. εύΠΫʹڧ͍Φʔτεέʔϧ HPA + Custom Metrics ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ w

    W͔Β࢖͑Δ w ྫ͑͹ʜϦΫΤετ਺ ίωΫγϣϯ਺ w Ξμϓλ͕ඞཁ
  13. ©2018 Wantedly, Inc. εύΠΫʹڧ͍Φʔτεέʔϧ HPA + Custom Metrics ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ w

    W͔Β࢖͑Δ w ྫ͑͹ʜϦΫΤετ਺ ίωΫγϣϯ਺ w Ξμϓλ͕ඞཁˠ,VCFSOFUFTʹϝτϦΫεΛૹΔͨΊͷαʔόʔ
  14. ©2018 Wantedly, Inc. HPA + Custom Metrics k8s-prometheus-adapter 1SPNFUIFVT͕ऩूͨ͠ϝτϦΫεΛ,VCFSOFUFTʹૹͬͯ͘ΕΔαʔόʔ

  15. ©2018 Wantedly, Inc. ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ
 Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ

  16. ©2018 Wantedly, Inc. ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ
 Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ Ͱ͖Ε͹ΠϯϑϥϨΠϠʔͰղܾ͍ͨ͠

  17. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ /HJOYͷTUVC@TUBUVT location /basic_status { stub_status; }

    Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106
  18. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ /HJOYͷTUVC@TUBUVT location /basic_status { stub_status; }

    Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏
  19. ©2018 Wantedly, Inc. εέʔϧͷ൑ఆج४ TUVC@TUBUVTΛFYQPSUFSͷܗࣜʹ͍ͨ͠ # HELP nginx_connections_current Number of

    connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
  20. ©2018 Wantedly, Inc. ,VCFSOFUFST
 "1*4FSWFS LTQSPNFUIFVTBEBQUFS 1SPNFUIFVT )PSJ[POUBM 1PE "VUPTDBMFS

    %FQMPZNFOU 3FQMJDB4FU 1PE 1PE 1PE OHJOY@FYQPSUFS /HJOY
 TUVC@TUBUVT
  21. ©2018 Wantedly, Inc. ݁Ռ w සൟʹεέʔϧ͢ΔΑ͏ʹͳͬͨ w $16ෛՙ͕҆ఆͨ͠ w Ξϥʔτ͕໐Βͳ͘ͳͬͨ

  22. ©2018 Wantedly, Inc. ߴ଎Φʔτεέʔϧ͕࣮ݱͰ͖ͨ

  23. ©2018 Wantedly, Inc. https://github.com/bgpat/k8s-hpa-example