PrometheusでKubernetesの高速オートスケールを実現した話

Transcript

1. ©2018 Wantedly, Inc.
   PrometheusͰKubernetesͷߴ଎ΦʔτεέʔϧΛ࣮ݱͨ͠࿩
   Cloud Native Meetup Tokyo #2 LT
   1.Jun.2018 - Atsushi Tanaka @bgpat
   

   View Slide

2. ©2018 Wantedly, Inc.
   Atsushi Tanaka
   • @bgpat
   • Infrastructure Engineer

   at Wantedly, Inc.
   • Favorites:
   Kubernetes, Docker, Golang

   JavaScript, Vue.js
   

   View Slide

3. ©2018 Wantedly, Inc.
   w Φʔτεέʔϧ͕஗͍
   w ଎͘εέʔϧ͢Δํ๏
   w ϓϩμΫγϣϯͰ࢖ͬͯΈͨ
   ࿩͢͜ͱ
   

   View Slide

4. ©2018 Wantedly, Inc.
   ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ
   Horizontal Pod Autoscaler (HPA) + CPU
   $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ
   

   View Slide

5. ©2018 Wantedly, Inc.
   ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ
   $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ
   ϝϦοτ
   w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍
   w ݹ͍όʔδϣϯͰ΋࢖͑Δ
   Horizontal Pod Autoscaler (HPA) + CPU
   

   View Slide

6. ©2018 Wantedly, Inc.
   ࠓ·Ͱ࢖͍ͬͯͨΦʔτεέʔϧ
   $16ෛՙ͕ߴ͘ͳΔͱεέʔϧ
   ϝϦοτ
   w ௕ظతͳΞΫηε਺ͷ૿Ճʹڧ͍
   w ݹ͍όʔδϣϯͰ΋࢖͑Δ
   σϝϦοτ
   w ॠؒతͳεύΠΫʹऑ͍
   Horizontal Pod Autoscaler (HPA) + CPU
   

   View Slide

7. ©2018 Wantedly, Inc.
   

   View Slide

8. ©2018 Wantedly, Inc.
   

   View Slide

9. ©2018 Wantedly, Inc.
   

   View Slide

10. ©2018 Wantedly, Inc.
    શવؒʹ߹͍ͬͯͳ͍
    

    View Slide

11. ©2018 Wantedly, Inc.
    ΋ͬͱૣ͘εέʔϧͯ͠΄͍͠
    

    View Slide

12. ©2018 Wantedly, Inc.
    εύΠΫʹڧ͍Φʔτεέʔϧ
    HPA + Custom Metrics
    ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ
    w W͔Β࢖͑Δ
    w ྫ͑͹ʜϦΫΤετ਺ ίωΫγϣϯ਺
    w Ξμϓλ͕ඞཁ
    

    View Slide

13. ©2018 Wantedly, Inc.
    εύΠΫʹڧ͍Φʔτεέʔϧ
    HPA + Custom Metrics
    ΞϓϦέʔγϣϯ͕ܾΊͨج४Ͱεέʔϧ
    w W͔Β࢖͑Δ
    w ྫ͑͹ʜϦΫΤετ਺ ίωΫγϣϯ਺
    w Ξμϓλ͕ඞཁˠ,VCFSOFUFTʹϝτϦΫεΛૹΔͨΊͷαʔόʔ
    

    View Slide

14. ©2018 Wantedly, Inc.
    HPA + Custom Metrics
    k8s-prometheus-adapter
    1SPNFUIFVT͕ऩूͨ͠ϝτϦΫεΛ,VCFSOFUFTʹૹͬͯ͘ΕΔαʔόʔ
    

    View Slide

15. ©2018 Wantedly, Inc.
    ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ

    Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ
    

    View Slide

16. ©2018 Wantedly, Inc.
    ͋ͱ͸ج४ʹ͢Δ஋ΛܾΊͯ

    Prometheus͕ಡΊΔϑΥʔϚοτͰग़ྗ͢Δ͚ͩ
    Ͱ͖Ε͹ΠϯϑϥϨΠϠʔͰղܾ͍ͨ͠
    

    View Slide

17. ©2018 Wantedly, Inc.
    εέʔϧͷ൑ఆج४
    /HJOYͷTUVC@TUBUVT
    location /basic_status {
    stub_status;
    }
    Active connections: 291
    server accepts handled requests
    16630948 16630948 31070465
    Reading: 6 Writing: 179 Waiting: 106
    

    View Slide

18. ©2018 Wantedly, Inc.
    εέʔϧͷ൑ఆج४
    /HJOYͷTUVC@TUBUVT
    location /basic_status {
    stub_status;
    }
    Active connections: 291
    server accepts handled requests
    16630948 16630948 31070465
    Reading: 6 Writing: 179 Waiting: 106
    ଴ͪίωΫγϣϯ਺Λ࢖͏
    

    View Slide

19. ©2018 Wantedly, Inc.
    εέʔϧͷ൑ఆج४
    TUVC@TUBUVTΛFYQPSUFSͷܗࣜʹ͍ͨ͠
    # HELP nginx_connections_current Number of
    connections currently processed by nginx
    # TYPE nginx_connections_current gauge
    nginx_connections_current{state="active"} 2
    nginx_connections_current{state="reading"} 0
    nginx_connections_current{state="waiting"} 1
    nginx_connections_current{state="writing"} 1
    

    View Slide

20. ©2018 Wantedly, Inc.
    ,VCFSOFUFST

    "1*4FSWFS
    LTQSPNFUIFVTBEBQUFS
    1SPNFUIFVT
    )PSJ[POUBM
    1PE
    "VUPTDBMFS
    %FQMPZNFOU
    3FQMJDB4FU
    1PE
    1PE
    1PE
    OHJOY@FYQPSUFS
    /HJOY

    TUVC@TUBUVT
    

    View Slide

21. ©2018 Wantedly, Inc.
    ݁Ռ
    w සൟʹεέʔϧ͢ΔΑ͏ʹͳͬͨ
    w $16ෛՙ͕҆ఆͨ͠
    w Ξϥʔτ͕໐Βͳ͘ͳͬͨ
    

    View Slide

22. ©2018 Wantedly, Inc.
    ߴ଎Φʔτεέʔϧ͕࣮ݱͰ͖ͨ
    

    View Slide

23. ©2018 Wantedly, Inc.
    https://github.com/bgpat/k8s-hpa-example
    

    View Slide