Kaburaya: CPU負荷に応じて継続的に上限値を最適化する動的セマフォ / wsa_3_kaburaya

2018.11.17 Web System Architecture 研究会 (WSA研)  #3
https://websystemarchitecture.hatenablog.jp/entry/2018/10/09/231937

Transcript

1. ࡾ୐༔հ / Pepabo R&D Institute, GMO Pepabo, Inc.
   2018.11.17 Web System Architecture ݚڀձ (WSAݚ) #3
   Kaburaya: CPUෛՙʹԠͯ͡ܧଓతʹ
   ্ݶ஋Λ࠷దԽ͢ΔಈతηϚϑΥ
   

   View Slide

2. ϓϦϯγύϧΤϯδχΞ
   ࡾ୐༔հ!NPOPDISPNFHBOF
   (.0ϖύϘגࣜձࣾϖύϘݚڀॴ
   IUUQTCMPHNPOPDISPNFHBOFDPN
   

   View Slide

3. 1. The Q.
   2. ฒߦॲཧͱνϡʔχϯάʹ͓͚Δ՝୊
   3. CPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ࠷దԽ͢ΔಈతηϚ
   ϑΥ
   4. ධՁ
   5. ·ͱΊ
   3
   ໨࣍
   

   View Slide

4. 1.
   The Q.
   

   View Slide

5. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡

   ൓Ԡత͔ͭܧଓతʹɼฒߦ਺Λ࠷దԽ͢Δ͜ͱ͸Մೳ͔ʁ
   5
   ຊݚڀͷʮ໰͍ʯ
   

   View Slide

6. 2.
   ฒߦॲཧͱνϡʔχϯάʹ͓͚Δ՝୊
   

   View Slide

7. • ϚϧνίΞ࣌୅ʹ͓͍ͯɼ୯ҰͷϚγϯͰॲཧੑೳΛ࠷େԽ͢ΔͨΊॲཧͷฒ
   ߦԽ͕ߦΘΕΔɽ
   • ࠷దͳฒߦ਺ΛٻΊ͍ͨɽ
   • ϊϯϒϩοΩϯάͳॲཧํࣜͰ͸࠷దͳฒߦ਺͸CPUίΞ਺ͱ౳͍͠
   • ϒϩοΩϯά͕Ұ෦෼Ͱ΋ൃੜ͢Δ৔߹͸CPUίΞ਺Ҏ্͕࠷దʹͳΔ৔
   ߹͕ଟ͍
   7
   ฒߦॲཧͱ࠷దͳฒߦ਺
   

   View Slide

8. • ϒϩοΩϯά͕ൃੜ͢Δঢ়گʹ͓͍ͯ͸ɼ࠷దͳฒߦ਺ͷܾఆ͸ґવͱͯ͠։
   ൃऀͷܦݧͱ஍ಓͳνϡʔχϯάʹґଘ
   • nginx, unicornͷworker_processesɼSidekiqͷconcurrencyɼGoroutineͷ
   ىಈ਺ͳͲ
   • ࣮ߦ͞ΕΔॲཧͷಛੑͱϥϯλΠϜ΍εέδϡʔϦϯάͷಛੑΛߟྀͨ͠
   νϡʔχϯά͕ඞཁ
   • ͜ΕΒΛશͯߟྀͨ͠ීวతͳνϡʔχϯάख๏ͷൃݟ͸ࠔ೉
   8
   ࠷దͳฒߦ਺ΛٻΊΔ
   

   View Slide

9. • ద੾ͳࢦඪ͕બఆͰ͖Ε͹୯७ͳࢦඪͰ΋ޮՌ͕͋Δ
   • Ծ૝αʔόͷ༧ଌతΦʔτεέʔϦϯά (*1)Ͱ͸ɼΞϓϦέʔγϣϯॲཧ಺
   ༰Ͱ͸ͳ͘ɼӡ༻ܦݧ͔Βಘͨαʔό୯ҐͷεϧʔϓοτΛࢦඪͱͨ͠ɽ
   • ʮશࣗಈύϥϝʔλνϡʔχϯά͞Μʯ(*2)Ͱ͸ϝτϦΫεΛҰ఺ʹߜΓε
   ϧʔϓοτ͕࠷େԽ͢Δ஋Λ୳ࡧతʹٻΊͨɽ
   9
   ैདྷख๏
   *1. ࡾ୐ ༔հ, দຊ ྄հ, ྗ෢ ݈࣍, ܀ྛ ݈ଠ࿠, ΞΫηεස౓༧ଌʹجͮ͘Ծ૝αʔόͷܭըతΦʔτεέʔϦϯά, FIT 2018 ୈ17ճ৘ใ
   Պֶٕज़ϑΥʔϥϜ, CL-002, Sep 2018.
   *2. https://github.com/mirakui/tuningsan
   

   View Slide

10. • ༧ଌతɼ୳ࡧతͳํࣜ͸WebαʔόʔϓϩηεͷΑ͏ʹ௕ظՔಈ͢Δ৔߹ʹ
    ޮՌత͕ͩɼCLIͷΑ͏ͳ୹ظՔಈͰଈ࣌݁Ռ͕ٻΊΒΕΔ৔߹ʹద༻͕೉͠
    ͍
    10
    ैདྷख๏
    • ࢦඪͷݶఆʹՃ͑ɼ൓Ԡత͔ͭޡ͕ࠩগͳ͘ࢦඪ΁௥ै͢Δख๏͕ٻΊΒΕΔ
    

    View Slide

11. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡

    ൓Ԡత͔ͭܧଓతʹɼฒߦ਺Λ࠷దԽ͢Δ͜ͱ͸Մೳ͔ʁ
    11
    ຊݚڀͷʮ໰͍ʯ
    

    View Slide

12. 3.
    CPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ
    ࠷దԽ͢ΔಈతηϚϑΥ
    

    View Slide

13. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡൓Ԡత͔ͭܧଓ
    తʹɼฒߦ਺Λ࠷దԽ͢Δ
    • 1. ॲཧ΍ϥϯλΠϜΛϒϥοΫϘοΫεͱͯ͠ѻ͍ɼܧଓతʹಈ࡞Λ੍ޚ͢
    Δ࢓૊ΈͰ͋ΔϑΟʔυόοΫ੍ޚΛ༻͍ͯ࠷దͳฒߦ਺ΛٻΊΔ
    • 2. ฒߦ਺ͷ੍ޚʹηϚϑΥΛ༻͍Δ
    • Kaburaya is “ధ໼”. It is japanese arrow with a whistle.
    • Goݴޠ࣮૷ΛOSSͰެ։: https://github.com/monochromegane/kaburaya
    13
    Kaburaya ΞʔΩςΫνϟ
    

    View Slide

14. • ໨ඪ஋ͱग़ྗͷϑΟʔυόοΫͷޡࠩΛ༻͍ͯγεςϜͷೖྗΛܾఆ͠ɼग़ྗ
    Λ໨ඪ஋ʹ͚ۙͮΔ
    • ภࠩΛॲཧ͢Δ੍ޚثɼͦͯ͠ೖྗͱग़ྗʹԿΛ༻͍Δ͔ͷઃܭ͕ॏཁ
    14
    ϑΟʔυόοΫ੍ޚ
    ੍ޚث γεςϜ
    ೖྗ ग़ྗ
    ภࠩ
    ϑΟʔυόοΫ
    ໨ඪ஋
    

    View Slide

15. • ର৅γεςϜ͸ϊϯϒϩοΩϯάॲཧํࣜͷ࠾༻Λલఏͱ͢Δɽ
    • ϒϩοΩϯάॲཧͷεέδϡʔϦϯάʹΑΓͰ͖Δ͚ͩCPUΛ࢖͍੾Δ
    • ར༻ՄೳͳCPUϦιʔε͸Ϛγϯ·ͨ͸ෛՙঢ়گʹΑͬͯมಈ͢Δ
    15
    ੍ޚثͱೖग़ྗͷઃܭ
    • ϫʔΧʔ਺ΛೖྗɼCPU࢖༻཰Λग़ྗͱ͢Δ
    • ໨ඪ஋ͱͳΔCPU࢖༻཰͸ঢ়گʹԠͯ͡มԽ͢Δ
    • ॲཧͷ։࢝ऴྃɼ࣮ߦதʹ࠷খͷϫʔΧʔ਺ͱͳΔΑ͏ௐ੔͢Δ
    

    View Slide

16. 1. ॳظ஋͔ΒϫʔΧʔ਺Λม͑ͳ͍FixController (ൺֱ༻)
    2. CPU࢖༻཰100%Λ໨ඪ஋ͱ͠ɼෆ଍ͨ͠৔߹͸ workerΛ1ͣͭ૿Ճͤ͞Δ SimpleController
    3. CPU࢖༻཰100%Λ໨ඪͱ͠ɼ໨ඪͱͷࠩ෼ͷKഒΛՃ͑Δ PController(ൺྫ੍ޚ)
    4. 3.ͷCPU࢖༻཰ͷ໨ඪ஋Λ90%ͱͨ͠΋ͷ
    5. ௚ۙ3؍ଌ఺ͷฏۉΛ໨ඪ஋ͱͯ͠3؍ଌ͝ͱʹPControllerͷ໨ඪ஋ΛมԽͤ͞ΔDynamicController
    6. ҰఆظؒͷCPU࢖༻཰ͷඪ४ภࠩΛͱͬͯͦΕ͕Ұఆͷ஋ҎԼͩͬͨΒ҆ఆͨ͠ͱΈͳͯ͠ɼworkerΛݮΒ͍ͯ͠
    ͘StabilityController
    7. CPU࢖༻཰Ͱ͸ͳͯ͘CPU࢖༻཰ͷมԽ཰Λݩʹ੍ޚ͢ΔRateController
    8. 5.ͷDynamicControllerΛݩʹఆظͰ͸ͳ͘େ͖ͳมಈ͝ͱʹ໨ඪ஋Λݟ௚͢ํࣜ
    9. 8.ͷDynamicControllerΛݩʹఆৗ࣌ͷෆཁͳworkerΛ࡟ݮ͢Δํࣜ
    10. 9.ͷDynamicControllerΛݩʹมಈͷਫ਼౓ͱ଎౓Λ޲্ͤ͞ΔͨΊʹੵ෼ඍ෼੍ޚΛಋೖ͢Δํࣜ
    16
    ࠷దͳ੍ޚثʹ޲͚ͯ
    

    View Slide

17. • ηϚϑΥͷ্ݶ஋͕ಈతʹมߋՄೳͰ͋Δ͜ͱ
    • ௨ৗͷηϚϑΥͱಉ༷ʹPૢ࡞ʹ͓͍ͯ஋͕ෛʹͳΔ৔߹ʹ࣮ߦ͕ϒϩοΫ͞
    ΕΔ͜ͱ
    • ͜ΕΒͷ஋ͷมߋ͕ΞτϛοΫʹߦΘΕΔ͜ͱ
    17
    ಈతηϚϑΥ
    • Goݴޠ࣮૷ʹ͓͍ͯɼೋͭͷνϟϯωϧΛ૊Έ߹Θ࣮ͤͯݱ
    

    View Slide

18. 4.
    ධՁ
    

    View Slide

19. • ઃܭ੍ͨ͠ޚثͷग़ྗ͢Δىಈ਺͕࠷దͰ͋Δ͜ͱΛݕূ͢Δ
    • ͜͜Ͱ͍͏࠷దͱ͸ॲཧର৅ͷλεΫશମΛɼ࠷௿ݶͷɼͷ΂ىಈ਺Ͱɼ࠷୹
    ͷ࣌ؒͰॲཧͰ͖Δىಈ਺Λࢦ͢ɽ
    • ධՁʹ͸ɼ൚༻ੑͱ࠶ݱՄೳੑΛߴΊΔͨΊɼγϛϡϨʔλʔΛ༻͍ͨ
    19
    ੍ޚثͷධՁ
    

    View Slide

20. • γϛϡϨʔλ͸࣮࣌ؒͰ͸ͳ͘ɼ୯Ґ࣌ؒΛεςοϓͱΈͳ͢
    • Job͸WorkloadΛ࣋ͪɼεςοϓ͝ͱͷCPUར༻཰Λఆٛ͢Δ
    • Workload͕0ͷεςοϓ͸ϒϩοΩϯάॲཧΛදݱ͢Δ
    • γϛϡϨʔλ͸εςοϓ͝ͱʹ೚ҙͷ਺ͷδϣϒΛ౤ೖ͢Δ
    • γϛϡϨʔλ͸εςοϓ͝ͱʹىಈՄೳͳϫʔΧʔ਺Λ੍ޚث͔Βऔಘ͢Δ
    • ϫʔΧʔ͸γϛϡϨʔλͰར༻ՄೳͳCPUར༻཰·Ͱδϣϒͷ֘౰εςοϓͷWorkloadΛফඅ
    ͢Δ
    • ফඅͰ͖ͳ͔ͬͨWorkload͸࣍ճͷεςοϓʹճ͞ΕΔ
    • Workload͕0ͷεςοϓ͸CPUࢿݯΛফඅ͠ͳ͍ͨΊແ৚݅ʹεςοϓΛਐΊΔ
    20
    γϛϡϨʔλʔཁ݅
    

    View Slide

21. • ॳظ஋͔ΒϫʔΧʔ਺Λม͑ͳ͍
    FixControllerɽ
    • worker:7 ͙Β͍͕࠷খ͔ͭ࠷୹ͷ
    ༷ࢠ
    γϛϡϨʔγϣϯ1
    21
    

    View Slide

22. • CPU࢖༻཰100%Λ໨ඪ஋ͱ͠ɼෆ
    ଍ͨ͠৔߹͸ workerΛ1ͣͭ૿Ճ͞
    ͤΔ SimpleController
    • ૿Ճ͕௥͍͍͍ͭͯͳ͍ͷͰࡹ͖͖
    Δͷʹ͕͔͔࣌ؒͬͯ͠·ͬͨɽ
    γϛϡϨʔγϣϯ2
    22
    

    View Slide

23. • CPU࢖༻཰90%Λ໨ඪͱ͠ɼ໨ඪͱ
    ͷࠩ෼ͷKഒΛՃ͑Δ
    PController(ൺྫ੍ޚ)
    • K͸0.1ͱͨ͠ɽ
    • CPU࢖༻཰ͷ໨ඪ஋͕ݻఆͷͨΊɼ
    ࡹ͖͖ͬͨޙʹ૿Ճ͠ଓ͚ͯ͠·͏
    γϛϡϨʔγϣϯ4
    23
    

    View Slide

24. • CPU࢖༻཰ͷ໨ඪ஋͕ݻఆͷͨΊɼ
    ࡹ͖͖ͬͨޙʹ૿Ճ͠ଓ͚ͯ͠·͏
    ໰୊ʹରԠ͢ΔͨΊCPU࢖༻཰ͷ໨
    ඪ஋ࣗମΛมಈͤ͞Δ
    • ௚ۙ3؍ଌ఺ͷฏۉΛ໨ඪ஋ͱͯ͠3
    ؍ଌ͝ͱʹPControllerͷ໨ඪ஋Λ
    มԽͤ͞Δ࢓૊Έ
    (DynamicController)
    γϛϡϨʔγϣϯ5
    24
    

    View Slide

25. • ఆظతͰ͸ͳ͘CPUͷେ͖ͳมಈ͝
    ͱʹ໨ඪ஋Λݟ௚͢ɽ
    • ໨ඪ஋Λ੺Ͱϓϩοτ
    • େ͖ͳ૿ݮΛݩʹΞδϟετ͍ͯ͠
    ͘࢓૊Έʹͳͬͨ͜ͱͰɼCPU҆ఆ
    ࣌ʹੵۃతʹϫʔΧʔΛ࡟ݮ͢Δ͜
    ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨ
    γϛϡϨʔγϣϯ9
    25
    

    View Slide

26. • සൟʹมߋ͞ΕΔ໨ඪ஋ʹߴ଎͔ͭ
    ޡ͕ࠩগͳ͘దԠ͍ͯͨ͘͠Ίݹయ
    ੍ޚͷख๏Ͱ͋ΔPID੍ޚΛ༻͍
    ͨɽ
    • ݱࡏͷௐ੔ͷ݁ՌɼPID੍ޚͷύϥ
    ϝλͱͯ͠Kp: 0.1, Ki: 0.5, Kd: 0.5
    ͕σϑΥϧτ஋ͱͯ͠ར༻Ͱ͖ͦ͏
    γϛϡϨʔγϣϯ11
    26
    

    View Slide

27. 5.
    ·ͱΊ
    

    View Slide

28. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡൓Ԡత͔ͭܧଓ
    తʹɼฒߦ਺Λ࠷దԽ͢Δख๏ͱͯ͠ɼCPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ࠷
    దԽ͢ΔಈతηϚϑΥΛఏҊ
    • γϛϡϨʔγϣϯ؀ڥʹ͓͍ͯɼCPU࢖༻཰ͷ໨ඪ஋Λෛՙ৘ใʹԠͯ͡มಈ
    ͤ͞Δ੍ޚثͷ༗ޮੑΛ֬ೝ
    • ຊํࣜ͸ฒߦ਺ΛٻΊͳ͚Ε͹ͳΒͳ͍༷ʑͳ৔໘ʹద༻Մೳ
    • ࠓޙ͸࣮؀ڥͰͷධՁΛਐΊΔ͜ͱͱPID੍ޚͷύϥϝλઃܭ͕ඞཁ͕ͩ
    ϑΟʔυόοΫ੍ޚ෼໺ͷϊ΢ϋ΢Λ༗ޮ׆༻ͯ͠ղܾ͍ͨ͠ɽ
    • దਖ਼ͳϫʔΧʔ਺ΛධՁ͢ΔείΞϦϯάͷઃܭ΋ඞཁ
    28
    ·ͱΊ
    

    View Slide

29. View Slide