$30 off During Our Annual Pro Sale. View Details »

Kaburaya: CPU負荷に応じて継続的に上限値を最適化する動的セマフォ / wsa_3_kaburaya

Kaburaya: CPU負荷に応じて継続的に上限値を最適化する動的セマフォ / wsa_3_kaburaya

2018.11.17 Web System Architecture 研究会 (WSA研)  #3
https://websystemarchitecture.hatenablog.jp/entry/2018/10/09/231937

monochromegane

November 17, 2018
Tweet

More Decks by monochromegane

Other Decks in Programming

Transcript

  1. ࡾ୐༔հ / Pepabo R&D Institute, GMO Pepabo, Inc.
    2018.11.17 Web System Architecture ݚڀձ (WSAݚ) #3
    Kaburaya: CPUෛՙʹԠͯ͡ܧଓతʹ
    ্ݶ஋Λ࠷దԽ͢ΔಈతηϚϑΥ

    View Slide

  2. ϓϦϯγύϧΤϯδχΞ
    ࡾ୐༔հ!NPOPDISPNFHBOF
    (.0ϖύϘגࣜձࣾϖύϘݚڀॴ
    IUUQTCMPHNPOPDISPNFHBOFDPN

    View Slide

  3. 1. The Q.
    2. ฒߦॲཧͱνϡʔχϯάʹ͓͚Δ՝୊
    3. CPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ࠷దԽ͢ΔಈతηϚ
    ϑΥ
    4. ධՁ
    5. ·ͱΊ
    3
    ໨࣍

    View Slide

  4. 1.
    The Q.

    View Slide

  5. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡

    ൓Ԡత͔ͭܧଓతʹɼฒߦ਺Λ࠷దԽ͢Δ͜ͱ͸Մೳ͔ʁ
    5
    ຊݚڀͷʮ໰͍ʯ

    View Slide

  6. 2.
    ฒߦॲཧͱνϡʔχϯάʹ͓͚Δ՝୊

    View Slide

  7. • ϚϧνίΞ࣌୅ʹ͓͍ͯɼ୯ҰͷϚγϯͰॲཧੑೳΛ࠷େԽ͢ΔͨΊॲཧͷฒ
    ߦԽ͕ߦΘΕΔɽ
    • ࠷దͳฒߦ਺ΛٻΊ͍ͨɽ
    • ϊϯϒϩοΩϯάͳॲཧํࣜͰ͸࠷దͳฒߦ਺͸CPUίΞ਺ͱ౳͍͠
    • ϒϩοΩϯά͕Ұ෦෼Ͱ΋ൃੜ͢Δ৔߹͸CPUίΞ਺Ҏ্͕࠷దʹͳΔ৔
    ߹͕ଟ͍
    7
    ฒߦॲཧͱ࠷దͳฒߦ਺

    View Slide

  8. • ϒϩοΩϯά͕ൃੜ͢Δঢ়گʹ͓͍ͯ͸ɼ࠷దͳฒߦ਺ͷܾఆ͸ґવͱͯ͠։
    ൃऀͷܦݧͱ஍ಓͳνϡʔχϯάʹґଘ
    • nginx, unicornͷworker_processesɼSidekiqͷconcurrencyɼGoroutineͷ
    ىಈ਺ͳͲ
    • ࣮ߦ͞ΕΔॲཧͷಛੑͱϥϯλΠϜ΍εέδϡʔϦϯάͷಛੑΛߟྀͨ͠
    νϡʔχϯά͕ඞཁ
    • ͜ΕΒΛશͯߟྀͨ͠ීวతͳνϡʔχϯάख๏ͷൃݟ͸ࠔ೉
    8
    ࠷దͳฒߦ਺ΛٻΊΔ

    View Slide

  9. • ద੾ͳࢦඪ͕બఆͰ͖Ε͹୯७ͳࢦඪͰ΋ޮՌ͕͋Δ
    • Ծ૝αʔόͷ༧ଌతΦʔτεέʔϦϯά (*1)Ͱ͸ɼΞϓϦέʔγϣϯॲཧ಺
    ༰Ͱ͸ͳ͘ɼӡ༻ܦݧ͔Βಘͨαʔό୯ҐͷεϧʔϓοτΛࢦඪͱͨ͠ɽ
    • ʮશࣗಈύϥϝʔλνϡʔχϯά͞Μʯ(*2)Ͱ͸ϝτϦΫεΛҰ఺ʹߜΓε
    ϧʔϓοτ͕࠷େԽ͢Δ஋Λ୳ࡧతʹٻΊͨɽ
    9
    ैདྷख๏
    *1. ࡾ୐ ༔հ, দຊ ྄հ, ྗ෢ ݈࣍, ܀ྛ ݈ଠ࿠, ΞΫηεස౓༧ଌʹجͮ͘Ծ૝αʔόͷܭըతΦʔτεέʔϦϯά, FIT 2018 ୈ17ճ৘ใ
    Պֶٕज़ϑΥʔϥϜ, CL-002, Sep 2018.
    *2. https://github.com/mirakui/tuningsan

    View Slide

  10. • ༧ଌతɼ୳ࡧతͳํࣜ͸WebαʔόʔϓϩηεͷΑ͏ʹ௕ظՔಈ͢Δ৔߹ʹ
    ޮՌత͕ͩɼCLIͷΑ͏ͳ୹ظՔಈͰଈ࣌݁Ռ͕ٻΊΒΕΔ৔߹ʹద༻͕೉͠
    ͍
    10
    ैདྷख๏
    • ࢦඪͷݶఆʹՃ͑ɼ൓Ԡత͔ͭޡ͕ࠩগͳ͘ࢦඪ΁௥ै͢Δख๏͕ٻΊΒΕΔ

    View Slide

  11. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡

    ൓Ԡత͔ͭܧଓతʹɼฒߦ਺Λ࠷దԽ͢Δ͜ͱ͸Մೳ͔ʁ
    11
    ຊݚڀͷʮ໰͍ʯ

    View Slide

  12. 3.
    CPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ
    ࠷దԽ͢ΔಈతηϚϑΥ

    View Slide

  13. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡൓Ԡత͔ͭܧଓ
    తʹɼฒߦ਺Λ࠷దԽ͢Δ
    • 1. ॲཧ΍ϥϯλΠϜΛϒϥοΫϘοΫεͱͯ͠ѻ͍ɼܧଓతʹಈ࡞Λ੍ޚ͢
    Δ࢓૊ΈͰ͋ΔϑΟʔυόοΫ੍ޚΛ༻͍ͯ࠷దͳฒߦ਺ΛٻΊΔ
    • 2. ฒߦ਺ͷ੍ޚʹηϚϑΥΛ༻͍Δ
    • Kaburaya is “ధ໼”. It is japanese arrow with a whistle.
    • Goݴޠ࣮૷ΛOSSͰެ։: https://github.com/monochromegane/kaburaya
    13
    Kaburaya ΞʔΩςΫνϟ

    View Slide

  14. • ໨ඪ஋ͱग़ྗͷϑΟʔυόοΫͷޡࠩΛ༻͍ͯγεςϜͷೖྗΛܾఆ͠ɼग़ྗ
    Λ໨ඪ஋ʹ͚ۙͮΔ
    • ภࠩΛॲཧ͢Δ੍ޚثɼͦͯ͠ೖྗͱग़ྗʹԿΛ༻͍Δ͔ͷઃܭ͕ॏཁ
    14
    ϑΟʔυόοΫ੍ޚ
    ੍ޚث γεςϜ
    ೖྗ ग़ྗ
    ภࠩ
    ϑΟʔυόοΫ
    ໨ඪ஋

    View Slide

  15. • ର৅γεςϜ͸ϊϯϒϩοΩϯάॲཧํࣜͷ࠾༻Λલఏͱ͢Δɽ
    • ϒϩοΩϯάॲཧͷεέδϡʔϦϯάʹΑΓͰ͖Δ͚ͩCPUΛ࢖͍੾Δ
    • ར༻ՄೳͳCPUϦιʔε͸Ϛγϯ·ͨ͸ෛՙঢ়گʹΑͬͯมಈ͢Δ
    15
    ੍ޚثͱೖग़ྗͷઃܭ
    • ϫʔΧʔ਺ΛೖྗɼCPU࢖༻཰Λग़ྗͱ͢Δ
    • ໨ඪ஋ͱͳΔCPU࢖༻཰͸ঢ়گʹԠͯ͡มԽ͢Δ
    • ॲཧͷ։࢝ऴྃɼ࣮ߦதʹ࠷খͷϫʔΧʔ਺ͱͳΔΑ͏ௐ੔͢Δ

    View Slide

  16. 1. ॳظ஋͔ΒϫʔΧʔ਺Λม͑ͳ͍FixController (ൺֱ༻)
    2. CPU࢖༻཰100%Λ໨ඪ஋ͱ͠ɼෆ଍ͨ͠৔߹͸ workerΛ1ͣͭ૿Ճͤ͞Δ SimpleController
    3. CPU࢖༻཰100%Λ໨ඪͱ͠ɼ໨ඪͱͷࠩ෼ͷKഒΛՃ͑Δ PController(ൺྫ੍ޚ)
    4. 3.ͷCPU࢖༻཰ͷ໨ඪ஋Λ90%ͱͨ͠΋ͷ
    5. ௚ۙ3؍ଌ఺ͷฏۉΛ໨ඪ஋ͱͯ͠3؍ଌ͝ͱʹPControllerͷ໨ඪ஋ΛมԽͤ͞ΔDynamicController
    6. ҰఆظؒͷCPU࢖༻཰ͷඪ४ภࠩΛͱͬͯͦΕ͕Ұఆͷ஋ҎԼͩͬͨΒ҆ఆͨ͠ͱΈͳͯ͠ɼworkerΛݮΒ͍ͯ͠
    ͘StabilityController
    7. CPU࢖༻཰Ͱ͸ͳͯ͘CPU࢖༻཰ͷมԽ཰Λݩʹ੍ޚ͢ΔRateController
    8. 5.ͷDynamicControllerΛݩʹఆظͰ͸ͳ͘େ͖ͳมಈ͝ͱʹ໨ඪ஋Λݟ௚͢ํࣜ
    9. 8.ͷDynamicControllerΛݩʹఆৗ࣌ͷෆཁͳworkerΛ࡟ݮ͢Δํࣜ
    10. 9.ͷDynamicControllerΛݩʹมಈͷਫ਼౓ͱ଎౓Λ޲্ͤ͞ΔͨΊʹੵ෼ඍ෼੍ޚΛಋೖ͢Δํࣜ
    16
    ࠷దͳ੍ޚثʹ޲͚ͯ

    View Slide

  17. • ηϚϑΥͷ্ݶ஋͕ಈతʹมߋՄೳͰ͋Δ͜ͱ
    • ௨ৗͷηϚϑΥͱಉ༷ʹPૢ࡞ʹ͓͍ͯ஋͕ෛʹͳΔ৔߹ʹ࣮ߦ͕ϒϩοΫ͞
    ΕΔ͜ͱ
    • ͜ΕΒͷ஋ͷมߋ͕ΞτϛοΫʹߦΘΕΔ͜ͱ
    17
    ಈతηϚϑΥ
    • Goݴޠ࣮૷ʹ͓͍ͯɼೋͭͷνϟϯωϧΛ૊Έ߹Θ࣮ͤͯݱ

    View Slide

  18. 4.
    ධՁ

    View Slide

  19. • ઃܭ੍ͨ͠ޚثͷग़ྗ͢Δىಈ਺͕࠷దͰ͋Δ͜ͱΛݕূ͢Δ
    • ͜͜Ͱ͍͏࠷దͱ͸ॲཧର৅ͷλεΫશମΛɼ࠷௿ݶͷɼͷ΂ىಈ਺Ͱɼ࠷୹
    ͷ࣌ؒͰॲཧͰ͖Δىಈ਺Λࢦ͢ɽ
    • ධՁʹ͸ɼ൚༻ੑͱ࠶ݱՄೳੑΛߴΊΔͨΊɼγϛϡϨʔλʔΛ༻͍ͨ
    19
    ੍ޚثͷධՁ

    View Slide

  20. • γϛϡϨʔλ͸࣮࣌ؒͰ͸ͳ͘ɼ୯Ґ࣌ؒΛεςοϓͱΈͳ͢
    • Job͸WorkloadΛ࣋ͪɼεςοϓ͝ͱͷCPUར༻཰Λఆٛ͢Δ
    • Workload͕0ͷεςοϓ͸ϒϩοΩϯάॲཧΛදݱ͢Δ
    • γϛϡϨʔλ͸εςοϓ͝ͱʹ೚ҙͷ਺ͷδϣϒΛ౤ೖ͢Δ
    • γϛϡϨʔλ͸εςοϓ͝ͱʹىಈՄೳͳϫʔΧʔ਺Λ੍ޚث͔Βऔಘ͢Δ
    • ϫʔΧʔ͸γϛϡϨʔλͰར༻ՄೳͳCPUར༻཰·Ͱδϣϒͷ֘౰εςοϓͷWorkloadΛফඅ
    ͢Δ
    • ফඅͰ͖ͳ͔ͬͨWorkload͸࣍ճͷεςοϓʹճ͞ΕΔ
    • Workload͕0ͷεςοϓ͸CPUࢿݯΛফඅ͠ͳ͍ͨΊແ৚݅ʹεςοϓΛਐΊΔ
    20
    γϛϡϨʔλʔཁ݅

    View Slide

  21. • ॳظ஋͔ΒϫʔΧʔ਺Λม͑ͳ͍
    FixControllerɽ
    • worker:7 ͙Β͍͕࠷খ͔ͭ࠷୹ͷ
    ༷ࢠ
    γϛϡϨʔγϣϯ1
    21

    View Slide

  22. • CPU࢖༻཰100%Λ໨ඪ஋ͱ͠ɼෆ
    ଍ͨ͠৔߹͸ workerΛ1ͣͭ૿Ճ͞
    ͤΔ SimpleController
    • ૿Ճ͕௥͍͍͍ͭͯͳ͍ͷͰࡹ͖͖
    Δͷʹ͕͔͔࣌ؒͬͯ͠·ͬͨɽ
    γϛϡϨʔγϣϯ2
    22

    View Slide

  23. • CPU࢖༻཰90%Λ໨ඪͱ͠ɼ໨ඪͱ
    ͷࠩ෼ͷKഒΛՃ͑Δ
    PController(ൺྫ੍ޚ)
    • K͸0.1ͱͨ͠ɽ
    • CPU࢖༻཰ͷ໨ඪ஋͕ݻఆͷͨΊɼ
    ࡹ͖͖ͬͨޙʹ૿Ճ͠ଓ͚ͯ͠·͏
    γϛϡϨʔγϣϯ4
    23

    View Slide

  24. • CPU࢖༻཰ͷ໨ඪ஋͕ݻఆͷͨΊɼ
    ࡹ͖͖ͬͨޙʹ૿Ճ͠ଓ͚ͯ͠·͏
    ໰୊ʹରԠ͢ΔͨΊCPU࢖༻཰ͷ໨
    ඪ஋ࣗମΛมಈͤ͞Δ
    • ௚ۙ3؍ଌ఺ͷฏۉΛ໨ඪ஋ͱͯ͠3
    ؍ଌ͝ͱʹPControllerͷ໨ඪ஋Λ
    มԽͤ͞Δ࢓૊Έ
    (DynamicController)
    γϛϡϨʔγϣϯ5
    24

    View Slide

  25. • ఆظతͰ͸ͳ͘CPUͷେ͖ͳมಈ͝
    ͱʹ໨ඪ஋Λݟ௚͢ɽ
    • ໨ඪ஋Λ੺Ͱϓϩοτ
    • େ͖ͳ૿ݮΛݩʹΞδϟετ͍ͯ͠
    ͘࢓૊Έʹͳͬͨ͜ͱͰɼCPU҆ఆ
    ࣌ʹੵۃతʹϫʔΧʔΛ࡟ݮ͢Δ͜
    ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨ
    γϛϡϨʔγϣϯ9
    25

    View Slide

  26. • සൟʹมߋ͞ΕΔ໨ඪ஋ʹߴ଎͔ͭ
    ޡ͕ࠩগͳ͘దԠ͍ͯͨ͘͠Ίݹయ
    ੍ޚͷख๏Ͱ͋ΔPID੍ޚΛ༻͍
    ͨɽ
    • ݱࡏͷௐ੔ͷ݁ՌɼPID੍ޚͷύϥ
    ϝλͱͯ͠Kp: 0.1, Ki: 0.5, Kd: 0.5
    ͕σϑΥϧτ஋ͱͯ͠ར༻Ͱ͖ͦ͏
    γϛϡϨʔγϣϯ11
    26

    View Slide

  27. 5.
    ·ͱΊ

    View Slide

  28. • ॲཧ΍ϥϯλΠϜͷಛੑʹґଘͤͣʹɼϚγϯͷෛՙʹԠͯ͡൓Ԡత͔ͭܧଓ
    తʹɼฒߦ਺Λ࠷దԽ͢Δख๏ͱͯ͠ɼCPUෛՙʹԠͯ͡ܧଓతʹ্ݶ஋Λ࠷
    దԽ͢ΔಈతηϚϑΥΛఏҊ
    • γϛϡϨʔγϣϯ؀ڥʹ͓͍ͯɼCPU࢖༻཰ͷ໨ඪ஋Λෛՙ৘ใʹԠͯ͡มಈ
    ͤ͞Δ੍ޚثͷ༗ޮੑΛ֬ೝ
    • ຊํࣜ͸ฒߦ਺ΛٻΊͳ͚Ε͹ͳΒͳ͍༷ʑͳ৔໘ʹద༻Մೳ
    • ࠓޙ͸࣮؀ڥͰͷධՁΛਐΊΔ͜ͱͱPID੍ޚͷύϥϝλઃܭ͕ඞཁ͕ͩ
    ϑΟʔυόοΫ੍ޚ෼໺ͷϊ΢ϋ΢Λ༗ޮ׆༻ͯ͠ղܾ͍ͨ͠ɽ
    • దਖ਼ͳϫʔΧʔ਺ΛධՁ͢ΔείΞϦϯάͷઃܭ΋ඞཁ
    28
    ·ͱΊ

    View Slide

  29. View Slide