Upgrade to Pro — share decks privately, control downloads, hide ads and more …

分散アプリケーションの高信頼化のための
運用技術に関する研究 / A Study on Operation Technology for High Reliability of Distributed Applications

分散アプリケーションの高信頼化のための
運用技術に関する研究 / A Study on Operation Technology for High Reliability of Distributed Applications

京都大学大学院 情報学研究科 知能情報学専攻 
中間発表(15min)。

Yuuki Tsubouchi (yuuk1)

August 05, 2021
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Research

Transcript

  1. ෼ࢄΞϓϦέʔγϣϯͷߴ৴པԽͷͨΊͷ

    ӡ༻ٕज़ʹؔ͢Δݚڀ
    ɾΫϥ΢υΛ࢖༻ͨ͠ΞϓϦέʔγϣϯͷ৴པੑΛ޲্ͤ͞Δ
    ΞʔΩςΫνϟΛݚڀ͍ͯ͠Δɻ


    ɾಛʹɺγεςϜͷӡ༻ঢ়گΛࣔ͢σʔλΛޮ཰తʹऩू͢Δख
    ๏ʹߩݙͨ͠ɻ


    ɾݱࡏ͸ɺऩू͞Εͨσʔλ͔ΒɺγεςϜʹൃੜ͢Δҟৗͷݪ
    ҼΛਝ଎ʹಛఆ͢ΔͨΊͷσʔλ෼ੳख๏Λݚڀ͍ͯ͠Δɻ
    ௶಺ ༎थ

    ژ౎େֶେֶӃ ৘ใֶݚڀՊ ஌ೳ৘ใֶઐ߈ ത࢜ޙظ՝ఔ 2೥

    தؒൃද 2021೥8݄5೔

    View Slide

  2. 2
    1. Ϋϥ΢υ্ͷ෼ࢄΞϓϦέʔγϣϯ


    Ϋϥ΢υ্ͷ෼ࢄΞϓϦέʔγϣϯͷෳࡶԽ


    ৴པੑͷ՝୊: ӡ༻σʔλྔͷ૿Ճ

    2. ݚڀख๏


    ӡ༻σʔλͷऩूͷߴޮ཰Խ


    • [ݚڀ੒Ռ1] ଟ਺ͷ࣌ܥྻσʔλΛૠೖɾอଘ͢ΔͨΊͷσʔλϕʔε


    • [ݚڀ੒Ռ2] ωοτϫʔΫ௨৴ͷґଘؔ܎ͷ௿Φʔόϔουऩू


    ӡ༻σʔλͷղੳ


    • [ݚڀ੒Ռ3] ࣌ܥྻσʔλͷத͔ΒݪҼ਍அʹ༗༻ͳܥྻΛநग़


    3. ·ͱΊ/ݚڀ࣮੷
    ൃද಺༰ͷߏ੒

    View Slide

  3. 3
    Ϋϥ΢υ্ͷ෼ࢄΞϓϦέʔγϣϯͷෳࡶԽ
    ɾ


    ɾ


    ɾ
    ΞΫηε૿Ճ
    εέʔϧΞ΢τʹΑΔ


    ϗετ਺ͷ૿Ճ
    ػೳͷ૿Ճ
    ΑΓେن໛ͳ

    ෼ࢄΞʔΩςΫνϟ
    ϛυϧ΢ΣΞͷ૿Ճ
    RDBαʔό
    Ωϟογϡ

    αʔό
    ݕࡧαʔό
    Webαʔό
    ωοτϫʔΫ

    αʔϏε
    TCP/UDP
    ɾ


    ɾ


    ɾ
    ϝσΟΞɺECαΠτɺSNSɺIoTͳͲΛߏ੒͢Δ෼ࢄΞϓϦέʔγϣϯ

    View Slide

  4. 4
    ɾίϯϙʔωϯτ਺ͷ૿େʹΑΓɺӡ༻σʔλྔ͕૿Ճ


    ɾσʔλऩूͷͨΊͷܭࢉػͷෛՙ͕૿Ճ


    ɾγεςϜ؅ཧऀͷೝ஌ෛՙ͕૿Ճ
    ෳࡶԽͨ݁͠Ռͷ৴པੑͷ௿Լ
    ӡ༻σʔλͷ

    ऩू
    ɾෛՙঢ়گΛࣔ࣌͢ܥྻσʔλ


    ɾωοτϫʔΫ௨৴ͷґଘؔ܎


    ɾςΩετϩά
    ৴པੑͷ௿Լ΁
    ো֐෮چ࣌ؒͷ૿େ

    View Slide

  5. 5
    ɾӡ༻σʔλͷऩूΛߴޮ཰Խ͢Δ ܭࢉػͷෛՙΛ௿ݮ


    [ݚڀ੒Ռ1] ࣌ܥྻσʔλΛߴޮ཰ʹૠೖɾอଘ͢Δσʔλϕʔε


    [ݚڀ੒Ռ2] ωοτϫʔΫ௨৴ͷґଘؔ܎ͷ௿Φʔόϔουऩू
    ໨త: ӡ༻σʔλͷऩूͱղੳͷߴ౓ԽʹΑΓ৴པੑΛ޲্
    ͍ͣΕ΋࣮؀ڥ΁ͷద༻༰қੑΛߟྀ͢Δ
    ɾӡ༻σʔλΛղੳ͠ɺো֐ରԠʹ༗༻ͳಎ࡯ΛಘΔ ೝ஌ෛՙΛ௿ݮ


    [ݚڀ੒Ռ3] ࣌ܥྻσʔλͷத͔ΒݪҼ਍அʹ༗༻ͳܥྻΛநग़
    ʢ↪࿦จࢽ౤ߘதʣ
    ʢ↪࿦จࢽ࠾࿥ࡁʣ
    ʢ↪ࠃࡍձٞ౤ߘ४උதʣ

    View Slide

  6. 6
    ɾҰൠʹɺσΟεΫϕʔεDB͸ɺࡧҾߏ଄ʹฏߧ໦Λ࢖༻͢Δ


    ɾ՝୊ɿܥྻ਺(n)ͷ૿Ճʹରͯ͠ɺܭࢉྔ͕O(log n)Ͱ૿Ճ͢Δ


    ɾ؅ཧͷෛ୲Λܰݮ͢ΔͨΊʹɺطଘͷDB্ʹߏ੒͍ͨ͠
    [ݚڀ੒Ռ1] ࣌ܥྻσʔλͷऩूͷߴޮ཰Խ
    ௶಺༎थ΄͔, HeteroTSDB: ҟछ෼ࢄKVSؒͷࣗಈ֊૚ԽʹΑΔߴੑೳͳ࣌ܥྻσʔλϕʔε, ৘ใॲཧֶձ࿦จࢽ, Vol.62, No.3, pp.818-828, 2021೥3݄.
    ϝϞϦ

    ϕʔεDB
    σΟεΫ

    ϕʔεDB
    ϋογϡද O(k) ฏߧ໦ O(log n)
    σʔλ఺ૠೖ
    dݸͷσʔλ఺Λ஝ੵޙʹ

    ϑϥογϡॻ͖ࠐΈ
    ղܾɿϝϞϦϕʔεDBͱσΟεΫϕʔεDBͷ֊૚ԽΛఏҊ
    M (insertions/s) M / d (insertions/s)

    View Slide

  7. 7
    [ݚڀ੒Ռ1] ܥྻ਺ͷ૿Ճʹର͢ΔૠೖεϧʔϓοτมԽ
    ɾσΟεΫϕʔεDBͷΈͱൺֱ͠ɺ
    ࠷େͰ3.96ഒͷεϧʔϓοτ


    ɾεϧʔϓοτͷ௿Լ཰΋վળͨ͠
    0
    20
    40
    60
    80
    100
    100 1K 10K 100k 1M
    0
    20
    40
    60
    Insertion throughput (kilo datapoints / sec)
    Throughput decrease rate (%)
    The number of series
    HeteroTSDB (Proposed)
    KairosDB
    HeteroTSDB (Proposed)
    KairosDB
    ϝϞϦDB → σΟεΫDBͷҠಈεϧʔ
    ϓοτͱϝϞϦDB΁ͷεϧʔϓοτͱ
    ಉఔ౓
    0
    20
    40
    60
    80
    100
    0 300 600 900 1200 1500 1800
    0
    500
    1000
    1500
    2000
    Insertion throughput (kilo datapoints / sec)
    Memory used size (MB)
    Elapsed time (sec)
    Flushed datapoints (/sec)
    Memory used size (MB)
    ௶಺༎थ΄͔, HeteroTSDB: ҟछ෼ࢄKVSؒͷࣗಈ֊૚ԽʹΑΔߴੑೳͳ࣌ܥྻσʔλϕʔε, ৘ใॲཧֶձ࿦จࢽ, Vol.62, No.3, pp.818-828, 2021೥3݄.

    View Slide

  8. (౤ߘத) Y. Tsubouchi, et al., Low Overhead TCP/UDP Socket-based Tracing for Discovering Network Services Dependencies, Journal of Information Processing 2022.
    [ݚڀ੒Ռ2] ωοτϫʔΫ௨৴ͷґଘؔ܎औಘͷޮ཰Խ
    .


    .


    .
    Kernel
    User
    Service
    Socket
    Tracing

    Process
    … Event
    Event
    Event
    ετϦʔϛϯά๏(Weave Scope)
    ϑϩʔू໿๏ ([Datadog], [SAC 20])
    ϑϩʔूଋ๏ʢఏҊʣ
    .


    .


    .
    Kernel
    Service
    Socket
    Tracing

    Process
    .


    .


    .
    Event
    Flow
    Event
    Event
    Event


    .


    .


    . .


    .


    .
    User
    Service
    Socket
    Tracing

    Process
    .


    .


    .
    ✗ ΧʔωϧˠϢʔβۭؒؒ
    ͷΠϕϯτͷίϐʔίετ
    ✗ TCP઀ଓϨʔτ͕૿Ճ͢Δ
    ͱɺίϐʔίετ͕૿Ճ ෳ਺ͷϑϩʔΛूଋ
    ϑϩʔ= ྆୺ͷΞυϨεͱϙʔτͷ
    ૊ʢλϓϧʣ͕ಉҰͷ௨৴୯Ґ
    Event
    Event …

    Event
    Event
    .


    .


    .
    Event
    Event …
    Event
    Event
    .


    .


    .
    Ұൠతʹ௨৴͸OSΧʔωϧͷTCP/UDPΛ࢖༻͢Δ͜ͱ
    ʹண໨
    8

    View Slide

  9. 9
    ɾఏҊख๏͸ɺCPUར༻཰͸2.2%ҎԼɻ


    ɾϑϩʔ਺ͷ૿େʹରͯ͠ɺϑϩʔूଋʹΑΓɺ

    CPUར༻཰Λ௿͘ҡ͍࣋ͯ͠Δ
    [ݚڀ੒Ռ2] ϑϩʔ਺ͷ૿େʹର͢ΔCPU࢖༻ྔͷมԽ
    ఏҊख๏
    (౤ߘத) Y. Tsubouchi, et al., Low Overhead TCP/UDP Socket-based Tracing for Discovering Network Services Dependencies, Journal of Information Processing 2022.
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    TCP round trips / sec (x103)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)

    View Slide

  10. εςοϓ1: ظؒதͷมಈͷ౓߹͍͕খ͍͞ܥྻΛআڈ


    εςοϓ2: άϥϑͷܗঢ়͕ྨࣅ͢Δ΋ͷ͔Β୅දܥྻΛநग़
    10
    ɾ՝୊: ίϯϙʔωϯτ͝ͱʹɺಛ௃ྔͱͯ͠ܥྻͷछผʢCPU࢖༻ɺϝϞϦ࢖
    ༻ɺωοτϫʔΫଳҬɺ…ʣΛࢦఆ͢Δඞཁ͋Γ


    ɾղܾ: ؔ࿈͢ΔશܥྻΛೖྗͱͯ͠༩͑ɺݪҼ਍அʹ༗༻ͳܥྻΛߴ଎ʹநग़
    [ݚڀ੒Ռ3] ࣌ܥྻσʔλ͔ΒݪҼ਍அʹ༗༻ͳܥྻΛநग़
    ҟৗ

    ݕ஌
    YES
    ࣌ܥྻσʔλͷऩू
    ݪҼ਍அ

    ҼՌάϥϑΛ

    ߏங
    [AutoMAP, Microscope,…]
    ఏҊ
    ࣌ܥྻσʔλͷ

    ࣍ݩ࡟ݮ
    ௶಺༎थ΄͔, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝ଎ͳ਍அʹ޲͍ͨ࣌ܥྻσʔλͷ࣍ݩ࡟ݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ࿦จू, 2020೥.

    View Slide

  11. 11
    [ݚڀ੒Ռ3] ܥྻ࡟ݮੑೳͱߴ଎ੑ
    ɾCPUίΞ਺4ɺܥྻ਺100k
    εςοϓ1

    ܥྻ୯Ґͷআڈ
    εςοϓ2


    ྨࣅܥྻͷू໿
    ߹ܭ

    ࣮ߦ࣌ؒ
    54.41 (sec) 8.68 (sec) 63.09 (sec)
    ߴ଎ੑ
    ܥྻ࡟ݮੑೳ
    ҟৗ஫ೖ ίϯϙʔωϯτ

    A
    ίϯϙʔωϯτ

    B
    CPUաෛՙ 1545/201/122

    92.1%
    1541/156/89

    94.2%
    ωοτϫʔΫ

    ஗Ԇ
    1596/248/128
    91.9%
    1543/262/128


    91.7%
    ɾܥྻ࡟ݮੑೳ͕ɺߴ͍ਫ਼౓ͷݪҼ
    ਍அͷͨΊʹे෼͔Ͳ͏͔͸ɺࠓ
    ޙݕূ༧ఆ
    ɾݱ৔ͷো֐ରԠʹద༻͢Δ্Ͱɺ
    े෼ͳੑೳΛಘΒΕͨ
    ௶಺༎थ΄͔, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝ଎ͳ਍அʹ޲͍ͨ࣌ܥྻσʔλͷ࣍ݩ࡟ݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ࿦จू, 2020೥.

    View Slide

  12. 12
    ɾΫϥ΢υ্ͷ෼ࢄΞϓϦέʔγϣϯͷ৴པੑΛ޲্ͤ͞ΔͨΊͷΞʔΩς
    ΫνϟΛݚڀ͍ͯ͠Δɻ


    ɾಛʹɺγεςϜͷӡ༻σʔλΛߴޮ཰ʹऩू͢Δख๏ʹߩݙͨ͠ɻ


    ɾ࣌ܥྻσʔλ


    ɾωοτϫʔΫ௨৴ͷґଘσʔλ


    ɾݱࡏ͸ɺऩू͞Εͨσʔλ͔ΒɺγεςϜʹൃੜ͢ΔҟৗͷݪҼΛಛఆ͢
    Δख๏Λݚڀ͓ͯ͠Γɺ੒Ռ͕Ͱͭͭ͋Δɻ


    ɾ͜ΕΒͷߩݙ͸ɺ৴པੑͷ޲্ʹد༩͢Δ΋ͷͰ͋Δɻ
    ·ͱΊ

    View Slide

  13. 13
    ࣮຿্Ͱ͸ɺӡ༻σʔλͷղੳͷࡍʹɺݱ࣮తʹऩूՄೳͳӡ༻σʔλͷ
    छྨ΍ཻ౓Λ૝ఆ͢Δɻ
    ֶज़తߩݙ
    σʔλͷऩूͱղੳͷ2ஈ֊Λ၆ᛌ্ͨ͠Ͱɺ࣮؀ڥ΁ͷద༻ੑͷߴ͍
    ΞʔΩςΫνϟΛఏࣔ͢Δɻ
    ࣮຿্ͷ՝୊
    ֤։ൃɾӡ༻૊৫ͷݸผͷࣄ৘ʹґଘ͢Δ
    σʔλͷऩूͱղੳ͸ݸผʹٞ࿦͞Ε͍ͯΔ
    طଘ࿦จͷ՝୊

    View Slide

  14. 14
    ɾ Y. Tsubouchi, M. Furukawa, R. Matsumoto, Low Overhead TCP/UDP Socket-based Tracing for
    Discovering Network Services Dependencies, Journal of Information Processing (JIP), 2022.
    ͜Ε·Ͱͷݚڀ੒Ռ: ओͳࠪಡ෇͖࿦จ
    δϟʔφϧ࿦จ
    ࠃࡍձٞ
    ɾ Y. Tsubouchi, M. Furukawa, R. Matsumoto, Transtracer: Socket-Based Tracing of Network Dependencies
    among Processes in Distributed Applications, The 1st IEEE International COMPSAC Workshop on
    Advanced IoT Computing (AIOT 2020), July 2020.
    ࠃ಺γϯϙδ΢Ϝ
    ɾ ௶಺༎थ, ࿬ࡔேਓ, ᖛా݈, দ໦խ޾, খྛོߒ, Ѩ෦ത, দຊ྄հ, HeteroTSDB: ҟछ෼ࢄKVSؒͷࣗಈ֊૚Խ
    ʹΑΔߴੑೳͳ࣌ܥྻσʔλϕʔε, ৘ใॲཧֶձ࿦จࢽ, Vol.62, No.3, pp.818-828, 2021೥3݄.
    ʢ౤ߘதʣ
    ɾ Y. Tsubouchi, A. Wakisaka, K. Hamada, M. Matsuki, H. Abe, R. Matsumoto, HeteroTSDB: An Extensible Time
    Series Database for Automatically Tiering on Heterogeneous Key-Value Stores, The 43rd Annual IEEE
    International Computers, Software & Applications Conference (COMPSAC), pp. 264-269, July 2019.
    ɾ ௶಺༎थ, ௽ాതจ, ݹ઒խେ, TSifter: ϚΠΫϩαʔϏεʹ͓͚Δੑೳҟৗͷਝ଎ͳ਍அʹ޲͍ͨ࣌ܥྻσʔλ
    ͷ࣍ݩ࡟ݮख๏, Πϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ࿦จू, 2020, 9-16 (2020-11-26), 2020೥12݄.
    ɾ ௶಺༎थ, ҏ໺จ඙, ஔాਅੜ, ࢁ઒૱, ദ໦ַ඙, ഡݪ݉Ұ, ॏෳഉআετϨʔδͷͨΊͷSHA-1ܭ
    ࢉγεςϜͷSSE໋ྩʹΑΔߴεϧʔϓοτԽ, ిࢠ৘ใ௨৴ֶձ࿦จࢽ D, 96(10), pp.2101-2109 2013೥10݄.
    ʢຊൃදʹؚ·ͣʣ

    View Slide