$30 off During Our Annual Pro Sale. View Details »

ネットワークサービスの依存発見に向いた
TCP/UDP通信の低負荷なトレース手法 / Low Overhead TCP-UDP Tracing in Kernel

ネットワークサービスの依存発見に向いた
TCP/UDP通信の低負荷なトレース手法 / Low Overhead TCP-UDP Tracing in Kernel

第8回WebSystemArchitecture研究会
https://wsa.connpass.com/event/207143/

Yuuki Tsubouchi (yuuk1)

June 05, 2021
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Research

Transcript

  1. ωοτϫʔΫαʔϏεͷґଘൃݟʹ޲͍ͨ

    TCP/UDP௨৴ͷ௿ෛՙͳτϨʔεख๏
    ୈ8ճWebSystemArchitectureݚڀձ


    2021೥6݄04೔
    @yuuk1t ͘͞ΒΠϯλʔωοτ


    id:masayoshi ͸ͯͳ

    @matsumotory ͘͞ΒΠϯλʔωοτ


    View Slide

  2. 2
    Ϋϥ΢υ্ͷΞϓϦέʔγϣϯͷґଘؔ܎ͷෳࡶԽ
    ɾ


    ɾ


    ɾ
    ΞΫηε૿Ճ
    εέʔϧΞ΢τʹΑΔ


    ϗετ਺ͷ૿Ճ
    ػೳͷ૿Ճ
    ϚΠΫϩαʔϏεͷ࠾༻
    ϛυϧ΢ΣΞͷ૿Ճ
    ɾ


    ɾ


    ɾ
    RDBαʔό
    KVαʔό
    ݕࡧαʔό
    Webαʔό
    ωοτϫʔΫ

    αʔϏε
    TCP/UDP

    View Slide

  3. 3
    ґଘؔ܎ͷෳࡶԽ͕΋ͨΒ͢໰୊
    ωοτϫʔΫαʔϏεؒͷ௨৴ͷґଘؔ܎Λ


    ࣗಈͰτϨʔε͢Δඞཁ͕͋Δ
    ɾґଘؔ܎Λ஌ΒͣʹγεςϜΛमਖ਼͢Δͱɺ૝ఆ֎ͷൣғͷӨڹ͕ൃੜ͢Δ


    ɾґଘؔ܎͕ෳࡶ͔ͭಈతʹมߋ͞ΕΔͨΊɺهԱ͢Δ͜ͱ΍υΩϡϝϯτԽ
    ͸೉͍͠
    ґଘάϥϑͷ׬શੑ ௿Φʔόϔου
    ύέοτϕʔεΑΓ΋ίωΫγϣϯϕʔε
    ΞϓϦέʔγϣϯมߋෆཁ

    View Slide

  4. ιέοτϕʔεͷτϨʔεख๏ͷൺֱ
    .


    .


    .
    Kernel
    User
    Service
    Socket
    Tracing

    Process
    … Event
    Event
    Event
    ετϦʔϛϯά๏(Weave Scope) ϑϩʔू໿๏ ([Datadog], [SAC 20]) ϑϩʔूଋ๏ʢఏҊʣ
    .


    .


    .
    Kernel
    User
    Service
    Socket
    Tracing

    Process
    .


    .


    .
    Event
    Flow
    Event
    Event
    Event


    .


    .


    .
    .


    .


    .
    User
    Service
    Socket
    Tracing

    Process
    Event
    Event
    .


    .


    .
    Event
    Event
    Event
    Event


    .


    .


    .
    Event
    Event


    .


    .


    .
    ✗ ΧʔωϧˠϢʔβۭؒؒ
    ͷΠϕϯτͷίϐʔίετ
    ✗ TCP୹໋઀ଓϨʔτ͕૿
    Ճ͢Δͱɺίϐʔίετ͕
    ૿Ճ
    ෳ਺ͷϑϩʔΛूଋ
    Flow = ྆୺ͷΞυϨεͱϙʔτͷ
    ૊ʢλϓϧʣ͕ಉҰͷ௨৴୯Ґ

    View Slide

  5. 5
    TCP୹໋઀ଓ τϨʔεॲཧͷCPUෛՙ
    ɾX࣠ TCP୹໋઀ଓ਺ 5k ~ 35k


    ɾY࣠ ίΞ͋ͨΓͷCPUར༻཰ (0-25%)
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    TCP round trips / sec (x103)
    Snap-Poll(client)
    Snap-Poll(server)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    ɾఏҊख๏ͷCPUར༻཰͸2.2%ҎԼ


    ɾx࣠૿Ճʹରͯ͠ɺ௿CPUར༻཰Λҡ࣋
    ɾετϦʔϛϯά๏͸ɺX࣠ʹൺྫͯ͠Y͕࣠
    ૿Ճɻ࠷େ20%ఔ౓ͷCPUར༻཰ɻ


    ɾΧʔωϧ಺ू໿๏͸ɺετϦʔϛϯά๏Α
    ΓY࣠஋͕௿͍ɻ

    View Slide

  6. 6
    ݚڀͷߩݙ
    1. TCP୹໋઀ଓ΍UDPͷϨʔτ͕େ͖ͳ؀ڥʹ͓͍ͯ΋ɺτϨʔεͷͨΊͷ
    CPUෛՙΛ௿ݮ͢ΔͨΊͷΧʔωϧ಺ϑϩʔूଋ๏ΛఏҊ͢Δɻ


    2. Linux eBPFʹΑΔద༻ੑΛॏࢹ࣮ͨ͠૷Λࣔ͢ɻ


    3. TCP/UDPϕϯνϚʔΧʔΛ༻͍࣮ͨݧʹΑΓɺϑϩʔ਺͕૿େͨ͠ͱ͠
    ͯ΋ɺίΞ͋ͨΓͷCPUར༻཰͕2.2%ҎԼͱͳΔ͜ͱΛݕূ͢Δɻ

    View Slide

  7. 7
    Χʔωϧ಺Ͱͷෳ਺ϑϩʔͷूଋ๏
    ɾωοτϫʔΫαʔϏεؒͷґଘΛ஌Δ͚ͩͳΒɺ୹໋ϙʔτ͸ෆཁ


    ɾsrc, destΞυϨεɺϦοεϯϙʔτΛҰҙΩʔͱͯ͠ूଋ
    TCP/UDP


    Connections
    Hash map
    .


    .


    .
    .


    .


    .
    Key
    Kernel
    User
    Service Service
    Value
    Socket
    daddr
    saddr
    lport
    protocol
    4-tuple
    Tracing

    Process
    ɾΧʔωϧۭؒ಺ͷϋογϡ
    දʹ஝ੵ


    ɾTracingϓϩηε͕Ұఆपظ
    (1sఔ౓)Ͱू໿ΠϕϯτΛ
    όονऔಘ
    connect,

    accept,

    read, write,…
    .


    .


    .
    .


    .


    .

    View Slide

  8. 8
    Χʔωϧ಺ϑϩʔूଋ๏ͷΞϧΰϦζϜ
    ೖྗ: ιέοτߏ଄ମ S, ଴ͪड͚ϙʔτϦετ P
    ू໿ͷͨΊͷϋογϡද H ͷ࡞੒


    function PROBE__TCP_CONNECT (S)


    INSERT_FLOW_EVENT(S, TCP)


    end function


    function PROBE__TCP_ACCEPT (S)


    INSERT_FLOW_EVENT(S, TCP)


    end function


    function PROBE__TCP_SENDMSG (S)


    INSERT_FLOW_EVENTS(S, TCP)


    end function


    function PROBE__TCP_RECVMSG (S)


    INSERT_FLOW_EVENTS(S, TCP)


    end function
    ग़ྗ: ूଋࡁΈͷϑϩʔϦετ
    function GET_LISTENING_PORTS_AND_DIRECTIONS (S)


    if P.lookup(S.sport) then


    return S.sport, INCOMING


    else


    return S.dport, OUTCOMING


    end if


    end function


    function INSERT_FLOW_EVENTS (S, proto)


    lport, dir = GET_LISTENING_PORTS_AND_DIRECTIONS(S)


    key ← {S.saddr, S.daddr, lport, dir, proto}


    H.update(key, stats)


    end function
    UDPলུ

    View Slide

  9. 9
    Χʔωϧ಺ूଋ๏ͷ࣮૷
    1. Χʔωϧؔ਺Λ๣ड͢Δ → LinuxͷkprobeΛ࢖༻


    ɾΧʔωϧ಺ͷΦϒδΣΫτʹ೚ҙͷϋϯυϥΛઃఆՄೳ


    2. Χʔωϧ಺Ͱ҆શʹϋϯυϥΛهड़͢Δ → Linux eBPFΛ࢖༻


    ɾִ཭͞ΕͨαϯυϘοΫε؀ڥͰϓϩάϥϜΛ࣮ߦՄೳ


    3. ϋογϡදʹूଋ͞ΕͨϑϩʔΛ֨ೲ → eBPF MapΛ࢖༻


    ɾMapʹ͸Ϣʔβۭ͔ؒΒΞΫηεՄೳ
    ࣮૷͸ https://github.com/yuuki/go-conntracer-bpf ʹͯެ։

    View Slide

  10. 10
    ࣮૷ͷུ֓ਤ
    TCP/UDP


    Connections
    Flow table .


    .


    .
    .


    .


    .
    4-tuple
    Kernel
    User
    Service Service
    Stat
    Socket
    Tracing

    Process
    connect,

    accept,

    sendto,

    recv
    .


    .


    .
    .


    .


    .
    tcp_v4_connect()
    inet_csk_accept()
    ىಈ࣌ʹ଴ͪड
    ͚ϙʔτͷ

    ϦετΛऔಘ
    tcp_sendmsg()
    tcp_cleanup_rbuf()
    UDPলུ
    kprobeͰΞλον͢Δ

    Χʔωϧؔ਺
    insert
    Batch APIʹΑ
    Γෳ਺Ϩίʔυ
    ಉ࣌औಘ
    MapΛ࢖༻

    View Slide

  11. 11
    ࣮ݧͷͨΊͷܭࢉػ؀ڥ
    Client (ཁٻଆ) Server (଴डଆ)
    Tracing
    Process
    Tracing
    Process
    benchmark
    process
    benchmark
    process
    ؀ڥ1: 1ର1 ωοτϫʔΫαʔϏε
    ؀ڥ2: 1ରN ωοτϫʔΫαʔϏε
    Client (ཁٻଆ) Server (଴डଆ)
    Tracing
    Process
    benchmark
    process
    ɾɾɾ
    ɾɾɾ
    ɾɾɾ
    • CPU: Intel Xeon Gold 6212U
    @2.40GHz x 6ίΞ



    ϝϞϦ: 16GB


    • OS: Ubuntu 20.10 Kernel 5.8.0-33
    Connperf: https://github.com/yuuki/connperf
    ෛՙੜ੒


    • ࣗ࡞ͷΤίʔαʔόɾΫϥΠΞϯτ


    • ඵؒͷ৽ن઀ଓ਺ΛݻఆՄೳ

    View Slide

  12. 12
    τϨʔεॲཧͷCPUෛՙʢ1ର1؀ڥʣ
    ɾఏҊख๏͸ɺCPUར༻཰Λ2.2%ҎԼ·Ͱ௿ݮ
    TCP୹໋઀ଓ UDP
    ఏҊख๏
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    Number of TCP connections (x103)
    Snap-Poll(client)
    Snap-Poll(server)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    TCP round trips / sec (x103)
    Snap-Poll(client)
    Snap-Poll(server)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    UDP round trips / sec (x103)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    TCPӬଓ઀ଓ

    View Slide

  13. 13
    ௨৴ઌͷαʔϏε਺ʹର͢ΔCPUෛՙʢ1ରN؀ڥʣ
    ɾඵؒ઀ଓ਺Λ10kʹݻఆ (T=10k)


    ɾαʔϏε਺Λ200ʙ1000·Ͱ૿Ճ

    (R=0.98 .. 0.9)


    ɾαʔϏε਺ͷ૿େʹରͯ͠ɺCPUར༻཰͸2%Ҏ
    Լ
    ɾҟͳΔ଴ͪड͚ϙʔτΛ΋ͭ௨৴ઌ͕૿͑Δͱɺूଋ཰͕௿Լ


    ↪ CPUෛՙ͕૿େ͢Δ͸ͣ
    0
    1
    2
    3
    4
    5
    200 400 600 800 1000
    CPU usage per core (%)
    Number of network services
    In-Kernel-Bundling(client,TCP short-lived)
    In-Kernel-Bundling(server,TCP short-lived)
    In-Kernel-Bundling(client,TCP persistent)
    In-Kernel-Bundling(server,TCP persistent)
    In-Kernel-Bundling(server,UDP)
    In-Kernel-Bundling(client,UDP)
    ूଋ཰ R = 1 - B / T


    B: ूଋ͞Εͨϑϩʔ਺


    T: ϢχʔΫϑϩʔ਺

    View Slide

  14. 14
    ΞϓϦέʔγϣϯͷ஗ԆΦʔόϔου
    TCP୹໋઀ଓ UDP
    Ping RTT 300μs ʹରͯ͠ɺఏҊख๏ͷΦʔόϔου͸ 5.8 μsɻ

    2%ͷΦʔόϔου૿Ճɻ
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    UDP round trips / sec
    Streaming
    3.6
    3.2 3.1 3.1
    In−Kernel−Aggr
    4.6
    4.1 4.0
    4.2
    In−Kernel−Bundling
    4.1
    3.4 3.3 3.4
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    Number of TCP connections
    Streaming
    0 0 0 0
    In−Kernel−Aggr
    2.4
    2.1
    2.3 2.2
    In−Kernel−Bundling
    2.4
    2.2 2.1 2.1
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    TCP round trips / sec
    Streaming
    2.2 2.1 2.1
    2.5
    In−Kernel−Aggr
    4.7 4.7
    5.0
    5.7
    In−Kernel−Bundling
    4.8 4.9 5.0
    5.8
    TCP࣋ଓత઀ଓ

    View Slide

  15. 15
    ·ͱΊ
    ɾTCP/UDP௨৴Λ௿ΦʔόϔουͰτϨʔε͢ΔͨΊʹɺΧʔωϧ಺
    ϑϩʔूଋ๏ΛఏҊͨ͠


    ɾ࣮ݧͷ݁ՌɺTCPͷ୹໋ͳ઀ଓ਺ͷ૿Ճʹରͯ͠ɺఏҊख๏͸2.2%
    ҎԼͷCPUར༻཰Λҡ࣋ͨ͠


    ɾࠓޙͷ༧ఆ


    ɾπʔϧͱͯ͠ͷ׬੒౓ΛߴΊͯɺ࣮ࡍʹར༻ͯ͠΋Β͏

    View Slide