ネットワークサービスの依存発見に向いた TCP/UDP通信の低負荷なトレース手法 / Low Overhead TCP-UDP Tracing in Kernel

第8回WebSystemArchitecture研究会
https://wsa.connpass.com/event/207143/

Transcript

1. ωοτϫʔΫαʔϏεͷґଘൃݟʹ޲͍ͨ
   

   TCP/UDP௨৴ͷ௿ෛՙͳτϨʔεख๏
   ୈ8ճWebSystemArchitectureݚڀձ
   
   
   2021೥6݄04೔
   @yuuk1t ͘͞ΒΠϯλʔωοτ
   
   
   id:masayoshi ͸ͯͳ
   

   @matsumotory ͘͞ΒΠϯλʔωοτ
   
   
   

   View Slide

2. 2
   Ϋϥ΢υ্ͷΞϓϦέʔγϣϯͷґଘؔ܎ͷෳࡶԽ
   ɾ
   
   
   ɾ
   
   
   ɾ
   ΞΫηε૿Ճ
   εέʔϧΞ΢τʹΑΔ
   
   
   ϗετ਺ͷ૿Ճ
   ػೳͷ૿Ճ
   ϚΠΫϩαʔϏεͷ࠾༻
   ϛυϧ΢ΣΞͷ૿Ճ
   ɾ
   
   
   ɾ
   
   
   ɾ
   RDBαʔό
   KVαʔό
   ݕࡧαʔό
   Webαʔό
   ωοτϫʔΫ
   

   αʔϏε
   TCP/UDP
   

   View Slide

3. 3
   ґଘؔ܎ͷෳࡶԽ͕΋ͨΒ͢໰୊
   ωοτϫʔΫαʔϏεؒͷ௨৴ͷґଘؔ܎Λ
   
   
   ࣗಈͰτϨʔε͢Δඞཁ͕͋Δ
   ɾґଘؔ܎Λ஌ΒͣʹγεςϜΛमਖ਼͢Δͱɺ૝ఆ֎ͷൣғͷӨڹ͕ൃੜ͢Δ
   
   
   ɾґଘؔ܎͕ෳࡶ͔ͭಈతʹมߋ͞ΕΔͨΊɺهԱ͢Δ͜ͱ΍υΩϡϝϯτԽ
   ͸೉͍͠
   ґଘάϥϑͷ׬શੑ ௿Φʔόϔου
   ύέοτϕʔεΑΓ΋ίωΫγϣϯϕʔε
   ΞϓϦέʔγϣϯมߋෆཁ
   

   View Slide

4. ιέοτϕʔεͷτϨʔεख๏ͷൺֱ
   .
   
   
   .
   
   
   .
   Kernel
   User
   Service
   Socket
   Tracing
   

   Process
   … Event
   Event
   Event
   ετϦʔϛϯά๏(Weave Scope) ϑϩʔू໿๏ ([Datadog], [SAC 20]) ϑϩʔूଋ๏ʢఏҊʣ
   .
   
   
   .
   
   
   .
   Kernel
   User
   Service
   Socket
   Tracing
   

   Process
   .
   
   
   .
   
   
   .
   Event
   Flow
   Event
   Event
   Event
   …
   …
   .
   
   
   .
   
   
   .
   .
   
   
   .
   
   
   .
   User
   Service
   Socket
   Tracing
   

   Process
   Event
   Event
   .
   
   
   .
   
   
   .
   Event
   Event
   Event
   Event
   …
   …
   .
   
   
   .
   
   
   .
   Event
   Event
   …
   …
   .
   
   
   .
   
   
   .
   ✗ ΧʔωϧˠϢʔβۭؒؒ
   ͷΠϕϯτͷίϐʔίετ
   ✗ TCP୹໋઀ଓϨʔτ͕૿
   Ճ͢Δͱɺίϐʔίετ͕
   ૿Ճ
   ෳ਺ͷϑϩʔΛूଋ
   Flow = ྆୺ͷΞυϨεͱϙʔτͷ
   ૊ʢλϓϧʣ͕ಉҰͷ௨৴୯Ґ
   

   View Slide

5. 5
   TCP୹໋઀ଓ τϨʔεॲཧͷCPUෛՙ
   ɾX࣠ TCP୹໋઀ଓ਺ 5k ~ 35k
   
   
   ɾY࣠ ίΞ͋ͨΓͷCPUར༻཰ (0-25%)
   0
   5
   10
   15
   20
   25
   5 10 15 20 25 30 35
   CPU usage / core (%)
   TCP round trips / sec (x103)
   Snap-Poll(client)
   Snap-Poll(server)
   Streaming(client)
   Streaming(server)
   In-Kernel-Aggr(client)
   In-Kernel-Aggr(server)
   In-Kernel-Bundling(client)
   In-Kernel-Bundling(server)
   ɾఏҊख๏ͷCPUར༻཰͸2.2%ҎԼ
   
   
   ɾx࣠૿Ճʹରͯ͠ɺ௿CPUར༻཰Λҡ࣋
   ɾετϦʔϛϯά๏͸ɺX࣠ʹൺྫͯ͠Y͕࣠
   ૿Ճɻ࠷େ20%ఔ౓ͷCPUར༻཰ɻ
   
   
   ɾΧʔωϧ಺ू໿๏͸ɺετϦʔϛϯά๏Α
   ΓY࣠஋͕௿͍ɻ
   

   View Slide

6. 6
   ݚڀͷߩݙ
   1. TCP୹໋઀ଓ΍UDPͷϨʔτ͕େ͖ͳ؀ڥʹ͓͍ͯ΋ɺτϨʔεͷͨΊͷ
   CPUෛՙΛ௿ݮ͢ΔͨΊͷΧʔωϧ಺ϑϩʔूଋ๏ΛఏҊ͢Δɻ
   
   
   2. Linux eBPFʹΑΔద༻ੑΛॏࢹ࣮ͨ͠૷Λࣔ͢ɻ
   
   
   3. TCP/UDPϕϯνϚʔΧʔΛ༻͍࣮ͨݧʹΑΓɺϑϩʔ਺͕૿େͨ͠ͱ͠
   ͯ΋ɺίΞ͋ͨΓͷCPUར༻཰͕2.2%ҎԼͱͳΔ͜ͱΛݕূ͢Δɻ
   

   View Slide

7. 7
   Χʔωϧ಺Ͱͷෳ਺ϑϩʔͷूଋ๏
   ɾωοτϫʔΫαʔϏεؒͷґଘΛ஌Δ͚ͩͳΒɺ୹໋ϙʔτ͸ෆཁ
   
   
   ɾsrc, destΞυϨεɺϦοεϯϙʔτΛҰҙΩʔͱͯ͠ूଋ
   TCP/UDP
   
   
   Connections
   Hash map
   .
   
   
   .
   
   
   .
   .
   
   
   .
   
   
   .
   Key
   Kernel
   User
   Service Service
   Value
   Socket
   daddr
   saddr
   lport
   protocol
   4-tuple
   Tracing
   

   Process
   ɾΧʔωϧۭؒ಺ͷϋογϡ
   දʹ஝ੵ
   
   
   ɾTracingϓϩηε͕Ұఆपظ
   (1sఔ౓)Ͱू໿ΠϕϯτΛ
   όονऔಘ
   connect,
   

   accept,
   

   read, write,…
   .
   
   
   .
   
   
   .
   .
   
   
   .
   
   
   .
   

   View Slide

8. 8
   Χʔωϧ಺ϑϩʔूଋ๏ͷΞϧΰϦζϜ
   ೖྗ: ιέοτߏ଄ମ S, ଴ͪड͚ϙʔτϦετ P
   ू໿ͷͨΊͷϋογϡද H ͷ࡞੒
   
   
   function PROBE__TCP_CONNECT (S)
   
   
   INSERT_FLOW_EVENT(S, TCP)
   
   
   end function
   
   
   function PROBE__TCP_ACCEPT (S)
   
   
   INSERT_FLOW_EVENT(S, TCP)
   
   
   end function
   
   
   function PROBE__TCP_SENDMSG (S)
   
   
   INSERT_FLOW_EVENTS(S, TCP)
   
   
   end function
   
   
   function PROBE__TCP_RECVMSG (S)
   
   
   INSERT_FLOW_EVENTS(S, TCP)
   
   
   end function
   ग़ྗ: ूଋࡁΈͷϑϩʔϦετ
   function GET_LISTENING_PORTS_AND_DIRECTIONS (S)
   
   
   if P.lookup(S.sport) then
   
   
   return S.sport, INCOMING
   
   
   else
   
   
   return S.dport, OUTCOMING
   
   
   end if
   
   
   end function
   
   
   function INSERT_FLOW_EVENTS (S, proto)
   
   
   lport, dir = GET_LISTENING_PORTS_AND_DIRECTIONS(S)
   
   
   key ← {S.saddr, S.daddr, lport, dir, proto}
   
   
   H.update(key, stats)
   
   
   end function
   UDPলུ
   

   View Slide

9. 9
   Χʔωϧ಺ूଋ๏ͷ࣮૷
   1. Χʔωϧؔ਺Λ๣ड͢Δ → LinuxͷkprobeΛ࢖༻
   
   
   ɾΧʔωϧ಺ͷΦϒδΣΫτʹ೚ҙͷϋϯυϥΛઃఆՄೳ
   
   
   2. Χʔωϧ಺Ͱ҆શʹϋϯυϥΛهड़͢Δ → Linux eBPFΛ࢖༻
   
   
   ɾִ཭͞ΕͨαϯυϘοΫε؀ڥͰϓϩάϥϜΛ࣮ߦՄೳ
   
   
   3. ϋογϡදʹूଋ͞ΕͨϑϩʔΛ֨ೲ → eBPF MapΛ࢖༻
   
   
   ɾMapʹ͸Ϣʔβۭ͔ؒΒΞΫηεՄೳ
   ࣮૷͸ https://github.com/yuuki/go-conntracer-bpf ʹͯެ։
   

   View Slide

10. 10
    ࣮૷ͷུ֓ਤ
    TCP/UDP
    
    
    Connections
    Flow table .
    
    
    .
    
    
    .
    .
    
    
    .
    
    
    .
    4-tuple
    Kernel
    User
    Service Service
    Stat
    Socket
    Tracing
    

    Process
    connect,
    

    accept,
    

    sendto,
    

    recv
    .
    
    
    .
    
    
    .
    .
    
    
    .
    
    
    .
    tcp_v4_connect()
    inet_csk_accept()
    ىಈ࣌ʹ଴ͪड
    ͚ϙʔτͷ
    

    ϦετΛऔಘ
    tcp_sendmsg()
    tcp_cleanup_rbuf()
    UDPলུ
    kprobeͰΞλον͢Δ
    

    Χʔωϧؔ਺
    insert
    Batch APIʹΑ
    Γෳ਺Ϩίʔυ
    ಉ࣌औಘ
    MapΛ࢖༻
    

    View Slide

11. 11
    ࣮ݧͷͨΊͷܭࢉػ؀ڥ
    Client (ཁٻଆ) Server (଴डଆ)
    Tracing
    Process
    Tracing
    Process
    benchmark
    process
    benchmark
    process
    ؀ڥ1: 1ର1 ωοτϫʔΫαʔϏε
    ؀ڥ2: 1ରN ωοτϫʔΫαʔϏε
    Client (ཁٻଆ) Server (଴डଆ)
    Tracing
    Process
    benchmark
    process
    ɾɾɾ
    ɾɾɾ
    ɾɾɾ
    • CPU: Intel Xeon Gold 6212U
    @2.40GHz x 6ίΞ
    
    
    •
    ϝϞϦ: 16GB
    
    
    • OS: Ubuntu 20.10 Kernel 5.8.0-33
    Connperf: https://github.com/yuuki/connperf
    ෛՙੜ੒
    
    
    • ࣗ࡞ͷΤίʔαʔόɾΫϥΠΞϯτ
    
    
    • ඵؒͷ৽ن઀ଓ਺ΛݻఆՄೳ
    

    View Slide

12. 12
    τϨʔεॲཧͷCPUෛՙʢ1ର1؀ڥʣ
    ɾఏҊख๏͸ɺCPUར༻཰Λ2.2%ҎԼ·Ͱ௿ݮ
    TCP୹໋઀ଓ UDP
    ఏҊख๏
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    Number of TCP connections (x103)
    Snap-Poll(client)
    Snap-Poll(server)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    TCP round trips / sec (x103)
    Snap-Poll(client)
    Snap-Poll(server)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    0
    5
    10
    15
    20
    25
    5 10 15 20 25 30 35
    CPU usage / core (%)
    UDP round trips / sec (x103)
    Streaming(client)
    Streaming(server)
    In-Kernel-Aggr(client)
    In-Kernel-Aggr(server)
    In-Kernel-Bundling(client)
    In-Kernel-Bundling(server)
    TCPӬଓ઀ଓ
    

    View Slide

13. 13
    ௨৴ઌͷαʔϏε਺ʹର͢ΔCPUෛՙʢ1ରN؀ڥʣ
    ɾඵؒ઀ଓ਺Λ10kʹݻఆ (T=10k)
    
    
    ɾαʔϏε਺Λ200ʙ1000·Ͱ૿Ճ
    

    (R=0.98 .. 0.9)
    
    
    ɾαʔϏε਺ͷ૿େʹରͯ͠ɺCPUར༻཰͸2%Ҏ
    Լ
    ɾҟͳΔ଴ͪड͚ϙʔτΛ΋ͭ௨৴ઌ͕૿͑Δͱɺूଋ཰͕௿Լ
    
    
    ↪ CPUෛՙ͕૿େ͢Δ͸ͣ
    0
    1
    2
    3
    4
    5
    200 400 600 800 1000
    CPU usage per core (%)
    Number of network services
    In-Kernel-Bundling(client,TCP short-lived)
    In-Kernel-Bundling(server,TCP short-lived)
    In-Kernel-Bundling(client,TCP persistent)
    In-Kernel-Bundling(server,TCP persistent)
    In-Kernel-Bundling(server,UDP)
    In-Kernel-Bundling(client,UDP)
    ूଋ཰ R = 1 - B / T
    
    
    B: ूଋ͞Εͨϑϩʔ਺
    
    
    T: ϢχʔΫϑϩʔ਺
    

    View Slide

14. 14
    ΞϓϦέʔγϣϯͷ஗ԆΦʔόϔου
    TCP୹໋઀ଓ UDP
    Ping RTT 300μs ʹରͯ͠ɺఏҊख๏ͷΦʔόϔου͸ 5.8 μsɻ
    

    2%ͷΦʔόϔου૿Ճɻ
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    UDP round trips / sec
    Streaming
    3.6
    3.2 3.1 3.1
    In−Kernel−Aggr
    4.6
    4.1 4.0
    4.2
    In−Kernel−Bundling
    4.1
    3.4 3.3 3.4
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    Number of TCP connections
    Streaming
    0 0 0 0
    In−Kernel−Aggr
    2.4
    2.1
    2.3 2.2
    In−Kernel−Bundling
    2.4
    2.2 2.1 2.1
    0
    1
    2
    3
    4
    5
    6
    5 10 15 20
    RTT overhead (µs)
    TCP round trips / sec
    Streaming
    2.2 2.1 2.1
    2.5
    In−Kernel−Aggr
    4.7 4.7
    5.0
    5.7
    In−Kernel−Bundling
    4.8 4.9 5.0
    5.8
    TCP࣋ଓత઀ଓ
    

    View Slide

15. 15
    ·ͱΊ
    ɾTCP/UDP௨৴Λ௿ΦʔόϔουͰτϨʔε͢ΔͨΊʹɺΧʔωϧ಺
    ϑϩʔूଋ๏ΛఏҊͨ͠
    
    
    ɾ࣮ݧͷ݁ՌɺTCPͷ୹໋ͳ઀ଓ਺ͷ૿Ճʹରͯ͠ɺఏҊख๏͸2.2%
    ҎԼͷCPUར༻཰Λҡ࣋ͨ͠
    
    
    ɾࠓޙͷ༧ఆ
    
    
    ɾπʔϧͱͯ͠ͷ׬੒౓ΛߴΊͯɺ࣮ࡍʹར༻ͯ͠΋Β͏
    

    View Slide