Linux Delay Accounting

Ever wondered how long is your program spending while waiting for I/O to finish? Or if it is spending lots of time while waiting for a turn to run on one of the CPUs? Linux provides delay accounting information that may help to answer questions like these and we are going to see how they work in this talk!

Transcript

1. Linux Delay Accounting
   André Carvalho
   Gophercon 2018 - Lightning Talks
   1
   @andresantostc
   

   View Slide

2. André Carvalho
   Engineer at Globo.com
   tsuru.io
   @andresantostc
   https://andrestc.com
   2
   

   View Slide

3. Our code runs alongside
   3
   1. Daemons
   2. Agents
   3. Kernel Threads
   4. Other apps
   5. ...
   @andresantostc
   

   View Slide

4. Tasks may encounter delays in execution
   waiting for Kernel resources
   4
   @andresantostc
   Process
   Hey Mr. Kernel, I’m
   ready to run, sir.
   Kernel
   Wait in line, please.
   

   View Slide

5. How long are those delays?
   5
   @andresantostc
   

   View Slide

6. Linux Delay Accounting
   ● Linux exposes delays encountered by tasks(threads/processes)
   ○ waiting for a CPU (while being runnable)
   ○ completion of synchronous block I/O
   ○ swapping in pages
   ○ memory reclaim
   ● Can be used to
   ○ Analyze system saturation
   ○ Configure task priorities
   ● Available since Kernel 2.6.18 (we are at 4.18)
   ● Recently added to htop (https://github.com/hishamhm/htop)
   6
   @andresantostc
   

   View Slide

7. ● Via Netlink interface (taskstats)
   ● Example code on the Linux source (tools/accounting/getdelays.c)
   ○ Example using higher level library (https://github.com/andrestc/linux-delays)
   ● Requires root or CAP_NET_ADMIN
   7
   @andresantostc
   

   View Slide

8. ● https://github.com/mdlayher/taskstats
   ○ Library
   ● https://github.com/andrestc/delaystat
   ○ CLI tool
   ○ Work in progress!
   ○ Looking for contributors
   8
   @andresantostc
   

   View Slide

9. Demo
   9
   @andresantostc
   

   View Slide

10. References
    1. https://github.com/hishamhm/htop/issues/665
    2. https://andrestc.com/post/linux-delay-accounting/
    3. https://www.kernel.org/doc/Documentation/accounting/delay-accounting.txt
    4. https://github.com/mdlayher/taskstats
    5. https://github.com/andrestc/linux-delays
    6. https://people.seas.harvard.edu/~apw/stress/
    10
    @andresantostc
    

    View Slide

11. Thanks!
    https://andrestc.com/talks
    @andresantostc
    11
    

    View Slide