$30 off During Our Annual Pro Sale. View Details »

Linux Delay Accounting

Linux Delay Accounting

Ever wondered how long is your program spending while waiting for I/O to finish? Or if it is spending lots of time while waiting for a turn to run on one of the CPUs? Linux provides delay accounting information that may help to answer questions like these and we are going to see how they work in this talk!

André Carvalho

August 30, 2018
Tweet

More Decks by André Carvalho

Other Decks in Programming

Transcript

  1. Linux Delay Accounting
    André Carvalho
    Gophercon 2018 - Lightning Talks
    1
    @andresantostc

    View Slide

  2. André Carvalho
    Engineer at Globo.com
    tsuru.io
    @andresantostc
    https://andrestc.com
    2

    View Slide

  3. Our code runs alongside
    3
    1. Daemons
    2. Agents
    3. Kernel Threads
    4. Other apps
    5. ...
    @andresantostc

    View Slide

  4. Tasks may encounter delays in execution
    waiting for Kernel resources
    4
    @andresantostc
    Process
    Hey Mr. Kernel, I’m
    ready to run, sir.
    Kernel
    Wait in line, please.

    View Slide

  5. How long are those delays?
    5
    @andresantostc

    View Slide

  6. Linux Delay Accounting
    ● Linux exposes delays encountered by tasks(threads/processes)
    ○ waiting for a CPU (while being runnable)
    ○ completion of synchronous block I/O
    ○ swapping in pages
    ○ memory reclaim
    ● Can be used to
    ○ Analyze system saturation
    ○ Configure task priorities
    ● Available since Kernel 2.6.18 (we are at 4.18)
    ● Recently added to htop (https://github.com/hishamhm/htop)
    6
    @andresantostc

    View Slide

  7. ● Via Netlink interface (taskstats)
    ● Example code on the Linux source (tools/accounting/getdelays.c)
    ○ Example using higher level library (https://github.com/andrestc/linux-delays)
    ● Requires root or CAP_NET_ADMIN
    7
    @andresantostc

    View Slide

  8. ● https://github.com/mdlayher/taskstats
    ○ Library
    ● https://github.com/andrestc/delaystat
    ○ CLI tool
    ○ Work in progress!
    ○ Looking for contributors
    8
    @andresantostc

    View Slide

  9. Demo
    9
    @andresantostc

    View Slide

  10. References
    1. https://github.com/hishamhm/htop/issues/665
    2. https://andrestc.com/post/linux-delay-accounting/
    3. https://www.kernel.org/doc/Documentation/accounting/delay-accounting.txt
    4. https://github.com/mdlayher/taskstats
    5. https://github.com/andrestc/linux-delays
    6. https://people.seas.harvard.edu/~apw/stress/
    10
    @andresantostc

    View Slide

  11. Thanks!
    https://andrestc.com/talks
    @andresantostc
    11

    View Slide