Linux Delay Accounting

Ever wondered how long is your program spending while waiting for I/O to finish? Or if it is spending lots of time while waiting for a turn to run on one of the CPUs? Linux provides delay accounting information that may help to answer questions like these and we are going to see how they work in this talk!

Transcript

1. Linux Delay Accounting André Carvalho Gophercon 2018 - Lightning Talks 1 @andresantostc
2. André Carvalho Engineer at Globo.com tsuru.io @andresantostc https://andrestc.com 2
3. Our code runs alongside 3 1. Daemons 2. Agents 3. Kernel Threads 4. Other apps 5. ... @andresantostc
4. Tasks may encounter delays in execution waiting for Kernel resources 4 @andresantostc Process Hey Mr. Kernel, I’m ready to run, sir. Kernel Wait in line, please.
5. How long are those delays? 5 @andresantostc
6. Linux Delay Accounting • Linux exposes delays encountered by tasks(threads/processes) ◦ waiting for a CPU (while being runnable) ◦ completion of synchronous block I/O ◦ swapping in pages ◦ memory reclaim • Can be used to ◦ Analyze system saturation ◦ Configure task priorities • Available since Kernel 2.6.18 (we are at 4.18) • Recently added to htop (https://github.com/hishamhm/htop) 6 @andresantostc
7. • Via Netlink interface (taskstats) • Example code on the Linux source (tools/accounting/getdelays.c) ◦ Example using higher level library (https://github.com/andrestc/linux-delays) • Requires root or CAP_NET_ADMIN 7 @andresantostc
8. • https://github.com/mdlayher/taskstats ◦ Library • https://github.com/andrestc/delaystat ◦ CLI tool ◦ Work in progress! ◦ Looking for contributors 8 @andresantostc
9. Demo 9 @andresantostc
10. References 1. https://github.com/hishamhm/htop/issues/665 2. https://andrestc.com/post/linux-delay-accounting/ 3. https://www.kernel.org/doc/Documentation/accounting/delay-accounting.txt 4. https://github.com/mdlayher/taskstats 5. https://github.com/andrestc/linux-delays 6. https://people.seas.harvard.edu/~apw/stress/ 10 @andresantostc
11. Thanks! https://andrestc.com/talks @andresantostc 11