Linux Delay Accounting - Speaker Deck

Slide 1

Slide 1 text

Linux Delay Accounting André Carvalho Gophercon 2018 - Lightning Talks 1 @andresantostc

Slide 2

Slide 2 text

André Carvalho Engineer at Globo.com tsuru.io @andresantostc https://andrestc.com 2

Slide 3

Slide 3 text

Our code runs alongside 3 1. Daemons 2. Agents 3. Kernel Threads 4. Other apps 5. ... @andresantostc

Slide 4

Slide 4 text

Tasks may encounter delays in execution waiting for Kernel resources 4 @andresantostc Process Hey Mr. Kernel, I’m ready to run, sir. Kernel Wait in line, please.

Slide 5

Slide 5 text

How long are those delays? 5 @andresantostc

Slide 6

Slide 6 text

Linux Delay Accounting ● Linux exposes delays encountered by tasks(threads/processes) ○ waiting for a CPU (while being runnable) ○ completion of synchronous block I/O ○ swapping in pages ○ memory reclaim ● Can be used to ○ Analyze system saturation ○ Configure task priorities ● Available since Kernel 2.6.18 (we are at 4.18) ● Recently added to htop (https://github.com/hishamhm/htop) 6 @andresantostc

Slide 7

Slide 7 text

● Via Netlink interface (taskstats) ● Example code on the Linux source (tools/accounting/getdelays.c) ○ Example using higher level library (https://github.com/andrestc/linux-delays) ● Requires root or CAP_NET_ADMIN 7 @andresantostc

Slide 8

Slide 8 text

● https://github.com/mdlayher/taskstats ○ Library ● https://github.com/andrestc/delaystat ○ CLI tool ○ Work in progress! ○ Looking for contributors 8 @andresantostc

Slide 9

Slide 9 text

Demo 9 @andresantostc

Slide 10

Slide 10 text

References 1. https://github.com/hishamhm/htop/issues/665 2. https://andrestc.com/post/linux-delay-accounting/ 3. https://www.kernel.org/doc/Documentation/accounting/delay-accounting.txt 4. https://github.com/mdlayher/taskstats 5. https://github.com/andrestc/linux-delays 6. https://people.seas.harvard.edu/~apw/stress/ 10 @andresantostc

Slide 11

Slide 11 text

Thanks! https://andrestc.com/talks @andresantostc 11