Slide 1

Slide 1 text

Spying Linux processes @javierhonduco

Slide 2

Slide 2 text

Before this summer

Slide 3

Slide 3 text

CODE CODE *printf debugging* CODE

Slide 4

Slide 4 text

This summer

Slide 5

Slide 5 text

CODE *printf debugging* *debugging* *wth is wrong… it _should work_!* CODE *WAT* *debugging* *moar printf debugging* CODE

Slide 6

Slide 6 text

*debugging* *WAT* *debugging* *debugging* *debugging* *eats a cookie* *debugging* CODE

Slide 7

Slide 7 text

¯\_(ツ)_/¯

Slide 8

Slide 8 text

The OS ~= API ● Network: bind(), listen(), accept(), read()... ● Files: open(), read(), write()... ● Memory allocation: mmap(), malloc(), brk()? ● Threads et al: pthreads_

Slide 9

Slide 9 text

Mmmm, something’s not quite ok!

Slide 10

Slide 10 text

1. My process is kindof stuck for no reason! 2. The file it should write is empty! 3. Its socket doesn’t get a single byte! 4. I just run out of FDs, but I was only using one 5. {CPU, disk, memory} usage is too high

Slide 11

Slide 11 text

Let’s peek into!

Slide 12

Slide 12 text

1. Stuck process. Let’s strace!

Slide 13

Slide 13 text

strace -p

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

2. It’s writing to another file! Let’s strace again

Slide 16

Slide 16 text

It opens the wrong file!!! But I set another path in the “WHATEVER_LOGGER” ENV variable...

Slide 17

Slide 17 text

Let’s into /proc//environment

Slide 18

Slide 18 text

Ooops! The ENV var was not set in the appropriate place

Slide 19

Slide 19 text

/proc//<*> is pretty rad Exposes kernel data structures in the VFS

Slide 20

Slide 20 text

3. It doesn’t reach a server ngrep to the rescue!

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

4. I run out of FDs asks coworker :D

Slide 24

Slide 24 text

“Javier, try with lsof”

Slide 25

Slide 25 text

lsof -p

Slide 26

Slide 26 text

Thousands like this!

Slide 27

Slide 27 text

It ended up being a bug on a Ruby library written in C

Slide 28

Slide 28 text

5. CPU DISK RAM

Slide 29

Slide 29 text

perf (A bit out of the scope of this talk. Also, I’m even more newbie on this! )

Slide 30

Slide 30 text

Graphs!

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Linux 4.1 and above… BPF Compiler Collection (BCC) Basically low overhead kernel tracing!

Slide 33

Slide 33 text

Flamegraphs, heatmaps, histograms etc etc

Slide 34

Slide 34 text

valgrind --leak-check=yes \ ./maybe_leaking_program

Slide 35

Slide 35 text

What have I learnt? ● Unix tools are awesome! ● /proc//<*>!! ● I write lots of bugs! (but hopefully, got a bit better at debugging) ● Things are going to fail in every single way they can (and that could be fun!)

Slide 36

Slide 36 text

Merci!

Slide 37

Slide 37 text

Interesting links/ bibliography [1] iovisor: https://github.com/iovisor/bcc/ [2] Julia Evans: http://jvns.ca/ [3] perf: https://perf.wiki.kernel.org/index.php/Main_Page [4] Brendan Gregg: http://www.brendangregg.com/ [6] BPF syntax http://biot.com/capstats/bpf.html [7] Man pages are useful too! (but I do need examples as well :P)