CODE *printf debugging* *debugging* *wth
is wrong… it _should work_!* CODE *WAT*
*debugging* *moar printf debugging* CODE
Slide 6
Slide 6 text
*debugging* *WAT* *debugging* *debugging*
*debugging* *eats a cookie* *debugging*
CODE
Slide 7
Slide 7 text
¯\_(ツ)_/¯
Slide 8
Slide 8 text
The OS ~= API
● Network: bind(), listen(), accept(), read()...
● Files: open(), read(), write()...
● Memory allocation: mmap(), malloc(), brk()?
● Threads et al: pthreads_
Slide 9
Slide 9 text
Mmmm, something’s not quite ok!
Slide 10
Slide 10 text
1. My process is kindof stuck for no reason!
2. The file it should write is empty!
3. Its socket doesn’t get a single byte!
4. I just run out of FDs, but I was only using one
5. {CPU, disk, memory} usage is too high
Slide 11
Slide 11 text
Let’s peek into!
Slide 12
Slide 12 text
1. Stuck process.
Let’s strace!
Slide 13
Slide 13 text
strace -p
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
2. It’s writing to another file!
Let’s strace again
Slide 16
Slide 16 text
It opens the wrong file!!!
But I set another path in the
“WHATEVER_LOGGER” ENV variable...
Slide 17
Slide 17 text
Let’s into /proc//environment
Slide 18
Slide 18 text
Ooops! The ENV var was not set
in the appropriate place
Slide 19
Slide 19 text
/proc//<*> is pretty rad
Exposes kernel data structures in the VFS
Slide 20
Slide 20 text
3. It doesn’t reach a server
ngrep to the rescue!
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
4. I run out of FDs
asks coworker :D
Slide 24
Slide 24 text
“Javier, try with lsof”
Slide 25
Slide 25 text
lsof -p
Slide 26
Slide 26 text
Thousands like this!
Slide 27
Slide 27 text
It ended up
being a bug on a
Ruby library
written in C
Slide 28
Slide 28 text
5.
CPU
DISK
RAM
Slide 29
Slide 29 text
perf
(A bit out of the scope of this talk. Also, I’m even
more newbie on this! )
Slide 30
Slide 30 text
Graphs!
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
Linux 4.1 and above…
BPF Compiler Collection (BCC)
Basically low overhead kernel tracing!
What have I learnt?
● Unix tools are awesome!
● /proc//<*>!!
● I write lots of bugs! (but hopefully, got a bit better at debugging)
● Things are going to fail in every single way they can (and that could
be fun!)
Slide 36
Slide 36 text
Merci!
Slide 37
Slide 37 text
Interesting links/ bibliography
[1] iovisor: https://github.com/iovisor/bcc/
[2] Julia Evans: http://jvns.ca/
[3] perf: https://perf.wiki.kernel.org/index.php/Main_Page
[4] Brendan Gregg: http://www.brendangregg.com/
[6] BPF syntax http://biot.com/capstats/bpf.html
[7] Man pages are useful too! (but I do need examples as well :P)