Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Spying Linux processes

Spying Linux processes


Javier Honduvilla Coto

November 24, 2016


  1. Spying Linux processes @javierhonduco

  2. Before this summer

  3. CODE CODE *printf debugging* CODE

  4. This summer

  5. CODE *printf debugging* *debugging* *wth is wrong… it _should work_!*

    CODE *WAT* *debugging* *moar printf debugging* CODE
  6. *debugging* *WAT* *debugging* *debugging* *debugging* *eats a cookie* *debugging* CODE

  7. ¯\_(ツ)_/¯

  8. The OS ~= API • Network: bind(), listen(), accept(), read()...

    • Files: open(), read(), write()... • Memory allocation: mmap(), malloc(), brk()? • Threads et al: pthreads_
  9. Mmmm, something’s not quite ok!

  10. 1. My process is kindof stuck for no reason! 2.

    The file it should write is empty! 3. Its socket doesn’t get a single byte! 4. I just run out of FDs, but I was only using one 5. {CPU, disk, memory} usage is too high
  11. Let’s peek into!

  12. 1. Stuck process. Let’s strace!

  13. strace -p <pid>

  14. None
  15. 2. It’s writing to another file! Let’s strace again

  16. It opens the wrong file!!! But I set another path

    in the “WHATEVER_LOGGER” ENV variable...
  17. Let’s into /proc/<pid>/environment

  18. Ooops! The ENV var was not set in the appropriate

  19. /proc/<pid>/<*> is pretty rad Exposes kernel data structures in the

  20. 3. It doesn’t reach a server ngrep to the rescue!

  21. None
  22. None
  23. 4. I run out of FDs asks coworker :D

  24. “Javier, try with lsof”

  25. lsof -p <pid>

  26. Thousands like this!

  27. It ended up being a bug on a Ruby library

    written in C
  28. 5. CPU DISK RAM

  29. perf (A bit out of the scope of this talk.

    Also, I’m even more newbie on this! )
  30. Graphs!

  31. None
  32. Linux 4.1 and above… BPF Compiler Collection (BCC) Basically low

    overhead kernel tracing!
  33. Flamegraphs, heatmaps, histograms etc etc

  34. valgrind --leak-check=yes \ ./maybe_leaking_program

  35. What have I learnt? • Unix tools are awesome! •

    /proc/<pid>/<*>!! • I write lots of bugs! (but hopefully, got a bit better at debugging) • Things are going to fail in every single way they can (and that could be fun!)
  36. Merci!

  37. Interesting links/ bibliography [1] iovisor: https://github.com/iovisor/bcc/ [2] Julia Evans: http://jvns.ca/

    [3] perf: https://perf.wiki.kernel.org/index.php/Main_Page [4] Brendan Gregg: http://www.brendangregg.com/ [6] BPF syntax http://biot.com/capstats/bpf.html [7] Man pages are useful too! (but I do need examples as well :P)