D1-I1 Henri Dubois-Ferriere - Cloudy with a Chance of Syscalls

Ignite Talk

TLDR; The Linux system call interface is becoming an increasingly useful instrumentation point for monitoring, troubleshooting, and security. This talk covers why that is happening, what you can get from it, and how it will make your life better.

Once upon a time, you could use packet capture to answer questions like: How many outbound TCP connections in the past hour? Top HTTP requests? Did the latest SQL injection attack happen to my database?

But then the cloud happened, we lost access to a span port, and no longer had a good place to capture traffic. And then containers and their orchestration happened (hello k8s!), and we lost the straightforward mapping between network tuples and applications.

But there’s good news! There’s another instrumentation point which is still largely overlooked, but is even richer in the detail it provides: system calls. By tapping syscalls, you can get pretty much anything you can get from tapping the network, and a lot more. And we’re now getting good support (both kernel and associated tooling) to tap into this instrumentation point.

We’ll give highlights of some of those open-source tools, such as go-audit, sysdig, falco, and maybe even a taste of ebpf. Most importantly, we’ll cover practical examples of using system calls to answer analytics, security, and monitoring questions on realistic environments.

Transcript

1. Cloudy with a Chance of Syscalls DevopsDays Zürich 2017 @henridf 12th Ave Labs From Pixar’s “Partly Cloudy”
2. Cloudy with a Chance of Syscalls DevopsDays Zürich 2017 @henridf 12th Ave Labs From Pixar’s “Partly Cloudy”
3. Instrumentation points galore
4. read send listen write select fork exec nsenter connect open mmap poll
5. SYSTEM CALLS From https://www.slideshare.net/segofensiva/from-kernel-space-to-user-heaven-ndh2k13
7. Y U IGNORE MY CONFIG?
8. $ strace -e open /usr/sbin/nginx open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3 open("/var/log/nginx/error.log", O_WRONLY|O_CREAT| O_APPEND, 0644) = 3 open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 4 open("/sys/devices/system/cpu/online", O_RDONLY| O_CLOEXEC) = 4 open(“/opt/var/weird/nginx.conf", O_RDONLY) = 4 open("/var/log/nginx/access.log", O_WRONLY|O_CREAT| O_APPEND, 0644) = 4 open("/var/log/nginx/error.log", O_WRONLY|O_CREAT| O_APPEND, 0644) = 5
9. “Look at that that nasty-looking outbound connection!” From IBM’s marketing collateral
10. “Can you really tell nastiness just from the packets?”
11. “Hmmm….”
12. “No I can’t! But I can by looking at connect()system calls!
13. Monitoring write performance
15. From https://sysdig.com/blog/50-shades-of-system-calls/
17. As close to the app as you can get… without touching it.
18. Syscall instrumentation is polyglot out of the box
19. At the sweet spot between app and infra From https://www.wsj.com/articles/SB10001424052748704075604575356863337638270
20. @henridf Thank you