Instrumentation and Tracing of Systems

Precursors of Security and Performance: Instrumentation and Tracing of Systems

Presented at "Security and DevOps" meetup @ ShiftLeftHQ

The talk discusses instrumentation and tracing as the bedrock of observability in modern systems with 2 case studies - Kprobes in Linux kernel and and Linux Security Modules . Strategies for securing modern systems have also been mentioned along with the variables that decide insertion of security

Transcript

1. Precursors of Security and Performance Instrumentation and Tracing of Systems Suchakra Sharma 17th August 2017 Security and DevOps Meetup, Santa Clara
2. Le Plan ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Observability Layers Instrumentation of Systems - Challenges - Techniques Foundations - Performance - Security Security Tooling - Preventive (Isolation) - Passive (Monitoring) - Active (Protection)
3. whoami ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, Polytechnique Montréal - University of Montréal - Loves tracing, security, performance analysis, hardware dev, poutine and samosas - @tuxology
4. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma
5. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Host OS Kernel functions, hardware perf counters
6. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Host OS Guest OS Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs
7. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls
8. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions
9. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls
10. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Logic Subatomic Particles Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls CPU pins EM waves ??
11. Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App
12. Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App Hardware VM App VM Containers App App App App Hardware Container App Container App App
13. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Why - Applications can now assist you in performance and security analysis – Understand program and data flow – Analyze timings and compare executions - Powerful debugging using traces where debugging is prohibitively expensive How - Simple. Insert extra code at desired locations in any layer of the system (app, library, host OS) - Add a printf(). Congrats ☺
14. Tracing 101 Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
15. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
16. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program
17. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
18. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program void set_tire_dim() { tire_dia = 26; tire_width 2; }
19. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
20. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation
21. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation Tracepoints
22. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
23. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
24. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Events
25. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
26. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data
27. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data
28. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Av Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest
29. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Program Flow Av Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest
30. Deep Dive Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
31. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Static Instrumentation - Development time, eg. insert code that takes a timestamp at function entry and saves it - Compile time. Compiler inserts hooks that you can latch onto at runtime Dynamic Instrumentation - Patch a live application, insert your own observation code, let it run – Reliability – Security - Translate code to another form, instrument it, run it synthetically
32. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma bar() foo() baz() call_me_maybe() call_me_maybe() Fill Buffer Collect Data Event Event Event Program Flow, Arguments, Latency Compiled Observation Function
33. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe
34. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe
35. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe JMP back
36. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe CALL pre_handler User handler eBPF program JMP back
37. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation
38. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
39. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
40. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
41. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
42. Other Techniques ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Ftrace - Dynamic (Kernel) - Build kernel functions with mcount (-pg in GCC) - Patch it to NOPs at boot. Add jmp to handler for activated functions - Add hooks, save function arguments, timestamp - Generate function graph Static Tracepoints (Kernel) - trace_* in most kernel functions - Uses TRACE_EVENT based static tracepoints - Well defined kernel trace events, can be attached to perf, Ftrace (Static)
43. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF + Kprobe BPF Code Kprobe Kernel Function trace.bpf LLVM/Clang Perf Buffer bpf() bpf()
44. Case Study ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF + Kprobe - IOVisor BCC – Python, C++, Lua, Go (gobpf) APIs - Compile BPF programs directly via LLVM interface - Helper functions to manage maps, buffers, probes Example from bcc import BPF prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="hello") print "PID MESSAGE" b.trace_print(fmt="{1} {5}") Attach to Kprobe event prog compiled to BPF bytecode Print trace pipe Complete Program trace_fields.py
45. Suchakrapani Datt Sharma eBPF + Uprobes Example bpf_text = """ #include <uapi/linux/ptrace.h> #include <uapi/linux/limits.h> int get_fname(struct pt_regs *ctx) { if (!ctx->si) return 0; char str[NAME_MAX] = {}; bpf_probe_read(&str, sizeof(str), (void *)ctx->si); bpf_trace_printk("%s\\n", &str); return 0; }; """ b = BPF(text=bpf_text) b.attach_uprobe(name="/usr/bin/vim", sym="readfile", fn_name="get_fname") Get 2nd argument Program Excerpt Process Symbol # ./vim-test.py TASK PID FILENAME vim 23707 /tmp/wololo Output Case Study
46. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Linux Security Modules (LSM) - Static Instrumentation in the kernel code - Hooks to attach LSM implementations for defining and inserting MAC policies – SELinux – AppArmor, LandLock LSM (eBPF) open() do_sys_open() do_filp_open() path_openat() .. vfs_open() do_dentry_open() security_file_open() Syscall from userspace LSM call
47. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Policy Load Policy Store LSM HOOKS Compile and Build Binary
48. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Compile and Build Binary
49. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Compile and Build Binary Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Audit Log Deny access type=AVC msg=audit(XXXXX): avc: denied { getattr } for pid=18923 comm="httpd" path="/var/www/html/file42" dev=dm-0 ino=284133
50. Securing Stuff Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.
51. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Preventive (Isolation) - Virtualization (Hypervisor/VMs) – Inherent isolation, by virtue of hardware/software design – Robust, smaller attack surface - Linux Namespaces/Cgroups (Containers) – Isolation by host kernel/userspace support – Isolate resources and groups of processes – Used to define containers - Linux Capabilities: Not just root/non-root now - SECCOMP* (Application) – Can be used to sandbox process and allow/deny syscalls
52. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Passive (Monitoring) - System level logs and audit messages (Auditd) – Get summary of AVC denials/syscalls to keep track of interesting events - Hook to system events such as capability, syscalls custom userspace events - Monitor network events across layers
53. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Active (Protection) - LSM Modules – Protect infrastructure and implement policies – Policies can now be programmable (eBPF) – Support with modern containers (policies for Docker, rkt, K8s) - SECCOMP – Policies for Docker, rkt, runc - Custom application/library instrumentation
54. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer
55. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra
56. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra
57. References ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Links - Kprobes Kernel Docs – https://www.kernel.org/doc/Documentation/kprobes.txt - Secrets of Ftrace Function Tracer (Steven Rostedt) – https://lwn.net/Articles/370423/ - Linux Performance/Tracing (Brendan Gregg & Julia Evans & Honeycomb) – http://www.brendangregg.com/linuxperf.html – https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ - https://honeycomb.io/blog/categories/instrumentation/ - Linux Security/Containers (Jessie Frazelle & Jérôme Petazzoni) – https://blog.jessfraz.com/post/a-rant-on-usable-security/ – https://www.slideshare.net/jpetazzo/anatomy-of-a-container- namespaces-cgroups-some-filesystem-magic-linuxcon) - eBPF/Landlock LSM (IOVisor devs, Mickaël Salaün et al.) – https://landlock-lsm.github.io/linux-doc/landlock- v5/security/landlock/index.html – http://www.brendangregg.com/ebpf.html – http://blogs.microsoft.co.il/sasha/2016/12/23/usdtbpf-tracing-tools-java- python-ruby-node-mysql-postgresql/
58. Fin Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. suchakra@shiftleft.io @tuxology