Instrumentation and Tracing of Systems

Slide 1

Slide 1 text

Precursors of Security and Performance Instrumentation and Tracing of Systems Suchakra Sharma 17th August 2017 Security and DevOps Meetup, Santa Clara

Slide 2

Slide 2 text

Le Plan ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Observability Layers Instrumentation of Systems - Challenges - Techniques Foundations - Performance - Security Security Tooling - Preventive (Isolation) - Passive (Monitoring) - Active (Protection)

Slide 3

Slide 3 text

whoami ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, Polytechnique Montréal - University of Montréal - Loves tracing, security, performance analysis, hardware dev, poutine and samosas - @tuxology

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Host OS Guest OS Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs

Slide 7

Slide 7 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls

Slide 8

Slide 8 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions

Slide 9

Slide 9 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls

Slide 10

Slide 10 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Logic Subatomic Particles Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls CPU pins EM waves ??

Slide 11

Slide 11 text

Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Why - Applications can now assist you in performance and security analysis – Understand program and data flow – Analyze timings and compare executions - Powerful debugging using traces where debugging is prohibitively expensive How - Simple. Insert extra code at desired locations in any layer of the system (app, library, host OS) - Add a printf(). Congrats ☺

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Static Instrumentation - Development time, eg. insert code that takes a timestamp at function entry and saves it - Compile time. Compiler inserts hooks that you can latch onto at runtime Dynamic Instrumentation - Patch a live application, insert your own observation code, let it run – Reliability – Security - Translate code to another form, instrument it, run it synthetically

Slide 32

Slide 32 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma bar() foo() baz() call_me_maybe() call_me_maybe() Fill Buffer Collect Data Event Event Event Program Flow, Arguments, Latency Compiled Observation Function

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe JMP back

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Other Techniques ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Ftrace - Dynamic (Kernel) - Build kernel functions with mcount (-pg in GCC) - Patch it to NOPs at boot. Add jmp to handler for activated functions - Add hooks, save function arguments, timestamp - Generate function graph Static Tracepoints (Kernel) - trace_* in most kernel functions - Uses TRACE_EVENT based static tracepoints - Well defined kernel trace events, can be attached to perf, Ftrace (Static)

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Case Study ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF + Kprobe - IOVisor BCC – Python, C++, Lua, Go (gobpf) APIs - Compile BPF programs directly via LLVM interface - Helper functions to manage maps, buffers, probes Example from bcc import BPF prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="hello") print "PID MESSAGE" b.trace_print(fmt="{1} {5}") Attach to Kprobe event prog compiled to BPF bytecode Print trace pipe Complete Program trace_fields.py

Slide 45

Slide 45 text

Suchakrapani Datt Sharma eBPF + Uprobes Example bpf_text = """ #include #include int get_fname(struct pt_regs *ctx) { if (!ctx->si) return 0; char str[NAME_MAX] = {}; bpf_probe_read(&str, sizeof(str), (void *)ctx->si); bpf_trace_printk("%s\\n", &str); return 0; }; """ b = BPF(text=bpf_text) b.attach_uprobe(name="/usr/bin/vim", sym="readfile", fn_name="get_fname") Get 2nd argument Program Excerpt Process Symbol # ./vim-test.py TASK PID FILENAME vim 23707 /tmp/wololo Output Case Study

Slide 46

Slide 46 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Linux Security Modules (LSM) - Static Instrumentation in the kernel code - Hooks to attach LSM implementations for defining and inserting MAC policies – SELinux – AppArmor, LandLock LSM (eBPF) open() do_sys_open() do_filp_open() path_openat() .. vfs_open() do_dentry_open() security_file_open() Syscall from userspace LSM call

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Compile and Build Binary

Slide 49

Slide 49 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Compile and Build Binary Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Audit Log Deny access type=AVC msg=audit(XXXXX): avc: denied { getattr } for pid=18923 comm="httpd" path="/var/www/html/file42" dev=dm-0 ino=284133

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Preventive (Isolation) - Virtualization (Hypervisor/VMs) – Inherent isolation, by virtue of hardware/software design – Robust, smaller attack surface - Linux Namespaces/Cgroups (Containers) – Isolation by host kernel/userspace support – Isolate resources and groups of processes – Used to define containers - Linux Capabilities: Not just root/non-root now - SECCOMP* (Application) – Can be used to sandbox process and allow/deny syscalls

Slide 52

Slide 52 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Passive (Monitoring) - System level logs and audit messages (Auditd) – Get summary of AVC denials/syscalls to keep track of interesting events - Hook to system events such as capability, syscalls custom userspace events - Monitor network events across layers

Slide 53

Slide 53 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Active (Protection) - LSM Modules – Protect infrastructure and implement policies – Policies can now be programmable (eBPF) – Support with modern containers (policies for Docker, rkt, K8s) - SECCOMP – Policies for Docker, rkt, runc - Custom application/library instrumentation

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra

Slide 56

Slide 56 text

Slide 57

Slide 57 text

References ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Links - Kprobes Kernel Docs – https://www.kernel.org/doc/Documentation/kprobes.txt - Secrets of Ftrace Function Tracer (Steven Rostedt) – https://lwn.net/Articles/370423/ - Linux Performance/Tracing (Brendan Gregg & Julia Evans & Honeycomb) – http://www.brendangregg.com/linuxperf.html – https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ - https://honeycomb.io/blog/categories/instrumentation/ - Linux Security/Containers (Jessie Frazelle & Jérôme Petazzoni) – https://blog.jessfraz.com/post/a-rant-on-usable-security/ – https://www.slideshare.net/jpetazzo/anatomy-of-a-container- namespaces-cgroups-some-filesystem-magic-linuxcon) - eBPF/Landlock LSM (IOVisor devs, Mickaël Salaün et al.) – https://landlock-lsm.github.io/linux-doc/landlock- v5/security/landlock/index.html – http://www.brendangregg.com/ebpf.html – http://blogs.microsoft.co.il/sasha/2016/12/23/usdtbpf-tracing-tools-java- python-ruby-node-mysql-postgresql/

Slide 58

Slide 58 text