Slide 1

Slide 1 text

Precursors of Security and Performance Instrumentation and Tracing of Systems Suchakra Sharma 17th August 2017 Security and DevOps Meetup, Santa Clara

Slide 2

Slide 2 text

Le Plan ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Observability Layers Instrumentation of Systems - Challenges - Techniques Foundations - Performance - Security Security Tooling - Preventive (Isolation) - Passive (Monitoring) - Active (Protection)

Slide 3

Slide 3 text

whoami ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, Polytechnique Montréal - University of Montréal - Loves tracing, security, performance analysis, hardware dev, poutine and samosas - @tuxology

Slide 4

Slide 4 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

Slide 5

Slide 5 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Host OS Kernel functions, hardware perf counters

Slide 6

Slide 6 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Host OS Guest OS Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs

Slide 7

Slide 7 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls

Slide 8

Slide 8 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions

Slide 9

Slide 9 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls

Slide 10

Slide 10 text

Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Logic Subatomic Particles Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls CPU pins EM waves ??

Slide 11

Slide 11 text

Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App

Slide 12

Slide 12 text

Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App Hardware VM App VM Containers App App App App Hardware Container App Container App App

Slide 13

Slide 13 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Why - Applications can now assist you in performance and security analysis – Understand program and data flow – Analyze timings and compare executions - Powerful debugging using traces where debugging is prohibitively expensive How - Simple. Insert extra code at desired locations in any layer of the system (app, library, host OS) - Add a printf(). Congrats ☺

Slide 14

Slide 14 text

Tracing 101 Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 15

Slide 15 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 16

Slide 16 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program

Slide 17

Slide 17 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 18

Slide 18 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program void set_tire_dim() { tire_dia = 26; tire_width 2; }

Slide 19

Slide 19 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 20

Slide 20 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation

Slide 21

Slide 21 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation Tracepoints

Slide 22

Slide 22 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 23

Slide 23 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 24

Slide 24 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Events

Slide 25

Slide 25 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 26

Slide 26 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data

Slide 27

Slide 27 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data

Slide 28

Slide 28 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Av Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest

Slide 29

Slide 29 text

Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Program Flow Av Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest

Slide 30

Slide 30 text

Deep Dive Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 31

Slide 31 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Static Instrumentation - Development time, eg. insert code that takes a timestamp at function entry and saves it - Compile time. Compiler inserts hooks that you can latch onto at runtime Dynamic Instrumentation - Patch a live application, insert your own observation code, let it run – Reliability – Security - Translate code to another form, instrument it, run it synthetically

Slide 32

Slide 32 text

Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma bar() foo() baz() call_me_maybe() call_me_maybe() Fill Buffer Collect Data Event Event Event Program Flow, Arguments, Latency Compiled Observation Function

Slide 33

Slide 33 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe

Slide 34

Slide 34 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe

Slide 35

Slide 35 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe JMP back

Slide 36

Slide 36 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe CALL pre_handler User handler eBPF program JMP back

Slide 37

Slide 37 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation

Slide 38

Slide 38 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back

Slide 39

Slide 39 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back

Slide 40

Slide 40 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back

Slide 41

Slide 41 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back

Slide 42

Slide 42 text

Other Techniques ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Ftrace - Dynamic (Kernel) - Build kernel functions with mcount (-pg in GCC) - Patch it to NOPs at boot. Add jmp to handler for activated functions - Add hooks, save function arguments, timestamp - Generate function graph Static Tracepoints (Kernel) - trace_* in most kernel functions - Uses TRACE_EVENT based static tracepoints - Well defined kernel trace events, can be attached to perf, Ftrace (Static)

Slide 43

Slide 43 text

Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF + Kprobe BPF Code Kprobe Kernel Function trace.bpf LLVM/Clang Perf Buffer bpf() bpf()

Slide 44

Slide 44 text

Case Study ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF + Kprobe - IOVisor BCC – Python, C++, Lua, Go (gobpf) APIs - Compile BPF programs directly via LLVM interface - Helper functions to manage maps, buffers, probes Example from bcc import BPF prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="hello") print "PID MESSAGE" b.trace_print(fmt="{1} {5}") Attach to Kprobe event prog compiled to BPF bytecode Print trace pipe Complete Program trace_fields.py

Slide 45

Slide 45 text

Suchakrapani Datt Sharma eBPF + Uprobes Example bpf_text = """ #include #include int get_fname(struct pt_regs *ctx) { if (!ctx->si) return 0; char str[NAME_MAX] = {}; bpf_probe_read(&str, sizeof(str), (void *)ctx->si); bpf_trace_printk("%s\\n", &str); return 0; }; """ b = BPF(text=bpf_text) b.attach_uprobe(name="/usr/bin/vim", sym="readfile", fn_name="get_fname") Get 2nd argument Program Excerpt Process Symbol # ./vim-test.py TASK PID FILENAME vim 23707 /tmp/wololo Output Case Study

Slide 46

Slide 46 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Linux Security Modules (LSM) - Static Instrumentation in the kernel code - Hooks to attach LSM implementations for defining and inserting MAC policies – SELinux – AppArmor, LandLock LSM (eBPF) open() do_sys_open() do_filp_open() path_openat() .. vfs_open() do_dentry_open() security_file_open() Syscall from userspace LSM call

Slide 47

Slide 47 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Policy Load Policy Store LSM HOOKS Compile and Build Binary

Slide 48

Slide 48 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Compile and Build Binary

Slide 49

Slide 49 text

Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma SELinux Access Vector Cache Security Server Policy Management Server Compile and Build Binary Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Audit Log Deny access type=AVC msg=audit(XXXXX): avc: denied { getattr } for pid=18923 comm="httpd" path="/var/www/html/file42" dev=dm-0 ino=284133

Slide 50

Slide 50 text

Securing Stuff Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

Slide 51

Slide 51 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Preventive (Isolation) - Virtualization (Hypervisor/VMs) – Inherent isolation, by virtue of hardware/software design – Robust, smaller attack surface - Linux Namespaces/Cgroups (Containers) – Isolation by host kernel/userspace support – Isolate resources and groups of processes – Used to define containers - Linux Capabilities: Not just root/non-root now - SECCOMP* (Application) – Can be used to sandbox process and allow/deny syscalls

Slide 52

Slide 52 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Passive (Monitoring) - System level logs and audit messages (Auditd) – Get summary of AVC denials/syscalls to keep track of interesting events - Hook to system events such as capability, syscalls custom userspace events - Monitor network events across layers

Slide 53

Slide 53 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Active (Protection) - LSM Modules – Protect infrastructure and implement policies – Policies can now be programmable (eBPF) – Support with modern containers (policies for Docker, rkt, K8s) - SECCOMP – Policies for Docker, rkt, runc - Custom application/library instrumentation

Slide 54

Slide 54 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer

Slide 55

Slide 55 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra

Slide 56

Slide 56 text

Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra

Slide 57

Slide 57 text

References ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Links - Kprobes Kernel Docs – https://www.kernel.org/doc/Documentation/kprobes.txt - Secrets of Ftrace Function Tracer (Steven Rostedt) – https://lwn.net/Articles/370423/ - Linux Performance/Tracing (Brendan Gregg & Julia Evans & Honeycomb) – http://www.brendangregg.com/linuxperf.html – https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ - https://honeycomb.io/blog/categories/instrumentation/ - Linux Security/Containers (Jessie Frazelle & Jérôme Petazzoni) – https://blog.jessfraz.com/post/a-rant-on-usable-security/ – https://www.slideshare.net/jpetazzo/anatomy-of-a-container- namespaces-cgroups-some-filesystem-magic-linuxcon) - eBPF/Landlock LSM (IOVisor devs, Mickaël Salaün et al.) – https://landlock-lsm.github.io/linux-doc/landlock- v5/security/landlock/index.html – http://www.brendangregg.com/ebpf.html – http://blogs.microsoft.co.il/sasha/2016/12/23/usdtbpf-tracing-tools-java- python-ruby-node-mysql-postgresql/

Slide 58

Slide 58 text

Fin Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. [email protected] @tuxology