Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instrumentation and Tracing of Systems

Instrumentation and Tracing of Systems

Precursors of Security and Performance: Instrumentation and Tracing of Systems

Presented at "Security and DevOps" meetup @ ShiftLeftHQ

The talk discusses instrumentation and tracing as the bedrock of observability in modern systems with 2 case studies - Kprobes in Linux kernel and and Linux Security Modules . Strategies for securing modern systems have also been mentioned along with the variables that decide insertion of security

6bade386c277c9ce9bec3ae260951ec6?s=128

Suchakra Sharma

August 17, 2017
Tweet

Transcript

  1. Precursors of Security and Performance Instrumentation and Tracing of Systems

    Suchakra Sharma 17th August 2017 Security and DevOps Meetup, Santa Clara
  2. Le Plan ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Observability Layers

    Instrumentation of Systems - Challenges - Techniques Foundations - Performance - Security Security Tooling - Preventive (Isolation) - Passive (Monitoring) - Active (Protection)
  3. whoami ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Suchakra - Staff

    Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, Polytechnique Montréal - University of Montréal - Loves tracing, security, performance analysis, hardware dev, poutine and samosas - @tuxology
  4. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

  5. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Host

    OS Kernel functions, hardware perf counters
  6. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization

    Containerization Host OS Guest OS Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs
  7. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization

    Containerization Applications Host OS Guest OS Applications Libs Libs Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls
  8. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization

    Containerization Applications Host OS Guest OS Applications Libs Libs Functions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions
  9. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization

    Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls
  10. Observability Layers ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Hardware Virtualization

    Containerization Applications Host OS Guest OS Applications Libs Libs Functions Instructions Logic Subatomic Particles Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls CPU pins EM waves ??
  11. Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges

    - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App
  12. Observing Modern Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Challanges

    - Horizontal spread of services has increased - Apps are distributed across machines and geographies - Visibility across horizontal and vertical layers - Preventive, Monitoring and Enforcing security for cloud-native applications is non-trivial now - Developers need awareness of production setups in the modern world Hardware VM Container App VM VM App App App App Hardware VM App VM Containers App App App App Hardware Container App Container App App
  13. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Why -

    Applications can now assist you in performance and security analysis – Understand program and data flow – Analyze timings and compare executions - Powerful debugging using traces where debugging is prohibitively expensive How - Simple. Insert extra code at desired locations in any layer of the system (app, library, host OS) - Add a printf(). Congrats ☺
  14. Tracing 101 Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  15. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  16. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program

  17. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  18. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Program void set_tire_dim() {

    tire_dia = 26; tire_width 2; }
  19. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  20. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation

  21. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Instrumentation Tracepoints

  22. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  23. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  24. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Events

  25. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  26. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data

  27. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data

  28. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Av

    Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest
  29. Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. Detailed trace data Program

    Flow Av Van Horne Av du Parc Cote des Neiges Rue Rachel Ouest
  30. Deep Dive Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  31. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Static Instrumentation

    - Development time, eg. insert code that takes a timestamp at function entry and saves it - Compile time. Compiler inserts hooks that you can latch onto at runtime Dynamic Instrumentation - Patch a live application, insert your own observation code, let it run – Reliability – Security - Translate code to another form, instrument it, run it synthetically
  32. Instrumenting Systems ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma bar() foo()

    baz() call_me_maybe() call_me_maybe() Fill Buffer Collect Data Event Event Event Program Flow, Arguments, Latency Compiled Observation Function
  33. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe
  34. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Prepare Kprobe
  35. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe JMP back
  36. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Prepare Kprobe CALL pre_handler User handler eBPF program JMP back
  37. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation
  38. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
  39. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
  40. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
  41. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Kernel Tracing with Kprobes - Dynamic instrumentation based on trap or jump- pad based instrumentation do_sys_open() jmp Restore registers Save registers CALL pre_handler Run Kprobe CALL pre_handler User handler eBPF program JMP back
  42. Other Techniques ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Ftrace -

    Dynamic (Kernel) - Build kernel functions with mcount (-pg in GCC) - Patch it to NOPs at boot. Add jmp to handler for activated functions - Add hooks, save function arguments, timestamp - Generate function graph Static Tracepoints (Kernel) - trace_* in most kernel functions - Uses TRACE_EVENT based static tracepoints - Well defined kernel trace events, can be attached to perf, Ftrace (Static)
  43. Case Study - I ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    eBPF + Kprobe BPF Code Kprobe Kernel Function trace.bpf LLVM/Clang Perf Buffer bpf() bpf()
  44. Case Study ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma eBPF +

    Kprobe - IOVisor BCC – Python, C++, Lua, Go (gobpf) APIs - Compile BPF programs directly via LLVM interface - Helper functions to manage maps, buffers, probes Example from bcc import BPF prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="hello") print "PID MESSAGE" b.trace_print(fmt="{1} {5}") Attach to Kprobe event prog compiled to BPF bytecode Print trace pipe Complete Program trace_fields.py
  45. Suchakrapani Datt Sharma eBPF + Uprobes Example bpf_text = """

    #include <uapi/linux/ptrace.h> #include <uapi/linux/limits.h> int get_fname(struct pt_regs *ctx) { if (!ctx->si) return 0; char str[NAME_MAX] = {}; bpf_probe_read(&str, sizeof(str), (void *)ctx->si); bpf_trace_printk("%s\\n", &str); return 0; }; """ b = BPF(text=bpf_text) b.attach_uprobe(name="/usr/bin/vim", sym="readfile", fn_name="get_fname") Get 2nd argument Program Excerpt Process Symbol # ./vim-test.py TASK PID FILENAME vim 23707 /tmp/wololo Output Case Study
  46. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    Linux Security Modules (LSM) - Static Instrumentation in the kernel code - Hooks to attach LSM implementations for defining and inserting MAC policies – SELinux – AppArmor, LandLock LSM (eBPF) open() do_sys_open() do_filp_open() path_openat() .. vfs_open() do_dentry_open() security_file_open() Syscall from userspace LSM call
  47. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    SELinux Access Vector Cache Security Server Policy Management Server Policy Load Policy Store LSM HOOKS Compile and Build Binary
  48. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    SELinux Access Vector Cache Security Server Policy Management Server Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Compile and Build Binary
  49. Case Study - II ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma

    SELinux Access Vector Cache Security Server Policy Management Server Compile and Build Binary Policy Load Process open(“foo”) Policy Store LSM HOOKS Query policy Audit Log Deny access type=AVC msg=audit(XXXXX): avc: denied { getattr } for pid=18923 comm="httpd" path="/var/www/html/file42" dev=dm-0 ino=284133
  50. Securing Stuff Suchakrapani Datt Sharma ©2017 ShiftLeft Inc.

  51. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Preventive (Isolation)

    - Virtualization (Hypervisor/VMs) – Inherent isolation, by virtue of hardware/software design – Robust, smaller attack surface - Linux Namespaces/Cgroups (Containers) – Isolation by host kernel/userspace support – Isolate resources and groups of processes – Used to define containers - Linux Capabilities: Not just root/non-root now - SECCOMP* (Application) – Can be used to sandbox process and allow/deny syscalls
  52. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Passive (Monitoring)

    - System level logs and audit messages (Auditd) – Get summary of AVC denials/syscalls to keep track of interesting events - Hook to system events such as capability, syscalls custom userspace events - Monitor network events across layers
  53. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Active (Protection)

    - LSM Modules – Protect infrastructure and implement policies – Policies can now be programmable (eBPF) – Support with modern containers (policies for Docker, rkt, K8s) - SECCOMP – Policies for Docker, rkt, runc - Custom application/library instrumentation
  54. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum

    - Two variables for defining and inserting security Software State Computing Layer
  55. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum

    - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra
  56. Securing Strategy ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Insertion Spectrum

    - Two variables for defining and inserting security Software State Computing Layer L I A V/C H L I A V/C H L I A V/C H L I A V/C H L I A V/C H Dev Test Pack Rel Deploy/Run Instructions Applications Libraries Virt/Containers Host Infra
  57. References ©2017 ShiftLeft Inc. Suchakrapani Datt Sharma Links - Kprobes

    Kernel Docs – https://www.kernel.org/doc/Documentation/kprobes.txt - Secrets of Ftrace Function Tracer (Steven Rostedt) – https://lwn.net/Articles/370423/ - Linux Performance/Tracing (Brendan Gregg & Julia Evans & Honeycomb) – http://www.brendangregg.com/linuxperf.html – https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ - https://honeycomb.io/blog/categories/instrumentation/ - Linux Security/Containers (Jessie Frazelle & Jérôme Petazzoni) – https://blog.jessfraz.com/post/a-rant-on-usable-security/ – https://www.slideshare.net/jpetazzo/anatomy-of-a-container- namespaces-cgroups-some-filesystem-magic-linuxcon) - eBPF/Landlock LSM (IOVisor devs, Mickaël Salaün et al.) – https://landlock-lsm.github.io/linux-doc/landlock- v5/security/landlock/index.html – http://www.brendangregg.com/ebpf.html – http://blogs.microsoft.co.il/sasha/2016/12/23/usdtbpf-tracing-tools-java- python-ruby-node-mysql-postgresql/
  58. Fin Suchakrapani Datt Sharma ©2017 ShiftLeft Inc. suchakra@shiftleft.io @tuxology