eBPF & the future of osquery on Linux

by Zach Wasserman

Slide 1

Slide 1 text

eBPF and the future of osquery on Linux Zach Wasserman – CTO, Fleet osquery@scale 2021

Slide 2

Slide 2 text

Who am I • Co-creator of osquery • Cofounder & CTO of FleetDM • (Former) Cofounder & Principal Engineer of Kolide

Slide 3

Slide 3 text

Silent Accomplice Alessandro Gario Senior Software Engineer Trail of Bits

Slide 4

Slide 4 text

Audit & Osquery

Slide 5

Slide 5 text

Audit & Osquery Tables • Audit powers many of the event-based tables for osquery on Linux • process_events • process_file_events • socket_events • user_events • apparmor_events • selinux_events

Slide 6

Slide 6 text

Audit & Osquery Con f i guration • Base con f i guration to enable audit • --disable_audit=false • --audit_allow_config=true • --audit_persist=true

Slide 7

Slide 7 text

Audit & Osquery Con f i guration • Enable each feature separately • --audit_allow_apparmor_events • --audit_allow_fim_events • --audit_allow_fork_process_events • --audit_allow_kill_process_events • --audit_allow_process_events • --audit_allow_selinux_events • --audit_allow_sockets • --audit_allow_user_events

Slide 8

Slide 8 text

Audit Drawbacks

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Audit Drawbacks There can be only one • Audit’s design allows only a single consumer of the generated events • Receiving audit events in osquery means disabling auditd • Disable auditd -> No audit events written to f i le • Some tools expect to be able to retrieve audit events from f i le! Highlander Quote - Linux Audit: Moving Beyond Kernel Namespaces to Audit Container IDs - Richard Guy Briggs (Linux Security Summit 2018)

Slide 11

Slide 11 text

Audit Drawbacks There can be only one • Many SELinux tools rely on audit logs in /var/log/audit/audit.log • sealert • audit2allow

Slide 12

Slide 12 text

Audit Drawbacks Containers • Audit does actually work with containers • Fixed in Red Hat Bug 893751 • Audit is not “aware” of containers • Lack of namespace information hinders usability in container workloads

Slide 13

Slide 13 text

eBPF

Slide 14

Slide 14 text

https://ebpf.io/what-is-ebpf eBPF Programmable Hooks

Slide 15

Slide 15 text

eBPF Safety • Required privileges • Program veri f i cation • Limited size • Limited complexity • Bounded loops • Controlled memory access

Slide 16

Slide 16 text

eBPF & Osquery

Slide 17

Slide 17 text

eBPF & Osquery State • eBPF functionality released in osquery 4.6.0 • Built primarily by Alessandro Gario • Implemented on top of github.com/trailofbits/ebpfpub

Slide 18

Slide 18 text

eBPF & Osquery Tables • bpf_process_events • bpf_socket_events • …

Slide 19

Slide 19 text

eBPF & Osquery Con f i guration • --enable_bpf_events • That’s it!

Slide 20

Slide 20 text

eBPF & Osquery Tuning • --bpf_buffer_storage_size (default 512) • --bpf_perf_event_array_exp (default 10)

Slide 21

Slide 21 text

eBPF & Osquery Compatibility • Targeting Kernels 4.18+ (2018) • Possible to extend compatibility back to 4.10+ (2017) • eBPF Probes are generated at runtime • One binary can work on most Kernels

Slide 22

Slide 22 text

eBPF & Osquery Coming soon… • Support for correlating BPF events with containers • Mapping cgroup_ids to Docker containers • process_dns_events

Slide 23

Slide 23 text

eBPF & Osquery Future • We now have a pattern for instrumenting nearly anything on Linux • System calls • Kernel tracepoints • User-space function calls • These can be dynamically con f i gured at osquery runtime

Slide 24

Slide 24 text

eBPF & Osquery Future - Security • Instrument any and all syscalls of interest • Track signals sent to processes • Kernel module loads/unloads • Track LD_PRELOAD

Slide 25

Slide 25 text

eBPF & Osquery Future - Devops/SRE • Instrument any and all syscalls of interest(!) • Measure latency and resource consumption of OS processes • Network stack • Filesystem • Other I/O • Count and measure functions within user-space

Slide 26

Slide 26 text

eBPF & Osquery Future - Imagine • Let’s look at the tools of today • bpftrace • BCC • Which of these use cases map well to osquery’s SQL model? • How can osquery be useful for shipping the aggregated information from hosts?