$30 off During Our Annual Pro Sale. View Details »

eBPF & the future of osquery on Linux

eBPF & the future of osquery on Linux

Slides from the talk at osquery@scale 2021.

Zach Wasserman

January 25, 2021
Tweet

More Decks by Zach Wasserman

Other Decks in Programming

Transcript

  1. eBPF and the future of osquery


    on Linux
    Zach Wasserman – CTO, Fleet
    osquery@scale 2021

    View Slide

  2. Who am I
    • Co-creator of osquery


    • Cofounder & CTO of
    FleetDM


    • (Former) Cofounder &
    Principal Engineer of Kolide

    View Slide

  3. Silent Accomplice
    Alessandro Gario


    Senior Software Engineer


    Trail of Bits

    View Slide

  4. Audit & Osquery

    View Slide

  5. Audit & Osquery
    Tables
    • Audit powers many of the event-based tables for osquery on Linux


    • process_events


    • process_file_events


    • socket_events


    • user_events


    • apparmor_events


    • selinux_events

    View Slide

  6. Audit & Osquery
    Con
    f
    i
    guration
    • Base con
    f
    i
    guration to enable audit


    • --disable_audit=false


    • --audit_allow_config=true


    • --audit_persist=true

    View Slide

  7. Audit & Osquery
    Con
    f
    i
    guration
    • Enable each feature separately


    • --audit_allow_apparmor_events


    • --audit_allow_fim_events


    • --audit_allow_fork_process_events


    • --audit_allow_kill_process_events


    • --audit_allow_process_events


    • --audit_allow_selinux_events


    • --audit_allow_sockets


    • --audit_allow_user_events

    View Slide

  8. Audit Drawbacks

    View Slide

  9. View Slide

  10. Audit Drawbacks
    There can be only one
    • Audit’s design allows only a single consumer of the generated events


    • Receiving audit events in osquery means disabling auditd


    • Disable auditd -> No audit events written to
    f
    i
    le


    • Some tools expect to be able to retrieve audit events from
    f
    i
    le!


    Highlander Quote - Linux Audit: Moving Beyond Kernel Namespaces to Audit Container IDs - Richard Guy Briggs (Linux Security Summit 2018)

    View Slide

  11. Audit Drawbacks
    There can be only one
    • Many SELinux tools rely on audit logs in /var/log/audit/audit.log


    • sealert


    • audit2allow

    View Slide

  12. Audit Drawbacks
    Containers
    • Audit does actually work with containers


    • Fixed in Red Hat Bug 893751


    • Audit is not “aware” of containers


    • Lack of namespace information hinders usability in container workloads

    View Slide

  13. eBPF

    View Slide

  14. https://ebpf.io/what-is-ebpf
    eBPF
    Programmable Hooks

    View Slide

  15. eBPF
    Safety
    • Required privileges


    • Program veri
    f
    i
    cation


    • Limited size


    • Limited complexity


    • Bounded loops


    • Controlled memory access

    View Slide

  16. eBPF & Osquery

    View Slide

  17. eBPF & Osquery
    State
    • eBPF functionality released in osquery 4.6.0


    • Built primarily by Alessandro Gario


    • Implemented on top of github.com/trailofbits/ebpfpub

    View Slide

  18. eBPF & Osquery
    Tables
    • bpf_process_events


    • bpf_socket_events


    • …

    View Slide

  19. eBPF & Osquery
    Con
    f
    i
    guration
    • --enable_bpf_events


    • That’s it!

    View Slide

  20. eBPF & Osquery
    Tuning
    • --bpf_buffer_storage_size (default 512)


    • --bpf_perf_event_array_exp (default 10)

    View Slide

  21. eBPF & Osquery
    Compatibility
    • Targeting Kernels 4.18+ (2018)


    • Possible to extend compatibility back to 4.10+ (2017)


    • eBPF Probes are generated at runtime


    • One binary can work on most Kernels

    View Slide

  22. eBPF & Osquery
    Coming soon…
    • Support for correlating BPF events with containers


    • Mapping cgroup_ids to Docker containers


    • process_dns_events

    View Slide

  23. eBPF & Osquery
    Future
    • We now have a pattern for instrumenting nearly anything on Linux


    • System calls


    • Kernel tracepoints


    • User-space function calls


    • These can be dynamically con
    f
    i
    gured at osquery runtime

    View Slide

  24. eBPF & Osquery
    Future - Security
    • Instrument any and all syscalls of interest


    • Track signals sent to processes


    • Kernel module loads/unloads


    • Track LD_PRELOAD

    View Slide

  25. eBPF & Osquery
    Future - Devops/SRE
    • Instrument any and all syscalls of interest(!)


    • Measure latency and resource consumption of OS processes


    • Network stack


    • Filesystem


    • Other I/O


    • Count and measure functions within user-space

    View Slide

  26. eBPF & Osquery
    Future - Imagine
    • Let’s look at the tools of today


    • bpftrace


    • BCC


    • Which of these use cases map well to osquery’s SQL model?


    • How can osquery be useful for shipping the aggregated information from hosts?

    View Slide

  27. Conclusion

    View Slide

  28. Audit & eBPF are both viable
    approaches

    View Slide

  29. eBPF has potential to dramatically increase
    scope of observability with osquery

    View Slide

  30. Thank you


    [email protected]


    @thezachw


    @zwass

    View Slide