Host intrusion detection with osquery

Bc60a5fc6a131ea6cfa80e000b40c743?s=47 Mike Arpaia
October 29, 2014

Host intrusion detection with osquery

A critical aspect of maintaining a robust infrastructure security posture is being able to ask low-level question of hosts in your environment. Even on a single host, performing operating system analytics can often be complex, error prone and computationally expensive. This presentation will examine how Facebook is attacking host instrumentation at scale. We’ll discuss how you can use the same software that we use, regardless of your scale or environment, for no cost at all.


Mike Arpaia

October 29, 2014


  1. None
  2. mike arpaia / facebook host intrusion detection with osquery

  3. javier marcos / facebook ted reed / facebook mimeframe /

  4. what’s the problem?

  5. it’s a hard problem we’re all trying to catch attackers

    •insider threats •espionage
 •external threats •APT •hacktivists •mass malware •the list is endless
  6. we need help we’re all deploying tools •many of us

    have too many vendor products •often times, a product solves too narrow of a use-case •new use-case? new vendor •mo’ money and mo’ problems
  7. we live in a windows centric world •more OS X

    laptops •most production infrastructure runs on Linux
 •few are successfully instrumenting their OS X and Linux hosts •how would we solve that problem? but, times are changing
  8. desired properties

  9. simple performant and reliable easy to integrate flexible

  10. simple •no complex coding for users •low-level details should be

    abstracted •easy to use, deploy and maintain
  11. performant and reliable •host degradation is unacceptable •sane resource utilization

    over time
 •company services should not be impacted
 •extensive logging and metrics
  12. easy to integrate •every company has existing infrastructure •distributed configurations

    •real-time logging •data warehousing
 •you should have the option to use existing infrastructure to help power your host instrumentation
  13. flexible •host instrumentation can help solve many problem domains •intrusion

    detection •vulnerability management •reliability •compliance •< insert domain here >
 •having a single solution reduces cognitive overhead and time spent
  14. osquery

  15. SQL for your infrastructure osquery use SQL queries to explore

    OS state •running processes •loaded kernel modules •active network connections •route table •firewall settings •installed software •and more
  16. osqueryi

  17. LaunchDaemons which run a binary at boot

  18. running processes

  19. processes listening on ports

  20. more tables are being written every day many tables are

    available •alf •alf_exceptions •alf_explicit_auths •alf_services •apps •ca_certs •cpuid •etc_hosts •groups •homebrew_packages •interface_addresses •interface_details •kextstat •last •launchd •listening_ports •nvram •osx_version •passwd_changes •processes •routes •suid_bin •time •users
  21. use simple tables, together osquery enables complex analysis by allowing

    users to join and aggregate across several simple tables •simple tables have many advantages •easier to write •easier to maintain •can be used in many contexts
  22. osqueryd

  23. daemon for low-level host monitoring osqueryd know how the results

    of a query change over time •schedule a query on your hosts via a config
 •the daemon takes care of periodically executing your queries •buffers results to disk and generates a log of state changes •logs results for aggregation and analytics
  24. event-based operating system introspection host event pub/sub stream subscribe to

    key OS events to create dynamically growing tables •subscribe to “publishers” •filesystem changes (inotify, FSEvents) •network setting changes (SCNetwork) •query the history of your host, as it evolves
  25. for config distribution and data infrastructure plugin system •simple plugin

    API •specify your plugins at runtime with a command-line flag filesystem http zookeeper configuration filesystem flume scribe logging
  26. #include "osquery/logger/plugin.h" #include <glog/logging.h> namespace osquery { class GlogPlugin :

    public LoggerPlugin { public: Status logString(const std::string& message) { LOG(INFO) << message; return Status(0, "OK"); } }; REGISTER_LOGGER_PLUGIN("glog", std::make_shared<osquery::GlogPlugin>()); } registering a glog plugin
  27. tables

  28. creating tables is easy easily define what your tables “look

    like” in Python and use C++ to implement what a full-table scan would return •the Python is used to generate faster C++ code transparently •you write a single C++ function which implements a full-table scan
  29. None
  30. table_name("time") schema([ Column(name="hour", type="int"), Column(name="minutes", type="int"), Column(name="seconds", type="int"), ]) implementation("time@genTime")

  31. namespace osquery { namespace tables { QueryData genTime() { QueryData

    results; struct tm* now = localtime(time(0)); // this could be done in a loop for many rows Row r; r["hour"] = now->tm_hour; r["minutes"] = now->tm_min; r["seconds"] = now->tm_sec; results.push_back(r); return results; } } }
  32. build and test

  33. cross-platform build infrastructure osquery supports Ubuntu 12.04+, CentOS 6.5 and

    OS X 10.8+ •dev tools include vagrant build VMs for all supported platforms •every commit publicly builds on Ubuntu and OS X via TravisCI •use build VMs to build and package osquery in a way that works for you
  34. cross-platform build infrastructure OS X pkg creation uses homebrew to

    manage dependencies •distribute relocatable homebrew artifacts such that they work with employees existing installation $ ./tools/ -c ~/Desktop/osquery.conf [+] calculating dependency tree [+] copying dependencies - rocksdb (/usr/local/Cellar/rocksdb/3.5) - boost (/usr/local/Cellar/boost/1.56.0) - gflags (/usr/local/Cellar/gflags/2.0) - glog (/usr/local/Cellar/glog/0.3.3) - thrift (/usr/local/Cellar/thrift/0.9.1) - lz4 (/usr/local/Cellar/lz4/r116) - pkg-config (/usr/local/Cellar/pkg-config/0.28) - snappy (/usr/local/Cellar/snappy/1.1.1) [+] copying osquery binaries [+] copying osquery configurations [+] finalizing preinstall and postinstall scripts [+] creating package [+] package created at ~/git/osquery/osqueryd.pkg
  35. many tutorials and guides extensive documentation

  36. open source

  37. all development happens in the open, on GitHub work on

    osquery with us the problem that osquery solves isn't unique to facebook • • this journey is 1% finished: get involved •we’re excited to take on future challenges in the open •let’s build together
  38. what’s next?

  39. contribute and help us build an awesome tool what we’re

    working on there’s a team of great engineers at facebook that are actively working on making osquery awesome for everyone •more tables •ad-hoc remote queries •kernel modules for lower-level behavior monitoring •deep systems visibility
  40. questions