Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Host intrusion detection with osquery

Mike Arpaia
October 29, 2014

Host intrusion detection with osquery

A critical aspect of maintaining a robust infrastructure security posture is being able to ask low-level question of hosts in your environment. Even on a single host, performing operating system analytics can often be complex, error prone and computationally expensive. This presentation will examine how Facebook is attacking host instrumentation at scale. We’ll discuss how you can use the same software that we use, regardless of your scale or environment, for no cost at all.

Mike Arpaia

October 29, 2014
Tweet

More Decks by Mike Arpaia

Other Decks in Programming

Transcript

  1. View Slide

  2. mike arpaia / facebook
    host intrusion detection with osquery

    View Slide

  3. javier marcos / facebook
    ted reed / facebook
    mimeframe / facebook

    View Slide

  4. what’s the
    problem?

    View Slide

  5. it’s a hard problem
    we’re all trying to catch attackers
    •insider threats
    •espionage

    •external threats
    •APT
    •hacktivists
    •mass malware
    •the list is endless

    View Slide

  6. we need help
    we’re all deploying tools
    •many of us have too many vendor products
    •often times, a product solves too narrow of a use-case
    •new use-case? new vendor
    •mo’ money and mo’ problems

    View Slide

  7. we live in a windows centric world
    •more OS X laptops
    •most production infrastructure runs on Linux

    •few are successfully instrumenting their OS X and Linux hosts
    •how would we solve that problem?
    but, times are changing

    View Slide

  8. desired
    properties

    View Slide

  9. simple
    performant and reliable
    easy to integrate
    flexible

    View Slide

  10. simple
    •no complex coding for users
    •low-level details should be abstracted
    •easy to use, deploy and maintain

    View Slide

  11. performant and reliable
    •host degradation is unacceptable
    •sane resource utilization over time

    •company services should not be impacted

    •extensive logging and metrics

    View Slide

  12. easy to integrate
    •every company has existing infrastructure
    •distributed configurations
    •real-time logging
    •data warehousing

    •you should have the option to use existing infrastructure to help power
    your host instrumentation

    View Slide

  13. flexible
    •host instrumentation can help solve many problem domains
    •intrusion detection
    •vulnerability management
    •reliability
    •compliance
    •< insert domain here >

    •having a single solution reduces cognitive overhead and time spent

    View Slide

  14. osquery

    View Slide

  15. SQL for your infrastructure
    osquery
    use SQL queries to explore OS state
    •running processes
    •loaded kernel modules
    •active network connections
    •route table
    •firewall settings
    •installed software
    •and more

    View Slide

  16. osqueryi

    View Slide

  17. LaunchDaemons which run a binary at boot

    View Slide

  18. running processes

    View Slide

  19. processes listening on ports

    View Slide

  20. more tables are being written every day
    many tables are available
    •alf
    •alf_exceptions
    •alf_explicit_auths
    •alf_services
    •apps
    •ca_certs
    •cpuid
    •etc_hosts
    •groups
    •homebrew_packages
    •interface_addresses
    •interface_details
    •kextstat
    •last
    •launchd
    •listening_ports
    •nvram
    •osx_version
    •passwd_changes
    •processes
    •routes
    •suid_bin
    •time
    •users

    View Slide

  21. use simple tables, together
    osquery enables complex analysis
    by allowing users to join and
    aggregate across several simple
    tables
    •simple tables have many advantages
    •easier to write
    •easier to maintain
    •can be used in many contexts

    View Slide

  22. osqueryd

    View Slide

  23. daemon for low-level host monitoring
    osqueryd
    know how the results of a query change over time
    •schedule a query on your hosts via a config

    •the daemon takes care of periodically executing your queries
    •buffers results to disk and generates a log of state changes
    •logs results for aggregation and analytics

    View Slide

  24. event-based operating system introspection
    host event pub/sub stream
    subscribe to key OS events to create dynamically growing tables
    •subscribe to “publishers”
    •filesystem changes (inotify, FSEvents)
    •network setting changes (SCNetwork)
    •query the history of your host, as it evolves

    View Slide

  25. for config distribution and data infrastructure
    plugin system
    •simple plugin API
    •specify your plugins at runtime with a command-line flag
    filesystem
    http
    zookeeper
    configuration
    filesystem
    flume
    scribe
    logging

    View Slide

  26. #include "osquery/logger/plugin.h"
    #include
    namespace osquery {
    class GlogPlugin : public LoggerPlugin {
    public:
    Status logString(const std::string& message) {
    LOG(INFO) << message;
    return Status(0, "OK");
    }
    };
    REGISTER_LOGGER_PLUGIN("glog", std::make_shared());
    }
    registering a glog plugin

    View Slide

  27. tables

    View Slide

  28. creating tables is easy
    easily define what your tables “look like” in Python and use C++ to
    implement what a full-table scan would return
    •the Python is used to generate faster C++ code transparently
    •you write a single C++ function which implements a full-table scan

    View Slide

  29. View Slide

  30. table_name("time")
    schema([
    Column(name="hour", type="int"),
    Column(name="minutes", type="int"),
    Column(name="seconds", type="int"),
    ])
    implementation("[email protected]")

    View Slide

  31. namespace osquery {
    namespace tables {
    QueryData genTime() {
    QueryData results;
    struct tm* now = localtime(time(0));
    // this could be done in a loop for many rows
    Row r;
    r["hour"] = now->tm_hour;
    r["minutes"] = now->tm_min;
    r["seconds"] = now->tm_sec;
    results.push_back(r);
    return results;
    }
    }
    }

    View Slide

  32. build and test

    View Slide

  33. cross-platform build infrastructure
    osquery supports Ubuntu 12.04+, CentOS 6.5 and OS X 10.8+
    •dev tools include vagrant build VMs for all supported platforms
    •every commit publicly builds on Ubuntu and OS X via TravisCI
    •use build VMs to build and package osquery in a way that works for you

    View Slide

  34. cross-platform build infrastructure
    OS X pkg creation uses homebrew
    to manage dependencies
    •distribute relocatable homebrew
    artifacts such that they work with
    employees existing installation
    $ ./tools/make_osx_package.sh -c ~/Desktop/osquery.conf
    [+] calculating dependency tree
    [+] copying dependencies
    - rocksdb (/usr/local/Cellar/rocksdb/3.5)
    - boost (/usr/local/Cellar/boost/1.56.0)
    - gflags (/usr/local/Cellar/gflags/2.0)
    - glog (/usr/local/Cellar/glog/0.3.3)
    - thrift (/usr/local/Cellar/thrift/0.9.1)
    - lz4 (/usr/local/Cellar/lz4/r116)
    - pkg-config (/usr/local/Cellar/pkg-config/0.28)
    - snappy (/usr/local/Cellar/snappy/1.1.1)
    [+] copying osquery binaries
    [+] copying osquery configurations
    [+] finalizing preinstall and postinstall scripts
    [+] creating package
    [+] package created at ~/git/osquery/osqueryd.pkg

    View Slide

  35. many tutorials and guides
    extensive documentation

    View Slide

  36. open source

    View Slide

  37. all development happens in the open, on GitHub
    work on osquery with us
    the problem that osquery solves isn't unique to facebook
    •https://github.com/facebook/osquery
    •http://osquery.io
    this journey is 1% finished: get involved
    •we’re excited to take on future challenges in the open
    •let’s build together

    View Slide

  38. what’s next?

    View Slide

  39. contribute and help us build an awesome tool
    what we’re working on
    there’s a team of great engineers at facebook that are actively
    working on making osquery awesome for everyone
    •more tables
    •ad-hoc remote queries
    •kernel modules for lower-level behavior monitoring
    •deep systems visibility

    View Slide

  40. questions
    http://osquery.io

    View Slide