Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Responding @ Scale: osquery for Mass Incident Response and Detection

Responding @ Scale: osquery for Mass Incident Response and Detection

Kevin Thompson & Scott Roberts at BSidesDFW

Scott J. Roberts

November 07, 2015

More Decks by Scott J. Roberts

Other Decks in Technology


  1. Kevin Thompson  Incident Responder @ Heroku  @bfist 

    I know more about WWE than all of you...
  2. Heroku  "A company that runs apps for you" 

    Largest PaaS provider  Tens of thousands of Linux servers in production
  3. Scott  DFIR Engineer @ GitHub  @sroberts  I

    know more about StarWars than all of you...
  4. GitHub  "Making it easier to work together than alone"

     Source code hosting & collaboration ( & )  Thousands of Linux servers & hundreds of Macs
  5. What is osquery? Expose system information via "SQL tables" 

    users, groups  installed packages, kernel modules  network communication, file system events Runs scheduled queries of tables
  6. The People  Facebook  @marpaia  @theopolis  Along

    with 63 other people as of this writing...
  7. What problems can osquery solve?  Single tool for finding

    system information  Easy to merge information from information sources  Results presented in multiple parsable format
  8. How do you use osquery  Ad Hoc  Scheduled

     Schedule a query  Collect the logs  Watch for changes
  9. osqueryi  osquery's Run, Evaluate, Print, Loop  Useful for

    testing & one off checks Out of the Box Tools
  10. osqueryi osqueryd  osquery's Daemon Tool  Useful for continuous

    detection  Schedules and runs pre-set queries and writes them to a logger  Takes some setup (we'll get to that) Out of the Box Tools
  11. osqueryi osqueryd osqueryctl  osquery's System Control Tool  Turns

    osqueryd on, gets status, turns it off, etc  Takes care of system specific stuff (like LaunchAgents, which aren't fun) Out of the Box Tools
  12. Basic Syntax Breakdown  Get all the data columns 

    From the Apt Sources (as in apt-get) Virtual Table SELECT * FROM apt_sources
  13. Basic Query What it means? List all the information about

    all the sources where apt can download & install software select * from apt_sources;
  14. Advanced Syntax Breakdown  Get the name & and path

    data  From the Kernel Extensions virtual Table SELECT name, path FROM kernel_extensions
  15. Advanced Syntax Breakdown  Get the name & and path

    data  From the Kernel Extensions virtual Table  Where the name field doesn't start with SELECT name, path FROM kernel_extensions com.apple WHERE name NOT LIKE 'com.apple%'
  16. Advanced Query What it means? Get the name & path

    about any loaded OSX Kernel Extensions that were not created by Apple SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';
  17. Joins This is sql syntax after all: What it means?

    Get the uid and name from any process with an open listening socket SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid;
  18. Real World Query Examples  Detects LoginWindow Persistence Mechanism 

    Detect RAT used by Hacking Team  Detect the Careto Malware LaunchDaemon select key, subkey, value from preferences where path = '/Library/Preferences/com.apple.loginwindow.plist'; select * from apps where bundle_identifier = 'com.ht.RCSMac' or bundle_package_type like 'OSAX'; select * from launchd where path like '%com.apple.launchport.plist';
  19. Needle Approach Write very specific queries looking only for verified

    indications of compromise and act immediately
  20. File Integrity Monitoring  Specify directory paths and wildcards 

    Creates an inotify watcher  Publishes changes to file_events table
  21. File Integrity Monitoring: Paths "file_paths": { "configuration": [ "/etc/%%" ],

    "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ], "ssh_keys": [ "/home/%/.ssh/authorized_keys" ] }
  22. File Integrity Monitoring: Messages osqueryd[22331]: message=Added kernel listener to: /usr/bin/

    osqueryd[22331]: message=Added kernel listener to: /usr/sbin/ osqueryd[22331]: message=Added kernel listener to: /bin/ osqueryd[22331]: message=Added kernel listener to: /sbin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/sbin/ osqueryd[22331]: message=Added kernel listener to: /opt/bin/ osqueryd[22331]: message=Added kernel listener to: /opt/sbin/ osqueryd[22331]: message=Added kernel listener to: /etc/ osqueryd[22331]: message=Added kernel listener to: /home/
  23. File Integrity Monitoring: Query "schedule": { "file_events": { "query": "select

    * from file_events;", "interval": 900, "removed":false } } }
  24. File Integrity Monitoring: Output { "name": "file_events", "hostIdentifier": "ip-172-31-28-89", "calendarTime":

    "Tue Nov 3 19:53:38 2015 UTC", "unixTime": "1446580418", "columns": { "action": "CREATED", "category": "configuration", "md5": "d41d8cd98f00b204e9800998ecf8427e", "sha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "slot": "default", "target_path": "/etc/test", "time": "1446579678", "transaction_id": "0" }, "action": "added" }
  25. Yara The pattern matching swiss knife for malware researchers 

    A pattern matching syntax for identifying malware  osquery can be configured to compare changed files to a set of Yara sigs  This is a beta feature at this time
  26. Yara  Start by setting up the file paths you

    want to watch just as before "file_paths": { "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ] }
  27. Yara  Then tell osquery which files contain Yara sigs

    Note the key, "binaries" has to be one of the file paths "Yara": { "signatures": { "group_1": ["/path/file1.sig", "/path/file2.sig"], "group_2": ["/path/file2.sig", "/path/file3.sig"] }, "binaries" : ["group_1"] }
  28. Yara  This is still beta, Heroku isn't using this

    in prod  Only in exploration right now  No idea what performance impact this has
  29. osquery.conf  JSON configuration file  holds the scheduled queries

     file paths to monitor  read from local filesystem or https osqueryd Configuration
  30. osquery.conf osquery.flags  configures osquery administration  where do logs

    go  where is the pidfile  where to find JSON config file osqueryd Configuration
  31. Managing Intelligence with Packs  osquery query packs are groups

    of queries to be added to the osquery schedule  Lets you group queries for easier management and distribution  Basically a specialized osquery.conf
  32. Why Packs?  Make use of  included intelligence &

    management  Share and collaborate on intelligence  Overall easier management without an unwieldy osquery.conf
  33. Building Your Own Packs Basic Template (stolen from osquery.io/docs/packs) Extend

    and add queries to your 's content! { "platform": "darwin", "version": "1.1.17", "queries": { "example_query": { "query": "select * from kernel_extensions;", "interval": "86400", "description": "Identifies a systems kext files", "value": "Kext's are a common OSX malware persistence mechanism" } } }
  34. Using Non-Facebook Packs  Place on file system  Point

    to in  ???  Profit! osquery.config
  35. Query Profiler  Python script that reads your config &

    rates queries  1 to 5 rating in CPU, Memory, File I/O.  RequirHow Heroku is Using osquery
  36. cgroups Control Groups: a feature of the Linux kernel 

    LXC uses cgroups to do it's work  Allows you to segment a group of processes into their own process space or file system
  37. cgroups Root cgroup - everything runs here  by default

    everything gets 1000 cpu shares  smallest you can provide is 2 shares  set up during init script
  38. cgroups Create the cgroups for memory and cpu cgcreate -g

    cpu:osquery cgcreate -g cpuacct:osquery cgcreate -g memory:osquery
  39. cgroups Set the cpu and memory limits cgset -r cpu.shares=2

    osquery cgset -r memory.limit_in_bytes=1073741824 cgset -r memory.kmem.limit_in_bytes=104857600 cgset -r memory.kmem.tcp.limit_in_bytes=104857600 cgexec -g cpuset:osquery osqueryd
  40. How Heroku is Using osquery  Only on the servers

     Our environment is very uniform  Look for things that are not uniform  Lookup IOCs just to get a heads up
  41. How GitHub is Using osquery  Very very very work

    in progress  osquery on OSX laptops hunting known IOCs  osquery on Linux servers hunting known IOCs, anomalies, and doing auditing
  42. Deploying osquery on OSX  Deploy the osquery package (or

    a custom version)  Set and/or  Start osqueryd  Figure out how to collect logs (or don't!) osquery.conf osquery.flags
  43. Deploying osquery on OSX #!/bin/sh set -e echo "==> Installing

    osquery (This requires the root password...)" curl https://osquery-packages.s3.amazonaws.com/darwin/osquery.pkg > osquery.pkg sudo installer -pkg osquery.pkg -target / echo "==> Setting osquery.conf" sudo mkdir -p /var/osquery/ sudo cp ./osquery.conf.json /var/osquery/osquery.conf echo "==> Setting the osquery.flags" sudo mkdir -p /etc/osquery/ sudo cp ./osquery.flags.txt /etc/osquery/osquery.flags echo "==> Cleaning up" rm osquery.pkg echo "==> Start osquery" sudo osqueryctl start echo "\nNow check out '/var/log/osquery/osqueryd.results.log' for results."
  44. Configuring Many Endpoints  Earlier mentioned that tells where to

    find config file  That config file can come from an https server osquery.flags osqueryd
  45. Windmill  Open source Ruby TLS Configuration Endpoint  Get

    it now on  at heroku/windmill  Organizes endpoints into Configuration Groups  Enables intelligent endpoint management
  46. So... What now?  Transport: syslog, logstash forwarder, fluentd, etc

     Analysis: Splunk, ELK, or a SIEM  You have logs now... GO FIND BAD STUFF!!!