Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Responding @ Scale: osquery for Mass Incident Response and Detection

Responding @ Scale: osquery for Mass Incident Response and Detection

Kevin Thompson & Scott Roberts at BSidesDFW

Ded29c7918dce50c65131df03c769004?s=128

Scott J. Roberts

November 07, 2015
Tweet

Transcript

  1. Responding @ Scale osquery for Mass Incident Detection & Response

  2. Introductions

  3. Kevin Thompson  Incident Responder @ Heroku  @bfist 

    I know more about WWE than all of you...
  4. Heroku  "A company that runs apps for you" 

    Largest PaaS provider  Tens of thousands of Linux servers in production
  5. Scott  DFIR Engineer @ GitHub  @sroberts  I

    know more about StarWars than all of you...
  6. GitHub  "Making it easier to work together than alone"

     Source code hosting & collaboration ( & )  Thousands of Linux servers & hundreds of Macs
  7. We ripped off @Scale... But for a good reason...

  8. A Story in Three Parts  osquery

  9. A Story in Three Parts  osquery  osquery use

  10. A Story in Three Parts  osquery  osquery use

     osquery use at scale
  11.  osquery

  12. What is osquery? Expose system information via "SQL tables" 

    users, groups  installed packages, kernel modules  network communication, file system events Runs scheduled queries of tables
  13. The People  Facebook  @marpaia  @theopolis  Along

    with 63 other people as of this writing...
  14. What problems can osquery solve?  Single tool for finding

    system information  Easy to merge information from information sources  Results presented in multiple parsable format
  15. How do you use osquery  Ad Hoc  Scheduled

     Schedule a query  Collect the logs  Watch for changes
  16. osqueryi  osquery's Run, Evaluate, Print, Loop  Useful for

    testing & one off checks Out of the Box Tools
  17. osqueryi osqueryd  osquery's Daemon Tool  Useful for continuous

    detection  Schedules and runs pre-set queries and writes them to a logger  Takes some setup (we'll get to that) Out of the Box Tools
  18. osqueryi osqueryd osqueryctl  osquery's System Control Tool  Turns

    osqueryd on, gets status, turns it off, etc  Takes care of system specific stuff (like LaunchAgents, which aren't fun) Out of the Box Tools
  19.  osquery use

  20. Getting started w/ osqueryi

  21. Query Syntax SELECT columns FROM table WHERE modifier;

  22. Basic Query select * from apt_sources;

  23. Basic Syntax Breakdown

  24. Basic Syntax Breakdown  Get all the data columns SELECT

    *
  25. Basic Syntax Breakdown  Get all the data columns 

    From the Apt Sources (as in apt-get) Virtual Table SELECT * FROM apt_sources
  26. Basic Query What it means? List all the information about

    all the sources where apt can download & install software select * from apt_sources;
  27. Advanced Query SELECT name, path FROM kernel_extensions WHERE name NOT

    LIKE 'com.apple%';
  28. Advanced Syntax Breakdown

  29. Advanced Syntax Breakdown  Get the name & and path

    data SELECT name, path
  30. Advanced Syntax Breakdown  Get the name & and path

    data  From the Kernel Extensions virtual Table SELECT name, path FROM kernel_extensions
  31. Advanced Syntax Breakdown  Get the name & and path

    data  From the Kernel Extensions virtual Table  Where the name field doesn't start with SELECT name, path FROM kernel_extensions com.apple WHERE name NOT LIKE 'com.apple%'
  32. Advanced Query What it means? Get the name & path

    about any loaded OSX Kernel Extensions that were not created by Apple SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';
  33. Advanced Query

  34. Joins This is sql syntax after all: What it means?

    Get the uid and name from any process with an open listening socket SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid;
  35. Real World Query Examples  Detects LoginWindow Persistence Mechanism 

    Detect RAT used by Hacking Team  Detect the Careto Malware LaunchDaemon select key, subkey, value from preferences where path = '/Library/Preferences/com.apple.loginwindow.plist'; select * from apps where bundle_identifier = 'com.ht.RCSMac' or bundle_package_type like 'OSAX'; select * from launchd where path like '%com.apple.launchport.plist';
  36. Query Writing Philosophy Kevin: Interesting Haystacks Scott: Interesting Needles

  37. Haystack Approach Write generalized queries that grab lots of potentially

    interesting information and sort it out later
  38. Needle Approach Write very specific queries looking only for verified

    indications of compromise and act immediately
  39. Special Capabilities  File Integrity Monitoring  Yara

  40. File Integrity Monitoring  Specify directory paths and wildcards 

    Creates an inotify watcher  Publishes changes to file_events table
  41. File Integrity Monitoring: Paths "file_paths": { "configuration": [ "/etc/%%" ],

    "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ], "ssh_keys": [ "/home/%/.ssh/authorized_keys" ] }
  42. File Integrity Monitoring: Messages osqueryd[22331]: message=Added kernel listener to: /usr/bin/

    osqueryd[22331]: message=Added kernel listener to: /usr/sbin/ osqueryd[22331]: message=Added kernel listener to: /bin/ osqueryd[22331]: message=Added kernel listener to: /sbin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/sbin/ osqueryd[22331]: message=Added kernel listener to: /opt/bin/ osqueryd[22331]: message=Added kernel listener to: /opt/sbin/ osqueryd[22331]: message=Added kernel listener to: /etc/ osqueryd[22331]: message=Added kernel listener to: /home/
  43. File Integrity Monitoring: Query "schedule": { "file_events": { "query": "select

    * from file_events;", "interval": 900, "removed":false } } }
  44. File Integrity Monitoring: Output { "name": "file_events", "hostIdentifier": "ip-172-31-28-89", "calendarTime":

    "Tue Nov 3 19:53:38 2015 UTC", "unixTime": "1446580418", "columns": { "action": "CREATED", "category": "configuration", "md5": "d41d8cd98f00b204e9800998ecf8427e", "sha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "slot": "default", "target_path": "/etc/test", "time": "1446579678", "transaction_id": "0" }, "action": "added" }
  45. Yara The pattern matching swiss knife for malware researchers 

    A pattern matching syntax for identifying malware  osquery can be configured to compare changed files to a set of Yara sigs  This is a beta feature at this time
  46. Yara  Start by setting up the file paths you

    want to watch just as before "file_paths": { "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ] }
  47. Yara  Then tell osquery which files contain Yara sigs

    Note the key, "binaries" has to be one of the file paths "Yara": { "signatures": { "group_1": ["/path/file1.sig", "/path/file2.sig"], "group_2": ["/path/file2.sig", "/path/file3.sig"] }, "binaries" : ["group_1"] }
  48. Yara  This is still beta, Heroku isn't using this

    in prod  Only in exploration right now  No idea what performance impact this has
  49. osquery.conf  JSON configuration file  holds the scheduled queries

     file paths to monitor  read from local filesystem or https osqueryd Configuration
  50. osquery.conf osquery.flags  configures osquery administration  where do logs

    go  where is the pidfile  where to find JSON config file osqueryd Configuration
  51. Managing Intelligence with Packs  osquery query packs are groups

    of queries to be added to the osquery schedule  Lets you group queries for easier management and distribution  Basically a specialized osquery.conf
  52. Why Packs?  Make use of  included intelligence &

    management  Share and collaborate on intelligence  Overall easier management without an unwieldy osquery.conf
  53. Building Your Own Packs Basic Template (stolen from osquery.io/docs/packs) Extend

    and add queries to your 's content! { "platform": "darwin", "version": "1.1.17", "queries": { "example_query": { "query": "select * from kernel_extensions;", "interval": "86400", "description": "Identifies a systems kext files", "value": "Kext's are a common OSX malware persistence mechanism" } } }
  54. Using Non-Facebook Packs  Place on file system  Point

    to in  ???  Profit! osquery.config
  55. Performance Impact & Testing heavy queries and how to avoid

    them  query profiler  cgroups
  56. Query Profiler  Python script that reads your config &

    rates queries  1 to 5 rating in CPU, Memory, File I/O.  RequirHow Heroku is Using osquery
  57. cgroups Control Groups: a feature of the Linux kernel 

    LXC uses cgroups to do it's work  Allows you to segment a group of processes into their own process space or file system
  58. cgroups Root cgroup - everything runs here  by default

    everything gets 1000 cpu shares  smallest you can provide is 2 shares  set up during init script
  59. cgroups Create the cgroups for memory and cpu cgcreate -g

    cpu:osquery cgcreate -g cpuacct:osquery cgcreate -g memory:osquery
  60. cgroups Set the cpu and memory limits cgset -r cpu.shares=2

    osquery cgset -r memory.limit_in_bytes=1073741824 cgset -r memory.kmem.limit_in_bytes=104857600 cgset -r memory.kmem.tcp.limit_in_bytes=104857600 cgexec -g cpuset:osquery osqueryd
  61.  use osquery at scale

  62. How Heroku is Using osquery  Only on the servers

     Our environment is very uniform  Look for things that are not uniform  Lookup IOCs just to get a heads up
  63. How GitHub is Using osquery  Very very very work

    in progress  osquery on OSX laptops hunting known IOCs  osquery on Linux servers hunting known IOCs, anomalies, and doing auditing
  64. Deploying osquery on Linux (w/Puppet or Chef) Ask your friendly

    neighborhood devops engineer...
  65. Deploying osquery on OSX  Deploy the osquery package (or

    a custom version)  Set and/or  Start osqueryd  Figure out how to collect logs (or don't!) osquery.conf osquery.flags
  66. Deploying osquery on OSX #!/bin/sh set -e echo "==> Installing

    osquery (This requires the root password...)" curl https://osquery-packages.s3.amazonaws.com/darwin/osquery.pkg > osquery.pkg sudo installer -pkg osquery.pkg -target / echo "==> Setting osquery.conf" sudo mkdir -p /var/osquery/ sudo cp ./osquery.conf.json /var/osquery/osquery.conf echo "==> Setting the osquery.flags" sudo mkdir -p /etc/osquery/ sudo cp ./osquery.flags.txt /etc/osquery/osquery.flags echo "==> Cleaning up" rm osquery.pkg echo "==> Start osquery" sudo osqueryctl start echo "\nNow check out '/var/log/osquery/osqueryd.results.log' for results."
  67. Configuring Many Endpoints  Earlier mentioned that tells where to

    find config file  That config file can come from an https server osquery.flags osqueryd
  68. Introducing Windmill Developed with  by Heroku & GitHub

  69. Windmill  Open source Ruby TLS Configuration Endpoint  Get

    it now on  at heroku/windmill  Organizes endpoints into Configuration Groups  Enables intelligent endpoint management
  70. Windmill: Configuration Groups

  71. Windmill: Configuration Groups

  72. Windmill  Focused on safety  versioned config files 

    canary deployments
  73. Windmill: Canary Deploy

  74. So... What now?  Transport: syslog, logstash forwarder, fluentd, etc

     Analysis: Splunk, ELK, or a SIEM  You have logs now... GO FIND BAD STUFF!!!
  75. Resources  osquery.io, facebook/osquery, & osquery-python  heroku/windmill  blackfist/osq_simulator

  76. Summary

  77.  osquery  osquery use  osquery use at scale

  78. Questions?

  79. Thanks