Share
Embed
Responding @ Scale: osquery for Mass Incident Response and Detection
Kevin Thompson & Scott Roberts at BSidesDFW
Scott J. Roberts
November 07, 2015
Transcript
79 slides
-
Kevin Thompson Incident Responder @ Heroku @bfist
I know more about WWE than all of you... -
Heroku "A company that runs apps for you"
Largest PaaS provider Tens of thousands of Linux servers in production -
Scott DFIR Engineer @ GitHub @sroberts I
know more about StarWars than all of you... -
GitHub "Making it easier to work together than alone"
Source code hosting & collaboration ( & ) Thousands of Linux servers & hundreds of Macs -
A Story in Three Parts osquery osquery use
osquery use at scale -
What is osquery? Expose system information via "SQL tables"
users, groups installed packages, kernel modules network communication, file system events Runs scheduled queries of tables -
The People Facebook @marpaia @theopolis Along
with 63 other people as of this writing... -
What problems can osquery solve? Single tool for finding
system information Easy to merge information from information sources Results presented in multiple parsable format -
How do you use osquery Ad Hoc Scheduled
Schedule a query Collect the logs Watch for changes -
osqueryi osquery's Run, Evaluate, Print, Loop Useful for
testing & one off checks Out of the Box Tools -
osqueryi osqueryd osquery's Daemon Tool Useful for continuous
detection Schedules and runs pre-set queries and writes them to a logger Takes some setup (we'll get to that) Out of the Box Tools -
osqueryi osqueryd osqueryctl osquery's System Control Tool Turns
osqueryd on, gets status, turns it off, etc Takes care of system specific stuff (like LaunchAgents, which aren't fun) Out of the Box Tools -
Basic Syntax Breakdown Get all the data columns
From the Apt Sources (as in apt-get) Virtual Table SELECT * FROM apt_sources -
Basic Query What it means? List all the information about
all the sources where apt can download & install software select * from apt_sources; -
Advanced Query SELECT name, path FROM kernel_extensions WHERE name NOT
LIKE 'com.apple%'; -
Advanced Syntax Breakdown Get the name & and path
data SELECT name, path -
Advanced Syntax Breakdown Get the name & and path
data From the Kernel Extensions virtual Table SELECT name, path FROM kernel_extensions -
Advanced Syntax Breakdown Get the name & and path
data From the Kernel Extensions virtual Table Where the name field doesn't start with SELECT name, path FROM kernel_extensions com.apple WHERE name NOT LIKE 'com.apple%' -
Advanced Query What it means? Get the name & path
about any loaded OSX Kernel Extensions that were not created by Apple SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%'; -
Joins This is sql syntax after all: What it means?
Get the uid and name from any process with an open listening socket SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid; -
Real World Query Examples Detects LoginWindow Persistence Mechanism
Detect RAT used by Hacking Team Detect the Careto Malware LaunchDaemon select key, subkey, value from preferences where path = '/Library/Preferences/com.apple.loginwindow.plist'; select * from apps where bundle_identifier = 'com.ht.RCSMac' or bundle_package_type like 'OSAX'; select * from launchd where path like '%com.apple.launchport.plist'; -
Haystack Approach Write generalized queries that grab lots of potentially
interesting information and sort it out later -
Needle Approach Write very specific queries looking only for verified
indications of compromise and act immediately -
File Integrity Monitoring Specify directory paths and wildcards
Creates an inotify watcher Publishes changes to file_events table -
File Integrity Monitoring: Paths "file_paths": { "configuration": [ "/etc/%%" ],
"binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ], "ssh_keys": [ "/home/%/.ssh/authorized_keys" ] } -
File Integrity Monitoring: Messages osqueryd[22331]: message=Added kernel listener to: /usr/bin/
osqueryd[22331]: message=Added kernel listener to: /usr/sbin/ osqueryd[22331]: message=Added kernel listener to: /bin/ osqueryd[22331]: message=Added kernel listener to: /sbin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/sbin/ osqueryd[22331]: message=Added kernel listener to: /opt/bin/ osqueryd[22331]: message=Added kernel listener to: /opt/sbin/ osqueryd[22331]: message=Added kernel listener to: /etc/ osqueryd[22331]: message=Added kernel listener to: /home/ -
File Integrity Monitoring: Query "schedule": { "file_events": { "query": "select
* from file_events;", "interval": 900, "removed":false } } } -
File Integrity Monitoring: Output { "name": "file_events", "hostIdentifier": "ip-172-31-28-89", "calendarTime":
"Tue Nov 3 19:53:38 2015 UTC", "unixTime": "1446580418", "columns": { "action": "CREATED", "category": "configuration", "md5": "d41d8cd98f00b204e9800998ecf8427e", "sha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "slot": "default", "target_path": "/etc/test", "time": "1446579678", "transaction_id": "0" }, "action": "added" } -
Yara The pattern matching swiss knife for malware researchers
A pattern matching syntax for identifying malware osquery can be configured to compare changed files to a set of Yara sigs This is a beta feature at this time -
Yara Start by setting up the file paths you
want to watch just as before "file_paths": { "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ] } -
Yara Then tell osquery which files contain Yara sigs
Note the key, "binaries" has to be one of the file paths "Yara": { "signatures": { "group_1": ["/path/file1.sig", "/path/file2.sig"], "group_2": ["/path/file2.sig", "/path/file3.sig"] }, "binaries" : ["group_1"] } -
Yara This is still beta, Heroku isn't using this
in prod Only in exploration right now No idea what performance impact this has -
osquery.conf JSON configuration file holds the scheduled queries
file paths to monitor read from local filesystem or https osqueryd Configuration -
osquery.conf osquery.flags configures osquery administration where do logs
go where is the pidfile where to find JSON config file osqueryd Configuration -
Managing Intelligence with Packs osquery query packs are groups
of queries to be added to the osquery schedule Lets you group queries for easier management and distribution Basically a specialized osquery.conf -
Why Packs? Make use of included intelligence &
management Share and collaborate on intelligence Overall easier management without an unwieldy osquery.conf -
Building Your Own Packs Basic Template (stolen from osquery.io/docs/packs) Extend
and add queries to your 's content! { "platform": "darwin", "version": "1.1.17", "queries": { "example_query": { "query": "select * from kernel_extensions;", "interval": "86400", "description": "Identifies a systems kext files", "value": "Kext's are a common OSX malware persistence mechanism" } } } -
Using Non-Facebook Packs Place on file system Point
to in ??? Profit! osquery.config -
Performance Impact & Testing heavy queries and how to avoid
them query profiler cgroups -
Query Profiler Python script that reads your config &
rates queries 1 to 5 rating in CPU, Memory, File I/O. RequirHow Heroku is Using osquery -
cgroups Control Groups: a feature of the Linux kernel
LXC uses cgroups to do it's work Allows you to segment a group of processes into their own process space or file system -
cgroups Root cgroup - everything runs here by default
everything gets 1000 cpu shares smallest you can provide is 2 shares set up during init script -
cgroups Create the cgroups for memory and cpu cgcreate -g
cpu:osquery cgcreate -g cpuacct:osquery cgcreate -g memory:osquery -
cgroups Set the cpu and memory limits cgset -r cpu.shares=2
osquery cgset -r memory.limit_in_bytes=1073741824 cgset -r memory.kmem.limit_in_bytes=104857600 cgset -r memory.kmem.tcp.limit_in_bytes=104857600 cgexec -g cpuset:osquery osqueryd -
How Heroku is Using osquery Only on the servers
Our environment is very uniform Look for things that are not uniform Lookup IOCs just to get a heads up -
How GitHub is Using osquery Very very very work
in progress osquery on OSX laptops hunting known IOCs osquery on Linux servers hunting known IOCs, anomalies, and doing auditing -
Deploying osquery on Linux (w/Puppet or Chef) Ask your friendly
neighborhood devops engineer... -
Deploying osquery on OSX Deploy the osquery package (or
a custom version) Set and/or Start osqueryd Figure out how to collect logs (or don't!) osquery.conf osquery.flags -
Deploying osquery on OSX #!/bin/sh set -e echo "==> Installing
osquery (This requires the root password...)" curl https://osquery-packages.s3.amazonaws.com/darwin/osquery.pkg > osquery.pkg sudo installer -pkg osquery.pkg -target / echo "==> Setting osquery.conf" sudo mkdir -p /var/osquery/ sudo cp ./osquery.conf.json /var/osquery/osquery.conf echo "==> Setting the osquery.flags" sudo mkdir -p /etc/osquery/ sudo cp ./osquery.flags.txt /etc/osquery/osquery.flags echo "==> Cleaning up" rm osquery.pkg echo "==> Start osquery" sudo osqueryctl start echo "\nNow check out '/var/log/osquery/osqueryd.results.log' for results." -
Configuring Many Endpoints Earlier mentioned that tells where to
find config file That config file can come from an https server osquery.flags osqueryd -
Windmill Open source Ruby TLS Configuration Endpoint Get
it now on at heroku/windmill Organizes endpoints into Configuration Groups Enables intelligent endpoint management -
Windmill Focused on safety versioned config files
canary deployments -
So... What now? Transport: syslog, logstash forwarder, fluentd, etc
Analysis: Splunk, ELK, or a SIEM You have logs now... GO FIND BAD STUFF!!!