Tweet
Tweet

Slide 1

Slide 1 text

Responding @ Scale osquery for Mass Incident Detection & Response

Slide 2

Slide 2 text

Introductions

Slide 3

Slide 3 text

Kevin Thompson  Incident Responder @ Heroku  @bfist  I know more about WWE than all of you...

Slide 4

Slide 4 text

Heroku  "A company that runs apps for you"  Largest PaaS provider  Tens of thousands of Linux servers in production

Slide 5

Slide 5 text

Scott  DFIR Engineer @ GitHub  @sroberts  I know more about StarWars than all of you...

Slide 6

Slide 6 text

GitHub  "Making it easier to work together than alone"  Source code hosting & collaboration ( & )  Thousands of Linux servers & hundreds of Macs

Slide 7

Slide 7 text

We ripped off @Scale... But for a good reason...

Slide 8

Slide 8 text

A Story in Three Parts  osquery

Slide 9

Slide 9 text

A Story in Three Parts  osquery  osquery use

Slide 10

Slide 10 text

A Story in Three Parts  osquery  osquery use  osquery use at scale

Slide 11

Slide 11 text

 osquery

Slide 12

Slide 12 text

What is osquery? Expose system information via "SQL tables"  users, groups  installed packages, kernel modules  network communication, file system events Runs scheduled queries of tables

Slide 13

Slide 13 text

The People  Facebook  @marpaia  @theopolis  Along with 63 other people as of this writing...

Slide 14

Slide 14 text

What problems can osquery solve?  Single tool for finding system information  Easy to merge information from information sources  Results presented in multiple parsable format

Slide 15

Slide 15 text

How do you use osquery  Ad Hoc  Scheduled  Schedule a query  Collect the logs  Watch for changes

Slide 16

Slide 16 text

osqueryi  osquery's Run, Evaluate, Print, Loop  Useful for testing & one off checks Out of the Box Tools

Slide 17

Slide 17 text

osqueryi osqueryd  osquery's Daemon Tool  Useful for continuous detection  Schedules and runs pre-set queries and writes them to a logger  Takes some setup (we'll get to that) Out of the Box Tools

Slide 18

Slide 18 text

osqueryi osqueryd osqueryctl  osquery's System Control Tool  Turns osqueryd on, gets status, turns it off, etc  Takes care of system specific stuff (like LaunchAgents, which aren't fun) Out of the Box Tools

Slide 19

Slide 19 text

 osquery use

Slide 20

Slide 20 text

Getting started w/ osqueryi

Slide 21

Slide 21 text

Query Syntax SELECT columns FROM table WHERE modifier;

Slide 22

Slide 22 text

Basic Query select * from apt_sources;

Slide 23

Slide 23 text

Basic Syntax Breakdown

Slide 24

Slide 24 text

Basic Syntax Breakdown  Get all the data columns SELECT *

Slide 25

Slide 25 text

Basic Syntax Breakdown  Get all the data columns  From the Apt Sources (as in apt-get) Virtual Table SELECT * FROM apt_sources

Slide 26

Slide 26 text

Basic Query What it means? List all the information about all the sources where apt can download & install software select * from apt_sources;

Slide 27

Slide 27 text

Advanced Query SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';

Slide 28

Slide 28 text

Advanced Syntax Breakdown

Slide 29

Slide 29 text

Advanced Syntax Breakdown  Get the name & and path data SELECT name, path

Slide 30

Slide 30 text

Advanced Syntax Breakdown  Get the name & and path data  From the Kernel Extensions virtual Table SELECT name, path FROM kernel_extensions

Slide 31

Slide 31 text

Advanced Syntax Breakdown  Get the name & and path data  From the Kernel Extensions virtual Table  Where the name field doesn't start with SELECT name, path FROM kernel_extensions com.apple WHERE name NOT LIKE 'com.apple%'

Slide 32

Slide 32 text

Advanced Query What it means? Get the name & path about any loaded OSX Kernel Extensions that were not created by Apple SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';

Slide 33

Slide 33 text

Advanced Query

Slide 34

Slide 34 text

Joins This is sql syntax after all: What it means? Get the uid and name from any process with an open listening socket SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid;

Slide 35

Slide 35 text

Real World Query Examples  Detects LoginWindow Persistence Mechanism  Detect RAT used by Hacking Team  Detect the Careto Malware LaunchDaemon select key, subkey, value from preferences where path = '/Library/Preferences/com.apple.loginwindow.plist'; select * from apps where bundle_identifier = 'com.ht.RCSMac' or bundle_package_type like 'OSAX'; select * from launchd where path like '%com.apple.launchport.plist';

Slide 36

Slide 36 text

Query Writing Philosophy Kevin: Interesting Haystacks Scott: Interesting Needles

Slide 37

Slide 37 text

Haystack Approach Write generalized queries that grab lots of potentially interesting information and sort it out later

Slide 38

Slide 38 text

Needle Approach Write very specific queries looking only for verified indications of compromise and act immediately

Slide 39

Slide 39 text

Special Capabilities  File Integrity Monitoring  Yara

Slide 40

Slide 40 text

File Integrity Monitoring  Specify directory paths and wildcards  Creates an inotify watcher  Publishes changes to file_events table

Slide 41

Slide 41 text

File Integrity Monitoring: Paths "file_paths": { "configuration": [ "/etc/%%" ], "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ], "ssh_keys": [ "/home/%/.ssh/authorized_keys" ] }

Slide 42

Slide 42 text

File Integrity Monitoring: Messages osqueryd[22331]: message=Added kernel listener to: /usr/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/sbin/ osqueryd[22331]: message=Added kernel listener to: /bin/ osqueryd[22331]: message=Added kernel listener to: /sbin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/sbin/ osqueryd[22331]: message=Added kernel listener to: /opt/bin/ osqueryd[22331]: message=Added kernel listener to: /opt/sbin/ osqueryd[22331]: message=Added kernel listener to: /etc/ osqueryd[22331]: message=Added kernel listener to: /home/

Slide 43

Slide 43 text

File Integrity Monitoring: Query "schedule": { "file_events": { "query": "select * from file_events;", "interval": 900, "removed":false } } }

Slide 44

Slide 44 text

File Integrity Monitoring: Output { "name": "file_events", "hostIdentifier": "ip-172-31-28-89", "calendarTime": "Tue Nov 3 19:53:38 2015 UTC", "unixTime": "1446580418", "columns": { "action": "CREATED", "category": "configuration", "md5": "d41d8cd98f00b204e9800998ecf8427e", "sha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "slot": "default", "target_path": "/etc/test", "time": "1446579678", "transaction_id": "0" }, "action": "added" }

Slide 45

Slide 45 text

Yara The pattern matching swiss knife for malware researchers  A pattern matching syntax for identifying malware  osquery can be configured to compare changed files to a set of Yara sigs  This is a beta feature at this time

Slide 46

Slide 46 text

Yara  Start by setting up the file paths you want to watch just as before "file_paths": { "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ] }

Slide 47

Slide 47 text

Yara  Then tell osquery which files contain Yara sigs Note the key, "binaries" has to be one of the file paths "Yara": { "signatures": { "group_1": ["/path/file1.sig", "/path/file2.sig"], "group_2": ["/path/file2.sig", "/path/file3.sig"] }, "binaries" : ["group_1"] }

Slide 48

Slide 48 text

Yara  This is still beta, Heroku isn't using this in prod  Only in exploration right now  No idea what performance impact this has

Slide 49

Slide 49 text

osquery.conf  JSON configuration file  holds the scheduled queries  file paths to monitor  read from local filesystem or https osqueryd Configuration

Slide 50

Slide 50 text

osquery.conf osquery.flags  configures osquery administration  where do logs go  where is the pidfile  where to find JSON config file osqueryd Configuration

Slide 51

Slide 51 text

Managing Intelligence with Packs  osquery query packs are groups of queries to be added to the osquery schedule  Lets you group queries for easier management and distribution  Basically a specialized osquery.conf

Slide 52

Slide 52 text

Why Packs?  Make use of  included intelligence & management  Share and collaborate on intelligence  Overall easier management without an unwieldy osquery.conf

Slide 53

Slide 53 text

Building Your Own Packs Basic Template (stolen from osquery.io/docs/packs) Extend and add queries to your 's content! { "platform": "darwin", "version": "1.1.17", "queries": { "example_query": { "query": "select * from kernel_extensions;", "interval": "86400", "description": "Identifies a systems kext files", "value": "Kext's are a common OSX malware persistence mechanism" } } }

Slide 54

Slide 54 text

Using Non-Facebook Packs  Place on file system  Point to in  ???  Profit! osquery.config

Slide 55

Slide 55 text

Performance Impact & Testing heavy queries and how to avoid them  query profiler  cgroups

Slide 56

Slide 56 text

Query Profiler  Python script that reads your config & rates queries  1 to 5 rating in CPU, Memory, File I/O.  RequirHow Heroku is Using osquery

Slide 57

Slide 57 text

cgroups Control Groups: a feature of the Linux kernel  LXC uses cgroups to do it's work  Allows you to segment a group of processes into their own process space or file system

Slide 58

Slide 58 text

cgroups Root cgroup - everything runs here  by default everything gets 1000 cpu shares  smallest you can provide is 2 shares  set up during init script

Slide 59

Slide 59 text

cgroups Create the cgroups for memory and cpu cgcreate -g cpu:osquery cgcreate -g cpuacct:osquery cgcreate -g memory:osquery

Slide 60

Slide 60 text

cgroups Set the cpu and memory limits cgset -r cpu.shares=2 osquery cgset -r memory.limit_in_bytes=1073741824 cgset -r memory.kmem.limit_in_bytes=104857600 cgset -r memory.kmem.tcp.limit_in_bytes=104857600 cgexec -g cpuset:osquery osqueryd

Slide 61

Slide 61 text

 use osquery at scale

Slide 62

Slide 62 text

How Heroku is Using osquery  Only on the servers  Our environment is very uniform  Look for things that are not uniform  Lookup IOCs just to get a heads up

Slide 63

Slide 63 text

How GitHub is Using osquery  Very very very work in progress  osquery on OSX laptops hunting known IOCs  osquery on Linux servers hunting known IOCs, anomalies, and doing auditing

Slide 64

Slide 64 text

Deploying osquery on Linux (w/Puppet or Chef) Ask your friendly neighborhood devops engineer...

Slide 65

Slide 65 text

Deploying osquery on OSX  Deploy the osquery package (or a custom version)  Set and/or  Start osqueryd  Figure out how to collect logs (or don't!) osquery.conf osquery.flags

Slide 66

Slide 66 text

Deploying osquery on OSX #!/bin/sh set -e echo "==> Installing osquery (This requires the root password...)" curl https://osquery-packages.s3.amazonaws.com/darwin/osquery.pkg > osquery.pkg sudo installer -pkg osquery.pkg -target / echo "==> Setting osquery.conf" sudo mkdir -p /var/osquery/ sudo cp ./osquery.conf.json /var/osquery/osquery.conf echo "==> Setting the osquery.flags" sudo mkdir -p /etc/osquery/ sudo cp ./osquery.flags.txt /etc/osquery/osquery.flags echo "==> Cleaning up" rm osquery.pkg echo "==> Start osquery" sudo osqueryctl start echo "\nNow check out '/var/log/osquery/osqueryd.results.log' for results."

Slide 67

Slide 67 text

Configuring Many Endpoints  Earlier mentioned that tells where to find config file  That config file can come from an https server osquery.flags osqueryd

Slide 68

Slide 68 text

Introducing Windmill Developed with  by Heroku & GitHub

Slide 69

Slide 69 text

Windmill  Open source Ruby TLS Configuration Endpoint  Get it now on  at heroku/windmill  Organizes endpoints into Configuration Groups  Enables intelligent endpoint management

Slide 70

Slide 70 text

Windmill: Configuration Groups

Slide 71

Slide 71 text

Windmill: Configuration Groups

Slide 72

Slide 72 text

Windmill  Focused on safety  versioned config files  canary deployments

Slide 73

Slide 73 text

Windmill: Canary Deploy

Slide 74

Slide 74 text

So... What now?  Transport: syslog, logstash forwarder, fluentd, etc  Analysis: Splunk, ELK, or a SIEM  You have logs now... GO FIND BAD STUFF!!!

Slide 75

Slide 75 text

Resources  osquery.io, facebook/osquery, & osquery-python  heroku/windmill  blackfist/osq_simulator

Slide 76

Slide 76 text

Summary

Slide 77

Slide 77 text

 osquery  osquery use  osquery use at scale

Slide 78

Slide 78 text

Questions?

Slide 79

Slide 79 text

Thanks