EkoParty 2015: Secure DevOps is possible: How osquery is built

Slide 1

Slide 1 text

Secure DevOps is possible:  How osquery is built Javier Marcos  Facebook @javutin Teddy Reed  Facebook @teddyreedv

Slide 2

Slide 2 text

Goals Share our experiences building osquery security tool for host intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration

Slide 3

Slide 3 text

Motivation We want to build more open source security software and awesome security communities Ask for feedback because security software must be continuously audited …from you!

Slide 4

Slide 4 text

Introductions Facebook defends itself in many ways Platform Hardening Operating system hardening Vulnerability management Configuration auditing External assessments White hat & bug bounty Security culture and awareness Threat intelligence Commercial solutions Incident response Network intrusion detection Anomaly detection Machine learning VPNs Multi-factor Dynamic analysis Mandatory code review

Slide 5

Slide 5 text

Introductions Facebook defends itself in many ways We make heavy use of open source software Security engineers spend time performing analysis or writing integration and automation infrastructure

Slide 6

Slide 6 text

Introductions Facebook defends itself in many ways We apply the same strategies and properties to defending our open source build infrastructure https://ruxcon.org.au/assets/2014/slides/rux+bpx-incident-response.pdf Home-brew Incident Response (Ruxcon 2014)

Slide 7

Slide 7 text

Accelerates Innovation We write better software We share our challenges “ So why open source? ” “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed

Slide 8

Slide 8 text

320 repos Total Facebook open source

Slide 9

Slide 9 text

320 repos 46,000 forks Total Facebook open source

Slide 10

Slide 10 text

320 repos 46,000 forks 91,000 commits Total Facebook open source

Slide 11

Slide 11 text

284,000 followers 320 repos 46,000 forks 91,000 commits Total Facebook open source

Slide 12

Slide 12 text

5,126 of 284,000 followers 1 of 320 repos 462 of 46,000 forks 2,290 91,000 commits

Slide 13

Slide 13 text

Building a community around an open source security tool is challenging commit 73a32b June 30, 2014 “Initial commit” today! Sep 7, 2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature Almost a year of open source activity

Slide 14

Slide 14 text

Facebook’s host intrusion detection agent •https://github.com/facebook/osquery •https://osquery.io •https://osquery.readthedocs.org Explore your operating system using SQL Host visibility motivated by intrusion detection 100% OS API usage, no fork execve What is osquery?

Slide 15

Slide 15 text

OS concepts are shared on Mac, Linux, and Windows most developers and administrators know SQL the “concepts” have attributes: user ids, process ids, descriptors, ports, paths Why use SQL?

Slide 16

Slide 16 text

Why use SQL? [concept] SELECT pid, name, pid FROM processes

Slide 17

Slide 17 text

[concept] Why use SQL? SELECT pid, name, pid FROM processes [attributes]

Slide 18

Slide 18 text

Why use SQL? SELECT pid, name, pid FROM processes WHERE uid != 0 [attributes] [constraints]

Slide 19

Slide 19 text

JOIN users ON processes.uid = users.uid SELECT pid, name, username FROM processes WHERE uid != 0 [join] [attribute] Why use SQL?

Slide 20

Slide 20 text

Over 100 tables to join •acpi_tables •arp_cache •crontab •ﬁle_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items

Slide 21

Slide 21 text

Plugins, extensions, modules, and more namespace osquery { namespace tables { QueryData genTime(QueryContext& ctx) { QueryData results; struct tm* now = localtime(time(0)); Row r; r["hour"] = INTEGER(now->tm_hour); r["minutes"] = INTEGER(now->tm_min); r["seconds"] = INTEGER(now->tm_sec); results.push_back(r); return results; } } } Tooling to allow rapid new table development! Plugins deﬁne conﬁg input and logger output

Slide 22

Slide 22 text

CMake Thrift C++11 Plugins Modules Extensions SQL Conﬁg Logging daemon interactive shell RocksDB LLVM osquery component diagram

Slide 23

Slide 23 text

Core libraries and build CMake Thrift C++11 Plugins Modules Extensions SQL Conﬁg Logging daemon interactive shell RocksDB LLVM

Slide 24

Slide 24 text

CMake Thrift C++11 Plugins Modules Extensions SQL Conﬁg Logging daemon interactive shell RocksDB LLVM Extensible components

Slide 25

Slide 25 text

osquery needs secure DevOps! Complicated and resource intensive C++ build 231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10

Slide 26

Slide 26 text

osquery needs secure DevOps! Originally used TravisCI then switched to Jenkins OMG the sky is falling our builds keep OOMing!

Slide 27

Slide 27 text

osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to test” Developer Javier Let the build begin! Code Review!

Slide 28

Slide 28 text

In Jenkins we trust as do most others… Production Infrastructure Vendor Infrastructure   Corporate Infrastructure POPs  Edges POPs  Edges POPs  Edges POPs  Edges POPs  Edges ISPs  Firewall ISPs  Firewall

Slide 29

Slide 29 text

In Jenkins we trust but not that much… Vendor Infrastructure ISPs  Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coﬀee shops” All metal are Mac Minis running ESX for virtualization licensing

Slide 30

Slide 30 text

In Jenkins we trust Vendor Infrastructure ISPs  Firewall Ubuntu 12 Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Review: Requirements Must build public code, and have public UI Must build C++11 and cannot build in TravisCI Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access

Slide 36

Slide 36 text

Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs

Slide 37

Slide 37 text

Jenkins: TLS proxy ServerAdmin [email protected] ServerName jenkins.osquery.io Redirect / https://jenkins.osquery.io/ ServerAdmin [email protected] ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode Order deny,allow Allow from all ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ Also see: https://github.com/hardening-io/chef-jenkins-hardening

Slide 38

Slide 38 text

Jenkins: Web endpoint blacklisting Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Reduce risk to Jenkins admins: based on last 3 years of 0-days

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Jenkins: Proper authorization Github organization controls the API token Matrix-based ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access

Slide 42

Slide 42 text

Jenkins: Require MFA Test User Now multi-factor checks can be done with the Github API

Slide 43

Slide 43 text

Jenkins: CSRF admins and bots Protect the speciﬁc set of admins using MFA Require attackers to   compromise admin’s   laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to   https://jenkins-ci.org/

Slide 44

Slide 44 text

Jenkins: SSH MFA There are a few ways to require a MFA token for SSH

Slide 45

Slide 45 text

Github: Pull Request hardening 1 2 3 Use Github API +  bots to enforce  license agreements Tag after code  review

Slide 46

Slide 46 text

Github: Pull Request hardening Do not allow PR to build without  a code review by admins ﬁrst This prevents clown-town build host DoSing,  and attempts to get a trivial reverse shell 

Slide 47

Slide 47 text

Doomsday Scenario 1 Innocuous pull request 2 @theopolis: “ok to test” 3 $ git commit --amend  $ git push -f 4 $ git commit --amend  $ git push -f 5 Compromise:  network, master, or   package build 6 osquery-1.5.4.deb  contains injection

Slide 48

Slide 48 text

1 2

Slide 49

Slide 49 text

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Why no shell!!?!?!?!?!?!?!?!?!?!? nooooooo!

Slide 55

Slide 55 text

Why no shell!!?!?!?!?!?!?!?!?!?!? nooooooo!

Slide 56

Slide 56 text

Gist: https://gist.github.com/maus-/e4caf81f4b5c847ab6f4 Wrap shell IO via S3 bucket updates

Slide 57

Slide 57 text

Gist: https://gist.github.com/maus-/e4caf81f4b5c847ab6f4 Wrap shell IO via S3 bucket updates

Slide 58

Slide 58 text

Slide 59

Slide 59 text

On the build host you see: Several concurrent builds  due to the force-update But no evidence of reverse shell!

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

ehhh, not really… Could not compromise Jenkins master from slave  Can perform L2/3 attacks  But at this point 6 is only a compromised credential or  build-slave vulnerability away 5 Compromise:  network, master, or   package build 6 osquery-1.5.4.deb  contains injection

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure  should not aﬀect package integrity

Slide 64

Slide 64 text

Why is this even a problem?  Build hosts should be virtualized and isolated! Doomsday Scenario: Mitigations Flexible and extensible CI means no assumptions about build  this means agent communication over TCP/22 It is easy to allow build hosts to route to the Internet  otherwise we would need to replicate TONs of repos (doable)

Slide 65

Slide 65 text

1 2 3 4 5 Pull requests Master

Slide 66

Slide 66 text

1 2 3 4 5 Master Master Pull requests

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

Isolated signing Resultant binaries from Jenkins are staged for signing

Slide 71

Slide 71 text

osquery on osquery “After hardening and isolating build infrastructure  log the hell out of it….” Luckily, the tool we are obsessed with keeping safe  can also help keep its own build infrastructure safe! …so meta!

Slide 72

Slide 72 text

osquery on osquery see docs at https://osquery.io/docs/tables/

Slide 73

Slide 73 text

see docs at https://osquery.io/docs/tables/#processes

Slide 74

Slide 74 text

osquery on osquery We recommend and run this on ALL* build infrastructure  except for the chaotic and variable build hosts { "options": { "disable_audit": "false", "audit_allow_conﬁg": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }

Slide 75

Slide 75 text

ELK - Elastic Search - Logstash - Kibana logstash-forwarder logstash-forwarder logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request  Jenkins Package  Jenkins AWS Proxy osquery on osquery

Slide 76

Slide 76 text

log stash conﬁguration 01-lumberjack-input.conf: input { lumberjack { port => 5000 type => "logs" ssl_certificate => “/path/to/file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } osquery on osquery

Slide 77

Slide 77 text

osquery on osquery

Slide 78

Slide 78 text

osquery on osquery Every move you make, every SSH you take…  ….I’ll be watching you

Slide 79

Slide 79 text

Conclusion Take extreme care of Jenkins Isolate & audit everything If you are building tools with CI try to use Github APIs Let us know if you are interested in assessing Jenkins or our build infrastructure! https://facebook.com/whitehat (get paid too)

Slide 80

Slide 80 text

Muchas Gracias! Questions? Javier Marcos  Facebook @javutin Teddy Reed  Facebook @teddyreedv