EkoParty 2015: Secure DevOps is possible: How osquery is built

Facebook's osquery is a Linux and OS X intrusion detection and response tool. It supports 10 OS flavors and is continuously built for 8 of those. It is very important that the infrastructure used to test, build, and publish security software be secure itself. This discussion presents how our Security team has enabled any Github contributor to submit C/C++/bash code to our CI and build server, safely. We will guide the audience through our CI hardening process and the attack and vulnerability reports we have received through our bug bounty targeting CI. This includes isolating a Mac Mini fleet of build slaves, not trusting Jenkins as much as possible, automatically building, signing, and publishing packages to AWS S3, doing the same for OS X kernel extension code, adding 2-factor to everything.

Transcript

1. Secure DevOps is possible:
 How osquery is built Javier Marcos
 Facebook @javutin Teddy Reed
 Facebook @teddyreedv
2. Goals Share our experiences building osquery security tool for host intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration
3. Motivation We want to build more open source security software and awesome security communities Ask for feedback because security software must be continuously audited …from you!
4. Introductions Facebook defends itself in many ways Platform Hardening Operating system hardening Vulnerability management Configuration auditing External assessments White hat & bug bounty Security culture and awareness Threat intelligence Commercial solutions Incident response Network intrusion detection Anomaly detection Machine learning VPNs Multi-factor Dynamic analysis Mandatory code review
5. Introductions Facebook defends itself in many ways We make heavy use of open source software Security engineers spend time performing analysis or writing integration and automation infrastructure
6. Introductions Facebook defends itself in many ways We apply the same strategies and properties to defending our open source build infrastructure https://ruxcon.org.au/assets/2014/slides/rux+bpx-incident-response.pdf Home-brew Incident Response (Ruxcon 2014)
7. Accelerates Innovation We write better software We share our challenges “ So why open source? ” “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed
8. 320 repos Total Facebook open source
9. 320 repos 46,000 forks Total Facebook open source
10. 320 repos 46,000 forks 91,000 commits Total Facebook open source
11. 284,000 followers 320 repos 46,000 forks 91,000 commits Total Facebook open source
12. 5,126 of 284,000 followers 1 of 320 repos 462 of 46,000 forks 2,290 91,000 commits
13. Building a community around an open source security tool is challenging commit 73a32b June 30, 2014 “Initial commit” today! Sep 7, 2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature Almost a year of open source activity
14. Facebook’s host intrusion detection agent •https://github.com/facebook/osquery •https://osquery.io •https://osquery.readthedocs.org Explore your operating system using SQL Host visibility motivated by intrusion detection 100% OS API usage, no fork execve What is osquery?
15. OS concepts are shared on Mac, Linux, and Windows most developers and administrators know SQL the “concepts” have attributes: user ids, process ids, descriptors, ports, paths Why use SQL?
16. Why use SQL? [concept] SELECT pid, name, pid FROM processes
17. [concept] Why use SQL? SELECT pid, name, pid FROM processes [attributes]
18. Why use SQL? SELECT pid, name, pid FROM processes WHERE uid != 0 [attributes] [constraints]
19. JOIN users ON processes.uid = users.uid SELECT pid, name, username FROM processes WHERE uid != 0 [join] [attribute] Why use SQL?
20. Over 100 tables to join •acpi_tables •arp_cache •crontab •file_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items
21. Plugins, extensions, modules, and more namespace osquery { namespace tables { QueryData genTime(QueryContext& ctx) { QueryData results; struct tm* now = localtime(time(0)); Row r; r["hour"] = INTEGER(now->tm_hour); r["minutes"] = INTEGER(now->tm_min); r["seconds"] = INTEGER(now->tm_sec); results.push_back(r); return results; } } } Tooling to allow rapid new table development! Plugins define config input and logger output
22. CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon interactive shell RocksDB LLVM osquery component diagram
23. Core libraries and build CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon interactive shell RocksDB LLVM
24. CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon interactive shell RocksDB LLVM Extensible components
25. osquery needs secure DevOps! Complicated and resource intensive C++ build 231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10
26. osquery needs secure DevOps! Originally used TravisCI then switched to Jenkins OMG the sky is falling our builds keep OOMing!
27. osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to test” Developer Javier Let the build begin! Code Review!
28. In Jenkins we trust as do most others… Production Infrastructure Vendor Infrastructure 
 Corporate Infrastructure POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges ISPs
 Firewall ISPs
 Firewall
29. In Jenkins we trust but not that much… Vendor Infrastructure ISPs
 Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coffee shops” All metal are Mac Minis running ESX for virtualization licensing
30. In Jenkins we trust Vendor Infrastructure ISPs
 Firewall Ubuntu 12 Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine
35. Review: Requirements Must build public code, and have public UI Must build C++11 and cannot build in TravisCI Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access
36. Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs
37. Jenkins: TLS proxy <VirtualHost *:80> ServerAdmin osquery@fb.com ServerName jenkins.osquery.io Redirect / https://jenkins.osquery.io/ </VirtualHost> <VirtualHost *:443> ServerAdmin osquery@fb.com ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ <VirtualHost *:443> Also see: https://github.com/hardening-io/chef-jenkins-hardening
38. Jenkins: Web endpoint blacklisting <Location /api> Order Deny,Allow Deny from all </Location> <Location /restart> Order Deny,Allow Deny from all </Location> <Location /quietDown> Order Deny,Allow Deny from all </Location> <Location /user/jenkins> Order Deny,Allow Deny from all </Location> <Location /jnlpJars> Order Deny,Allow Deny from all </Location> <Location /script> Order Deny,Allow Deny from all </Location> Reduce risk to Jenkins admins: based on last 3 years of 0-days
41. Jenkins: Proper authorization Github organization controls the API token Matrix-based ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access
42. Jenkins: Require MFA Test User Now multi-factor checks can be done with the Github API
43. Jenkins: CSRF admins and bots Protect the specific set of admins using MFA Require attackers to 
 compromise admin’s 
 laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to 
 https://jenkins-ci.org/
44. Jenkins: SSH MFA There are a few ways to require a MFA token for SSH
45. Github: Pull Request hardening 1 2 3 Use Github API +
 bots to enforce
 license agreements Tag after code
 review
46. Github: Pull Request hardening Do not allow PR to build without
 a code review by admins first This prevents clown-town build host DoSing,
 and attempts to get a trivial reverse shell

47. Doomsday Scenario 1 Innocuous pull request 2 @theopolis: “ok to test” 3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build 6 osquery-1.5.4.deb
 contains injection
48. 1 2
54. Why no shell!!?!?!?!?!?!?!?!?!?!? nooooooo!
55. Why no shell!!?!?!?!?!?!?!?!?!?!? nooooooo!
56. Gist: https://gist.github.com/maus-/e4caf81f4b5c847ab6f4 Wrap shell IO via S3 bucket updates
57. Gist: https://gist.github.com/maus-/e4caf81f4b5c847ab6f4 Wrap shell IO via S3 bucket updates
59. On the build host you see: Several concurrent builds
 due to the force-update But no evidence of reverse shell!
61. ehhh, not really… Could not compromise Jenkins master from slave
 Can perform L2/3 attacks
 But at this point 6 is only a compromised credential or
 build-slave vulnerability away 5 Compromise:
 network, master, or 
 package build 6 osquery-1.5.4.deb
 contains injection
63. Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure
 should not affect package integrity
64. Why is this even a problem?
 Build hosts should be virtualized and isolated! Doomsday Scenario: Mitigations Flexible and extensible CI means no assumptions about build
 this means agent communication over TCP/22 It is easy to allow build hosts to route to the Internet
 otherwise we would need to replicate TONs of repos (doable)
65. 1 2 3 4 5 Pull requests Master
66. 1 2 3 4 5 Master Master Pull requests
70. Isolated signing Resultant binaries from Jenkins are staged for signing
71. osquery on osquery “After hardening and isolating build infrastructure
 log the hell out of it….” Luckily, the tool we are obsessed with keeping safe
 can also help keep its own build infrastructure safe! …so meta!
72. osquery on osquery see docs at https://osquery.io/docs/tables/
73. see docs at https://osquery.io/docs/tables/#processes
74. osquery on osquery We recommend and run this on ALL* build infrastructure
 except for the chaotic and variable build hosts { "options": { "disable_audit": "false", "audit_allow_config": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }
75. ELK - Elastic Search - Logstash - Kibana logstash-forwarder logstash-forwarder logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request
 Jenkins Package
 Jenkins AWS Proxy osquery on osquery
76. log stash configuration 01-lumberjack-input.conf: input { lumberjack { port => 5000 type => "logs" ssl_certificate => “/path/to/file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } osquery on osquery
77. osquery on osquery
78. osquery on osquery Every move you make, every SSH you take…
 ….I’ll be watching you
79. Conclusion Take extreme care of Jenkins Isolate & audit everything If you are building tools with CI try to use Github APIs Let us know if you are interested in assessing Jenkins or our build infrastructure! https://facebook.com/whitehat (get paid too)
80. Muchas Gracias! Questions? Javier Marcos
 Facebook @javutin Teddy Reed
 Facebook @teddyreedv