Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Secure DevOps is possible: How osquery is built

Teddy Reed
October 23, 2015

Secure DevOps is possible: How osquery is built

Facebook's osquery is a Linux and OS X intrusion detection and response tool. It supports 10 OS flavors and is continuously built for 8 of those. It is very important that the infrastructure used to test, build, and publish security software be secure itself. This discussion presents how our Security team has enabled any Github contributor to submit C/C++/bash code to our CI and build server, safely. We will guide the audience through our CI hardening process and the attack and vulnerability reports we have received through our bug bounty targeting CI. This includes isolating a Mac Mini fleet of build slaves, not trusting Jenkins as much as possible, automatically building, signing, and publishing packages to AWS S3, doing the same for OS X kernel extension code, adding 2-factor to everything.

Teddy Reed

October 23, 2015
Tweet

More Decks by Teddy Reed

Other Decks in Technology

Transcript

  1. Secure DevOps is possible:
 How osquery is built Javier Marcos


    Facebook @javutin Teddy Reed
 Facebook @teddyreedv
  2. Goals Share our experiences building osquery security tool for host

    intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration
  3. Motivation We want to build more open source security software

    and awesome security communities Ask for feedback because security software must be continuously audited …from you!
  4. Introductions Facebook defends itself in many ways Platform Hardening Operating

    system hardening Vulnerability management Configuration auditing External assessments White hat & bug bounty Security culture and awareness Threat intelligence Commercial solutions Incident response Network intrusion detection Anomaly detection Machine learning VPNs Multi-factor Dynamic analysis Mandatory code review
  5. Introductions Facebook defends itself in many ways We make heavy

    use of open source software Security engineers spend time performing analysis or writing integration and automation infrastructure
  6. Introductions Facebook defends itself in many ways We apply the

    same strategies and properties to defending our open source build infrastructure https://ruxcon.org.au/assets/2014/slides/rux+bpx-incident-response.pdf Home-brew Incident Response (Ruxcon 2014)
  7. Accelerates Innovation We write better software We share our challenges

    “ So why open source? ” “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed
  8. 5,126 of 284,000 followers 1 of 320 repos 462 of

    46,000 forks 2,290 91,000 commits
  9. Building a community around an open source security tool is

    challenging commit 73a32b June 30, 2014 “Initial commit” today! Sep 7, 2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature Almost a year of open source activity
  10. Facebook’s host intrusion detection agent •https://github.com/facebook/osquery •https://osquery.io •https://osquery.readthedocs.org Explore your

    operating system using SQL Host visibility motivated by intrusion detection 100% OS API usage, no fork execve What is osquery?
  11. OS concepts are shared on Mac, Linux, and Windows most

    developers and administrators know SQL the “concepts” have attributes: user ids, process ids, descriptors, ports, paths Why use SQL?
  12. Why use SQL? SELECT pid, name, pid FROM processes WHERE

    uid != 0 [attributes] [constraints]
  13. JOIN users ON processes.uid = users.uid SELECT pid, name, username

    FROM processes WHERE uid != 0 [join] [attribute] Why use SQL?
  14. Over 100 tables to join •acpi_tables •arp_cache •crontab •file_events •kernel_info

    •listening_ports •logged_in_users •mounts •pci_devices •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items
  15. Plugins, extensions, modules, and more namespace osquery { namespace tables

    { QueryData genTime(QueryContext& ctx) { QueryData results; struct tm* now = localtime(time(0)); Row r; r["hour"] = INTEGER(now->tm_hour); r["minutes"] = INTEGER(now->tm_min); r["seconds"] = INTEGER(now->tm_sec); results.push_back(r); return results; } } } Tooling to allow rapid new table development! Plugins define config input and logger output
  16. CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon

    interactive shell RocksDB LLVM osquery component diagram
  17. Core libraries and build CMake Thrift C++11 Plugins Modules Extensions

    SQL Config Logging daemon interactive shell RocksDB LLVM
  18. CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon

    interactive shell RocksDB LLVM Extensible components
  19. osquery needs secure DevOps! Complicated and resource intensive C++ build

    231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10
  20. osquery needs secure DevOps! Originally used TravisCI then switched to

    Jenkins OMG the sky is falling our builds keep OOMing!
  21. osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to

    test” Developer Javier Let the build begin! Code Review!
  22. In Jenkins we trust as do most others… Production Infrastructure

    Vendor Infrastructure 
 Corporate Infrastructure POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges ISPs
 Firewall ISPs
 Firewall
  23. In Jenkins we trust but not that much… Vendor Infrastructure

    ISPs
 Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coffee shops” All metal are Mac Minis running ESX for virtualization licensing
  24. In Jenkins we trust Vendor Infrastructure ISPs
 Firewall Ubuntu 12

    Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine
  25. Review: Requirements Must build public code, and have public UI

    Must build C++11 and cannot build in TravisCI Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access
  26. Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint

    blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs
  27. Jenkins: TLS proxy <VirtualHost *:80> ServerAdmin [email protected] ServerName jenkins.osquery.io Redirect

    / https://jenkins.osquery.io/ </VirtualHost> <VirtualHost *:443> ServerAdmin [email protected] ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ <VirtualHost *:443> Also see: https://github.com/hardening-io/chef-jenkins-hardening
  28. Jenkins: Web endpoint blacklisting <Location /api> Order Deny,Allow Deny from

    all </Location> <Location /restart> Order Deny,Allow Deny from all </Location> <Location /quietDown> Order Deny,Allow Deny from all </Location> <Location /user/jenkins> Order Deny,Allow Deny from all </Location> <Location /jnlpJars> Order Deny,Allow Deny from all </Location> <Location /script> Order Deny,Allow Deny from all </Location> Reduce risk to Jenkins admins: based on last 3 years of 0-days
  29. Jenkins: Proper authorization Github organization controls the API token Matrix-based

    ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access
  30. Jenkins: CSRF admins and bots Protect the specific set of

    admins using MFA Require attackers to 
 compromise admin’s 
 laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to 
 https://jenkins-ci.org/
  31. Github: Pull Request hardening 1 2 3 Use Github API

    +
 bots to enforce
 license agreements Tag after code
 review
  32. Github: Pull Request hardening Do not allow PR to build

    without
 a code review by admins first This prevents clown-town build host DoSing,
 and attempts to get a trivial reverse shell

  33. Doomsday Scenario 1 Innocuous pull request 2 @theopolis: “ok to

    test” 3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build 6 osquery-1.5.4.deb
 contains injection
  34. 1 2

  35. 3

  36. 4

  37. 4

  38. 5

  39. On the build host you see: Several concurrent builds
 due

    to the force-update But no evidence of reverse shell!
  40. ehhh, not really… Could not compromise Jenkins master from slave


    Can perform L2/3 attacks
 But at this point 6 is only a compromised credential or
 build-slave vulnerability away 5 Compromise:
 network, master, or 
 package build 6 osquery-1.5.4.deb
 contains injection
  41. Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation

    Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure
 should not affect package integrity
  42. Why is this even a problem?
 Build hosts should be

    virtualized and isolated! Doomsday Scenario: Mitigations Flexible and extensible CI means no assumptions about build
 this means agent communication over TCP/22 It is easy to allow build hosts to route to the Internet
 otherwise we would need to replicate TONs of repos (doable)
  43. osquery on osquery “After hardening and isolating build infrastructure
 log

    the hell out of it….” Luckily, the tool we are obsessed with keeping safe
 can also help keep its own build infrastructure safe! …so meta!
  44. osquery on osquery We recommend and run this on ALL*

    build infrastructure
 except for the chaotic and variable build hosts { "options": { "disable_audit": "false", "audit_allow_config": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }
  45. ELK - Elastic Search - Logstash - Kibana logstash-forwarder logstash-forwarder

    logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request
 Jenkins Package
 Jenkins AWS Proxy osquery on osquery
  46. log stash configuration 01-lumberjack-input.conf: input { lumberjack { port =>

    5000 type => "logs" ssl_certificate => “/path/to/file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } osquery on osquery
  47. osquery on osquery Every move you make, every SSH you

    take…
 ….I’ll be watching you
  48. Conclusion Take extreme care of Jenkins Isolate & audit everything

    If you are building tools with CI try to use Github APIs Let us know if you are interested in assessing Jenkins or our build infrastructure! https://facebook.com/whitehat (get paid too)