Security @ Scale 2015: Building Open Source Software for Security

Facebook's osquery is a Linux and OS X intrusion detection and response tool. The underlying infrastructure must test, build, and publish security software be secure by default. This discussion will show how the Facebook security team enabled Github contributors to safely submit C/C++/bash code to its CI and build server. The discussion will include Facebook's CI hardening process and the attack and vulnerability reports the team received through bug bounty targeting CI.

Transcript

3. Security Engineer Javier Marcos de Prado Building Open Source Software for Security
4. Goals Share our experiences building osquery security tool for host intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration
5. “ So why open source? ” Accelerates Innovation We write better software We share our challenges “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed
6. 5,319 of 284,000 followers 1 of 320 repos 496 of 46,000 forks 2,420 of 91,000 commits
7. commit 73a32b June 30, 2014 “Initial commit” today! Sep 7, 2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature A full year of open source activity Building a community around an open source security tool is challenging
8. What is osquery? Explore your operating system using SQL Host visibility motivated by intrusion detection 100% OS API usage, no fork execve
9. Why use SQL? [concept] SELECT pid, name FROM processes
10. Why use SQL? [concept] SELECT pid, name FROM processes [attributes]
11. Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT pid, name FROM processes [attributes]
12. Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT pid, name FROM processes [attributes] JOIN users ON processes.uid = users.uid [join]
13. Why use SQL? WHERE uid != 0 SELECT pid, name, username FROM processes [attribute] JOIN users ON processes.uid = users.uid [join]
14. Over 100 tables to join •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items •acpi_tables •arp_cache •crontab •file_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices
15. osquery component diagram CMake Thrift C++11 Plugins Modules Extensions SQL Config Logging daemon interactive shell RocksDB LLVM
16. osquery needs secure DevOps! Complicated and resource intensive C++ build 231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10
17. Review: Requirements Must build public code, and have public UI Must build C++11 with few dependencies Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access
18. osquery needs secure DevOps! Originally used TravisCI then switched to Jenkins OMG the sky is falling our builds keep OOMing!
19. osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to test” Developer Javier Let the build begin! Code Review!
20. as do most others… In Jenkins we trust Production Infrastructure Vendor Infrastructure 
 Corporate Infrastructure POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges ISPs
 Firewall ISPs
 Firewall
21. but not that much… In Jenkins we trust Vendor Infrastructure ISPs
 Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coffee shops” All metal are Mac Minis running ESX for virtualization licensing
22. In Jenkins we trust Vendor Infrastructure ISPs
 Firewall Ubuntu 12 Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine
23. Subtitle goes here Title Text
24. Subtitle goes here Title Text
25. Subtitle goes here Title Text
26. Subtitle goes here Title Text
27. Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs
28. Jenkins: TLS proxy <VirtualHost *:80> ServerAdmin osquery@fb.com ServerName jenkins.osquery.io Redirect / https://jenkins.osquery.io/ </VirtualHost> <VirtualHost *:443> ServerAdmin osquery@fb.com ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ <VirtualHost *:443> Also see: https://github.com/hardening-io/chef-jenkins-hardening
29. Jenkins: Web endpoint blacklisting <Location /api> Order Deny,Allow Deny from all </Location> <Location /restart> Order Deny,Allow Deny from all </Location> <Location /quietDown> Order Deny,Allow Deny from all </Location> <Location /user/jenkins> Order Deny,Allow Deny from all </Location> <Location /jnlpJars> Order Deny,Allow Deny from all </Location> <Location /script> Order Deny,Allow Deny from all </Location> Reduce risk to Jenkins admins: based on last 3 years of 0-days
30. Subtitle goes here Title Text
31. Jenkins: Proper authorization Github organization controls the API token Matrix-based ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access
32. Jenkins: Require MFA Test User Now multi-factor checks can be done with the Github API
33. Jenkins: CSRF admins and bots Protect the specific set of admins using MFA Require attackers to 
 compromise admin’s 
 laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to 
 https://jenkins-ci.org/
34. Jenkins: SSH MFA There are a few ways to require a MFA token for SSH
35. Github: Pull Request hardening 1 2 3 Use Github API +
 bots to enforce
 license agreements Tag after code
 review
36. Github: Pull Request hardening Do not allow PR to build without
 a code review by admins first This prevents clown-town build host DoSing,
 and attempts to get a trivial reverse shell
 (at least)
38. 1 2 Pull requests Master
39. 1 2 3 Pull requests Master
40. 1 2 3 4 Pull requests Master
41. 1 2 3 4 5 Pull requests Master
42. Doomsday Scenario 1 Innocuous pull request
43. Doomsday Scenario 1 Innocuous pull request 2 “ok to test”
44. Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend
 $ git push -f
45. Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f
46. Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build
47. Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build 6 osquery package
 contains injection
49. Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure
 should not affect package integrity
51. 1 2 Pull requests Master
52. 1 2 3 Pull requests Master
53. 1 2 3 4 Pull requests Master Master
54. 1 2 3 4 5 Pull requests Master Master
55. 1 2 3 4 5 Pull requests Master Master
56. 1 2 3 4 5 Pull requests Master Master
57. 1 2 3 4 5 Pull requests Master Master
58. Isolated signing A compromised build infrastructure
 MUST not affect package integrity
59. osquery on osquery “After hardening and isolating build infrastructure
 log the hell out of it….” Luckily, the tool we are obsessed with keeping safe
 can also help keep its own build infrastructure safe! …so meta!
60. osquery on osquery see docs at https://osquery.io/docs/tables/
61. see docs at https://osquery.io/docs/tables/#processes
62. This should run in all build infra osquery on osquery { "options": { "disable_audit": "false", "audit_allow_config": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }
63. osquery on osquery ELK - Elastic Search - Logstash - Kibana logstash-forwarder logstash-forwarder logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request
 Jenkins Package
 Jenkins AWS Proxy
64. logstash configuration osquery on osquery 01-lumberjack-input.conf: input { lumberjack { port => 5000 type => "logs" ssl_certificate => “/path/to/ file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } }
65. osquery on osquery
66. osquery on osquery Every move you make, every SSH you take…
 ….I’ll be watching you
67. Conclusion Take extreme care of Jenkins Isolate & audit EVERYTHING If you are building tools with CI, try to use Github APIs
68. Thank you! /javuto @javutin