Security @ Scale 2015: Building Open Source Software for Security

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Security Engineer Javier Marcos de Prado Building Open Source Software for Security

Slide 4

Slide 4 text

Goals Share our experiences building osquery security tool for host intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration

Slide 5

Slide 5 text

“ So why open source? ” Accelerates Innovation We write better software We share our challenges “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed

Slide 6

Slide 6 text

5,319 of 284,000 followers 1 of 320 repos 496 of 46,000 forks 2,420 of 91,000 commits

Slide 7

Slide 7 text

commit 73a32b June 30, 2014 “Initial commit” today! Sep 7, 2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature A full year of open source activity Building a community around an open source security tool is challenging

Slide 8

Slide 8 text

What is osquery? Explore your operating system using SQL Host visibility motivated by intrusion detection 100% OS API usage, no fork execve

Slide 9

Slide 9 text

Why use SQL? [concept] SELECT pid, name FROM processes

Slide 10

Slide 10 text

Why use SQL? [concept] SELECT pid, name FROM processes [attributes]

Slide 11

Slide 11 text

Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT pid, name FROM processes [attributes]

Slide 12

Slide 12 text

Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT pid, name FROM processes [attributes] JOIN users ON processes.uid = users.uid [join]

Slide 13

Slide 13 text

Why use SQL? WHERE uid != 0 SELECT pid, name, username FROM processes [attribute] JOIN users ON processes.uid = users.uid [join]

Slide 14

Slide 14 text

Over 100 tables to join •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items •acpi_tables •arp_cache •crontab •ﬁle_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices

Slide 15

Slide 15 text

osquery component diagram CMake Thrift C++11 Plugins Modules Extensions SQL Conﬁg Logging daemon interactive shell RocksDB LLVM

Slide 16

Slide 16 text

osquery needs secure DevOps! Complicated and resource intensive C++ build 231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10

Slide 17

Slide 17 text

Review: Requirements Must build public code, and have public UI Must build C++11 with few dependencies Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access

Slide 18

Slide 18 text

osquery needs secure DevOps! Originally used TravisCI then switched to Jenkins OMG the sky is falling our builds keep OOMing!

Slide 19

Slide 19 text

osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to test” Developer Javier Let the build begin! Code Review!

Slide 20

Slide 20 text

as do most others… In Jenkins we trust Production Infrastructure Vendor Infrastructure   Corporate Infrastructure POPs  Edges POPs  Edges POPs  Edges POPs  Edges POPs  Edges ISPs  Firewall ISPs  Firewall

Slide 21

Slide 21 text

but not that much… In Jenkins we trust Vendor Infrastructure ISPs  Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coffee shops” All metal are Mac Minis running ESX for virtualization licensing

Slide 22

Slide 22 text

In Jenkins we trust Vendor Infrastructure ISPs  Firewall Ubuntu 12 Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine

Slide 23

Slide 23 text

Subtitle goes here Title Text

Slide 24

Slide 24 text

Subtitle goes here Title Text

Slide 25

Slide 25 text

Subtitle goes here Title Text

Slide 26

Slide 26 text

Subtitle goes here Title Text

Slide 27

Slide 27 text

Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs

Slide 28

Slide 28 text

Jenkins: TLS proxy ServerAdmin [email protected] ServerName jenkins.osquery.io Redirect / https://jenkins.osquery.io/ ServerAdmin [email protected] ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode Order deny,allow Allow from all ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ Also see: https://github.com/hardening-io/chef-jenkins-hardening

Slide 29

Slide 29 text

Jenkins: Web endpoint blacklisting Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Order Deny,Allow Deny from all Reduce risk to Jenkins admins: based on last 3 years of 0-days

Slide 30

Slide 30 text

Subtitle goes here Title Text

Slide 31

Slide 31 text

Jenkins: Proper authorization Github organization controls the API token Matrix-based ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access

Slide 32

Slide 32 text

Jenkins: Require MFA Test User Now multi-factor checks can be done with the Github API

Slide 33

Slide 33 text

Jenkins: CSRF admins and bots Protect the speciﬁc set of admins using MFA Require attackers to   compromise admin’s   laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to   https://jenkins-ci.org/

Slide 34

Slide 34 text

Jenkins: SSH MFA There are a few ways to require a MFA token for SSH

Slide 35

Slide 35 text

Github: Pull Request hardening 1 2 3 Use Github API +  bots to enforce  license agreements Tag after code  review

Slide 36

Slide 36 text

Github: Pull Request hardening Do not allow PR to build without  a code review by admins ﬁrst This prevents clown-town build host DoSing,  and attempts to get a trivial reverse shell  (at least)

Slide 37

Slide 37 text

Slide 38

Slide 38 text

1 2 Pull requests Master

Slide 39

Slide 39 text

1 2 3 Pull requests Master

Slide 40

Slide 40 text

1 2 3 4 Pull requests Master

Slide 41

Slide 41 text

1 2 3 4 5 Pull requests Master

Slide 42

Slide 42 text

Doomsday Scenario 1 Innocuous pull request

Slide 43

Slide 43 text

Doomsday Scenario 1 Innocuous pull request 2 “ok to test”

Slide 44

Slide 44 text

Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend  $ git push -f

Slide 45

Slide 45 text

Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend  $ git push -f 4 $ git commit --amend  $ git push -f

Slide 46

Slide 46 text

Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend  $ git push -f 4 $ git commit --amend  $ git push -f 5 Compromise:  network, master, or   package build

Slide 47

Slide 47 text

Doomsday Scenario 1 Innocuous pull request 2 “ok to test” 3 $ git commit --amend  $ git push -f 4 $ git commit --amend  $ git push -f 5 Compromise:  network, master, or   package build 6 osquery package  contains injection

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure  should not affect package integrity

Slide 50

Slide 50 text

Slide 51

Slide 51 text

1 2 Pull requests Master

Slide 52

Slide 52 text

1 2 3 Pull requests Master

Slide 53

Slide 53 text

1 2 3 4 Pull requests Master Master

Slide 54

Slide 54 text

1 2 3 4 5 Pull requests Master Master

Slide 55

Slide 55 text

1 2 3 4 5 Pull requests Master Master

Slide 56

Slide 56 text

1 2 3 4 5 Pull requests Master Master

Slide 57

Slide 57 text

1 2 3 4 5 Pull requests Master Master

Slide 58

Slide 58 text

Isolated signing A compromised build infrastructure  MUST not affect package integrity

Slide 59

Slide 59 text

osquery on osquery “After hardening and isolating build infrastructure  log the hell out of it….” Luckily, the tool we are obsessed with keeping safe  can also help keep its own build infrastructure safe! …so meta!

Slide 60

Slide 60 text

osquery on osquery see docs at https://osquery.io/docs/tables/

Slide 61

Slide 61 text

see docs at https://osquery.io/docs/tables/#processes

Slide 62

Slide 62 text

This should run in all build infra osquery on osquery { "options": { "disable_audit": "false", "audit_allow_conﬁg": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }

Slide 63

Slide 63 text

osquery on osquery ELK - Elastic Search - Logstash - Kibana logstash-forwarder logstash-forwarder logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request  Jenkins Package  Jenkins AWS Proxy

Slide 64

Slide 64 text

logstash configuration osquery on osquery 01-lumberjack-input.conf: input { lumberjack { port => 5000 type => "logs" ssl_certificate => “/path/to/ file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } }

Slide 65

Slide 65 text

osquery on osquery

Slide 66

Slide 66 text

osquery on osquery Every move you make, every SSH you take…  ….I’ll be watching you

Slide 67

Slide 67 text

Conclusion Take extreme care of Jenkins Isolate & audit EVERYTHING If you are building tools with CI, try to use Github APIs