DevOps & Security - A match made (and broken) in the cloud

DevOps is the new Agile, allowing organizations to move faster and deploy code quicker. Yet in the quest for continuous delivery, security can fall by the wayside, opening an organization up to data exposure and malicious exploitation. These slides cover current security research into the technology behind DevOps, examples of failures, and how to prevent the same from happening in your organization.

Presented at OWASP PHX on January 28, 2016.

Transcript

1. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 1 DevOps and Security A match made (and broken) in the cloud @sethlaw
2. • Seth Law • VP of Research & Development @ nVisium • Developer/Contributor - Swift.nV, Django.nV, SiRATool, RAFT • Hacker, AppSec Architect, Security Consultant • Soccer Hooligan Introduction
3. • Increase awareness around DevOps + infrastructure security • Provide solutions • Demonstrate impact, regardless of where infrastructure is deployed (internal, external, cloud) Why This Talk
4. TL;DR
5. • Don’t prioritize speed over security • Understand DevOps Tools auth model … or lack of it • Out-of-date or insecure implementation can lead to pawnage • DevOps infrastructure can be dangerous without thought and training around security. It’s ok to teach them :-) TL;DR
7. • Companies can go out of business because of this. • CodeSpaces • DevOps mistakes happen often (examples will be shown) Facts
9. • Searching • Searchcode, GitHub, APIs • Smashing • Jenkins, Redis, memcache • Hardening • AWS Agenda
10. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 10 Searching
11. • Searches for code on: • GitHub - Current leader • BitBucket - The peasant’s GitHub • Google Code - Your dad’s provider • SourceForge - You grandfather’s provider • CodePlex - ¯\_(ϑ)_/¯ • FedoraProject - Hats Projects SearchCode
15. Not so secret
19. • Be careful with external services • KNOW YOUR EXPOSURE • Use these tools defensively as well. SearchCode Lessons Learned
20. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 20 Smashing
21. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 21 Continuous Integration
22. • “Jenkins is [a] … continuous integration and continuous delivery application that increases your productivity. Use Jenkins to build and test your software projects continuously making it easier for developers to integrate changes to the project, and making it easier for users to obtain a fresh build.” Jenkins
25. • Multiple Remote Code Execution vulnerabilities • https:/ /wiki.jenkins-ci.org/display/SECURITY/Home • Advisories are not well publicized • ex: CVE-2015-1814 • Weak coverage with Vulnerability Scanners • API token has the same access as a password Jenkins Issues
26. Jenkins Metasploit Module
27. • If no authentication is required … • Trivial to gain remote code execution via script console • Metasploit to the rescue… • exploit/multi/http/jenkins_script_console • does both authenticated and unauthenticated Jenkins
28. • Groovy Code to run ‘whoami’ Jenkins
31. • Lock down script console access by turning on authentication. • HOWEVER, local auth allows anyone to register as a regular use. • and subsequently access /script… Jenkins
32. • Access to /view/All/newJob == build + run commands Jenkins
34. • Require authentication for everything (duh) • Monitor for security issues and updates • Challenging because full impact of issues can be watered down in the advisory. • Segment Jenkins from Corp • Logical separation by group • Either on single instance or multiple servers • Monitor Jenkins activity and network connections. Jenkins Lessons Learned
35. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 35 In-Memory Databases
36. • Defaults • No encrypted communications • https:/ /github.com/antirez/redis/issues/2178 <- getting closer… • No credentials • Port 6379 (TCP) • Binds on ALL interfaces • TL;DR: Keep this off the interwebs! Redis
38. • OMG RCE • http:/ /benmmurphy.github.io/blog/2015/06/04/redis-eval- lua-sandbox-escape/ Redis
39. • Navigate the DB using redis-cli Redis
40. • Or just use the Redis Desktop Manager Redis
42. • Free & open source, high performance, distributed memory object caching system • Default on TCP port 11211 • No code exec, but fun things get put into memcache memcache
46. • Apply Authentication (strong passwords!) • Bind to localhost, if possible • Enable SSL/TLS, if possible • Segment In-Memory Databases from Corp (and the public in general) • Be aware of the data you put in these databases • No keys, passwords, “secrets” In-Memory Lessons Learned
47. • We do this because: • Makes your environment hard to reach • Prevents abuse • Alerts organization to anomalies AWS Hardening
48. 1. Track Root account login 2. API + Two-Factor Authentication (2FA) 3. Track configuration changes 4. Billing alerts 5. MFA Overview
49. • Step 1: Turn on CloudTrail-Cloudwatch Logs integration • Step 2: Create a metric filter • Step 3: Assign a metric • Step 4: Create an alarm • Step 5: Test the alarm and receive an SNS notification Track Root Account Login
50. • Exact steps (with pics) exist here: • https:/ /blogs.aws.amazon.com/security/post/ Tx3PSPQSN8374D/How-to-Receive-Notifications-When- Your-AWS-Account-s-Root-Access-Keys-Are-Used Track Root Account Login
51. • This is it! At a minimum, apply to administrator and power user group policies. API + 2FA
52. • Go to the Config service and choose resources to track Track Configuration Changes
53. • Or choose to track it all! Track Configuration Changes
54. • Create a bucket, create an SNS topic Track Configuration Changes
55. • Allow the role to be created and you’re all set Track Configuration Changes
56. • Used to prevent abuse or mistakes from costing too much $$ • Configure via CloudWatch (and setup an SNS topic to email out an alert) • Exact steps can be found at: • http:/ /docs.aws.amazon.com/awsaccountbilling/latest/ aboutv2/free-tier-alarms.html Billing Alerts
57. • This is silly easy to setup and everyone should MFA
58. • Next manage the MFA device MFA
59. • Choose a virtual device MFA
60. • Lastly, use Google Authenticator to complete setup. MFA
61. • 5 easy steps • Does the following • Makes your environment harder to reach … for the bad guys • Alerts you to anomalies • Limits what stolen or “otherwise obtained” access keys could be used to do. AWS Recap
62. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 62 Let’s fix this
63. • If you are using Jenkins, make sure it requires authentication • Search github/bitbucket/google code for sensitive information • Update to the latest version of your devops tools • Subscribe to mailing lists of the tools you use • Understand that most DevOps tools take the approach of ”If you can talk to me I trust you” Actions you can take tomorrow
64. • It’s okay to empower DevOps people to do “security” too • Jenkins API key == password (protect them) • Monitor/review code for stored passwords/api keys • Redis: require authentication and setup SSL proxy && upgrade Actions you can take tomorrow (cont’d)
65. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com 65 Conclusion Security is hard, try harder
66. Copyright © 2015 nVisium LLC · 590 Herndon Parkway Suite 120, Herndon VA 20170 · 571.353.7551 · www.nvisium.com Questions? @sethlaw - Seth Law seth@nvisium.com 66