How They Hacked Your DevOps - Risk Mitigation in Development Environments

https://www.amirootyet.com/

In the fast-paced DevOps environments, security must keep up. But how? This talk presents the expanded attack surface in CI/CD workflows, highlights frequently observed security risks, and discusses practical mitigation. Audience will walk away with a holistic view of security in DevOps ecosystems.

Transcript

1. How They Hacked Your DevOps Risk Mitigation in Development Environments 02/02/2023 Pranshu Bajpai @amirootyet
2. About PhD, Computer Science, Michigan State University Principal Security Architect, Motorola Solutions Inc Speaker: DEFCON, GrrCon, ToorCon, APWG eCrime, IACP, CascadiaJS, Bsides, IEEE SecDev etc. @amirootyet in/pranshubajpai/ amirootyet.com Research: Malware, Threats, Forensics, Applied Crypto, DevOps, Cloud Security, Data Science
3. Agenda About 1 Common Security Risks 2 DevOps Culture 3 Conclusion 4
4. Disclaimer opinions are my own; not those of my employer tools shown are merely a means to an end @amirootyet
5. DevOps @amirootyet
6. Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline @amirootyet
7. Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline Security?? Security?? Security?? @amirootyet
8. Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery security
9. Pre-Commit Hooks 1 Pre-Build 2 Post-Build 3 Deploy Test Env 4 Deploy Prod 5 Enter DevSecOps Shift Security Left CI/CD Pipeline Secrets Management SCA & SAST DAST IaC Security @amirootyet
10. Common DevOps Security Risks
11. "So... which of our apps use log4j?"
12. Dependency Management A lot of software that exists in our application is external code Vulnerabilities can trickle down via third party libraries Software Composition Analysis (SCA) hunts for vulnerable components Application Internal Code @amirootyet
15. Static Analysis Security Testing White box automated testing of internal code Good for a first pass at security tests to fix obvious potential vulnerabilities False negatives exist + requires some security training Beware of the false positives and alert fatigue!
16. Language-specific SAST Tools
17. Enterprise-grade SAST Tools
18. Dynamic Analysis Security Testing Gray box automated testing of deployed application Able to detect deployment related security risks May require complex configuration False negatives on blind and/or asynchronous bugs
19. DAST Tools
20. DAST Tools
21. IaC Security Create and maintain secure infrastructure Secure configuration of provisioned infrastructure Secure deployment of application within infrastructure
22. Secrets Management Secrets are necessary for authentication Secrets can be exposed throughout the pipeline Exposure can be difficult to detect since secrets come in all shapes, sizes, forms
23. Pre-Commit Hooks to Discover Secrets Talisman Gitleaks
24. Build Systems Ephemeral versus persistent build systems Risk of vulnerable third party extensions Importance of patch management and hardening Least privileges in build environments
25. Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario @amirootyet
26. Build 2 Solarwinds Attack Scenario Is MSBuild.exe running? ElfHash(name) = 0x53D525 Is Orion being built? Extract CLI args from MSBuild.exe Extract directory path of Orion Replace a source code file Perform hash verification check @amirootyet
27. Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario Code Signing @amirootyet
28. Overreaching permissions Lingering stale accounts Local accounts created outside the scope of policies that enforce security practices in the pipelines Peer review all changes Access Control
29. Documentation Encourage and incentivize teams to document Little documentation is better than no documentation Do not seek perfection while documenting
30. You still need documentation, even it's all in your head "I AM the documentation" "But the auditors need documentation" When in doubt, always follow your documentation "Documentation? Ain't no dev got time for that!" @amirootyet
31. Artifacts Signing < 5% are signing code* *Source: Newman, Meyers, and Torres-Arias, “Sigstore: Signing is for Everyone,” ACM CCS, 2022 Not Signing 96% Signing 4%
32. Securing Modern CI/CD Pipeline Source: "Blueprint for building modern, secure software development pipelines", Veracode/Venafi https://github.com/Venafi/blueprint-securesoftwarepipeline
33. Build processes are automated and rapid Log and monitor events in SIEM systems to enable threat detection and response Adversaries persist where logs are absent Collect metrics on vulnerability management Visibility
34. Vulnerability Management Tools
35. DevSecOps Culture
36. Listen. Don't preach. Understand developer perspectives and constraints Suggest viable solutions within those constraints Provide implementation details to enable devsecops Create and support a security champions program Partnership (Security DevOps) Thou shalt sign thy code
37. The Golden Order of Impact Buy-in from stakeholders People Well-defined and easily adopted Process Effective, efficient, and fast execution Tools @amirootyet
38. Cyber Defense Matrix
39. Attack surface is wider than ever. Realign security efforts to incorporate the complexities Implement automated and manual security solutions. Tune, assess, evaluate, relearn, improve, repeat Do not fixate on tools: People > Process > Tools Identify gaps using a visual framework. Separate truth from marketing claims DevSecOps appears to artificially inject "security" into "devops." Security is an organic part of devops! Conclusion
40. Thank You @amirootyet