How They Hacked Your DevOps Risk Mitigation in Development Environments 02/02/2023 Pranshu Bajpai @amirootyet

About PhD, Computer Science, Michigan State University Principal Security Architect, Motorola Solutions Inc Speaker: DEFCON, GrrCon, ToorCon, APWG eCrime, IACP, CascadiaJS, Bsides, IEEE SecDev etc. @amirootyet in/pranshubajpai/ amirootyet.com Research: Malware, Threats, Forensics, Applied Crypto, DevOps, Cloud Security, Data Science

Agenda About 1 Common Security Risks 2 DevOps Culture 3 Conclusion 4

Disclaimer opinions are my own; not those of my employer tools shown are merely a means to an end @amirootyet

DevOps @amirootyet

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline @amirootyet

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline Security?? Security?? Security?? @amirootyet

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery security

Pre-Commit Hooks 1 Pre-Build 2 Post-Build 3 Deploy Test Env 4 Deploy Prod 5 Enter DevSecOps Shift Security Left CI/CD Pipeline Secrets Management SCA & SAST DAST IaC Security @amirootyet

Common DevOps Security Risks

"So... which of our apps use log4j?"

Dependency Management A lot of software that exists in our application is external code Vulnerabilities can trickle down via third party libraries Software Composition Analysis (SCA) hunts for vulnerable components Application Internal Code @amirootyet

No content

No content

Static Analysis Security Testing White box automated testing of internal code Good for a first pass at security tests to fix obvious potential vulnerabilities False negatives exist + requires some security training Beware of the false positives and alert fatigue!

Language-specific SAST Tools

Enterprise-grade SAST Tools

Dynamic Analysis Security Testing Gray box automated testing of deployed application Able to detect deployment related security risks May require complex configuration False negatives on blind and/or asynchronous bugs

DAST Tools

DAST Tools

IaC Security Create and maintain secure infrastructure Secure configuration of provisioned infrastructure Secure deployment of application within infrastructure

Secrets Management Secrets are necessary for authentication Secrets can be exposed throughout the pipeline Exposure can be difficult to detect since secrets come in all shapes, sizes, forms

Pre-Commit Hooks to Discover Secrets Talisman Gitleaks

Build Systems Ephemeral versus persistent build systems Risk of vulnerable third party extensions Importance of patch management and hardening Least privileges in build environments

Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario @amirootyet

Build 2 Solarwinds Attack Scenario Is MSBuild.exe running? ElfHash(name) = 0x53D525 Is Orion being built? Extract CLI args from MSBuild.exe Extract directory path of Orion Replace a source code file Perform hash verification check @amirootyet

Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario Code Signing @amirootyet

Overreaching permissions Lingering stale accounts Local accounts created outside the scope of policies that enforce security practices in the pipelines Peer review all changes Access Control

Documentation Encourage and incentivize teams to document Little documentation is better than no documentation Do not seek perfection while documenting

You still need documentation, even it's all in your head "I AM the documentation" "But the auditors need documentation" When in doubt, always follow your documentation "Documentation? Ain't no dev got time for that!" @amirootyet

Artifacts Signing < 5% are signing code* *Source: Newman, Meyers, and Torres-Arias, “Sigstore: Signing is for Everyone,” ACM CCS, 2022 Not Signing 96% Signing 4%

Securing Modern CI/CD Pipeline Source: "Blueprint for building modern, secure software development pipelines", Veracode/Venafi https://github.com/Venafi/blueprint-securesoftwarepipeline

Build processes are automated and rapid Log and monitor events in SIEM systems to enable threat detection and response Adversaries persist where logs are absent Collect metrics on vulnerability management Visibility

Vulnerability Management Tools

DevSecOps Culture

Listen. Don't preach. Understand developer perspectives and constraints Suggest viable solutions within those constraints Provide implementation details to enable devsecops Create and support a security champions program Partnership (Security DevOps) Thou shalt sign thy code

The Golden Order of Impact Buy-in from stakeholders People Well-defined and easily adopted Process Effective, efficient, and fast execution Tools @amirootyet

Cyber Defense Matrix

Attack surface is wider than ever. Realign security efforts to incorporate the complexities Implement automated and manual security solutions. Tune, assess, evaluate, relearn, improve, repeat Do not fixate on tools: People > Process > Tools Identify gaps using a visual framework. Separate truth from marketing claims DevSecOps appears to artificially inject "security" into "devops." Security is an organic part of devops! Conclusion

Thank You @amirootyet