Slide 1

Slide 1 text

How They Hacked Your DevOps Risk Mitigation in Development Environments 02/02/2023 Pranshu Bajpai @amirootyet

Slide 2

Slide 2 text

About PhD, Computer Science, Michigan State University Principal Security Architect, Motorola Solutions Inc Speaker: DEFCON, GrrCon, ToorCon, APWG eCrime, IACP, CascadiaJS, Bsides, IEEE SecDev etc. @amirootyet in/pranshubajpai/ Research: Malware, Threats, Forensics, Applied Crypto, DevOps, Cloud Security, Data Science

Slide 3

Slide 3 text

Agenda About 1 Common Security Risks 2 DevOps Culture 3 Conclusion 4

Slide 4

Slide 4 text

Disclaimer opinions are my own; not those of my employer tools shown are merely a means to an end @amirootyet

Slide 5

Slide 5 text

DevOps @amirootyet

Slide 6

Slide 6 text

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline @amirootyet

Slide 7

Slide 7 text

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery CI/CD Pipeline Security?? Security?? Security?? @amirootyet

Slide 8

Slide 8 text

Automated Testing 1 Build 2 Deploy Test Env 3 Further Testing 4 Deploy Prod 5 Modern DevOps Continuous and Seamless Software Delivery security

Slide 9

Slide 9 text

Pre-Commit Hooks 1 Pre-Build 2 Post-Build 3 Deploy Test Env 4 Deploy Prod 5 Enter DevSecOps Shift Security Left CI/CD Pipeline Secrets Management SCA & SAST DAST IaC Security @amirootyet

Slide 10

Slide 10 text

Common DevOps Security Risks

Slide 11

Slide 11 text

"So... which of our apps use log4j?"

Slide 12

Slide 12 text

Dependency Management A lot of software that exists in our application is external code Vulnerabilities can trickle down via third party libraries Software Composition Analysis (SCA) hunts for vulnerable components Application Internal Code @amirootyet

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Static Analysis Security Testing White box automated testing of internal code Good for a first pass at security tests to fix obvious potential vulnerabilities False negatives exist + requires some security training Beware of the false positives and alert fatigue!

Slide 16

Slide 16 text

Language-specific SAST Tools

Slide 17

Slide 17 text

Enterprise-grade SAST Tools

Slide 18

Slide 18 text

Dynamic Analysis Security Testing Gray box automated testing of deployed application Able to detect deployment related security risks May require complex configuration False negatives on blind and/or asynchronous bugs

Slide 19

Slide 19 text

DAST Tools

Slide 20

Slide 20 text

DAST Tools

Slide 21

Slide 21 text

IaC Security Create and maintain secure infrastructure Secure configuration of provisioned infrastructure Secure deployment of application within infrastructure

Slide 22

Slide 22 text

Secrets Management Secrets are necessary for authentication Secrets can be exposed throughout the pipeline Exposure can be difficult to detect since secrets come in all shapes, sizes, forms

Slide 23

Slide 23 text

Pre-Commit Hooks to Discover Secrets Talisman Gitleaks

Slide 24

Slide 24 text

Build Systems Ephemeral versus persistent build systems Risk of vulnerable third party extensions Importance of patch management and hardening Least privileges in build environments

Slide 25

Slide 25 text

Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario @amirootyet

Slide 26

Slide 26 text

Build 2 Solarwinds Attack Scenario Is MSBuild.exe running? ElfHash(name) = 0x53D525 Is Orion being built? Extract CLI args from MSBuild.exe Extract directory path of Orion Replace a source code file Perform hash verification check @amirootyet

Slide 27

Slide 27 text

Test 1 Build 2 Deploy 3 CI/CD Pipeline Solarwinds Attack Scenario Code Signing @amirootyet

Slide 28

Slide 28 text

Overreaching permissions Lingering stale accounts Local accounts created outside the scope of policies that enforce security practices in the pipelines Peer review all changes Access Control

Slide 29

Slide 29 text

Documentation Encourage and incentivize teams to document Little documentation is better than no documentation Do not seek perfection while documenting

Slide 30

Slide 30 text

You still need documentation, even it's all in your head "I AM the documentation" "But the auditors need documentation" When in doubt, always follow your documentation "Documentation? Ain't no dev got time for that!" @amirootyet

Slide 31

Slide 31 text

Artifacts Signing < 5% are signing code* *Source: Newman, Meyers, and Torres-Arias, “Sigstore: Signing is for Everyone,” ACM CCS, 2022 Not Signing 96% Signing 4%

Slide 32

Slide 32 text

Securing Modern CI/CD Pipeline Source: "Blueprint for building modern, secure software development pipelines", Veracode/Venafi

Slide 33

Slide 33 text

Build processes are automated and rapid Log and monitor events in SIEM systems to enable threat detection and response Adversaries persist where logs are absent Collect metrics on vulnerability management Visibility

Slide 34

Slide 34 text

Vulnerability Management Tools

Slide 35

Slide 35 text

DevSecOps Culture

Slide 36

Slide 36 text

Listen. Don't preach. Understand developer perspectives and constraints Suggest viable solutions within those constraints Provide implementation details to enable devsecops Create and support a security champions program Partnership (Security DevOps) Thou shalt sign thy code

Slide 37

Slide 37 text

The Golden Order of Impact Buy-in from stakeholders People Well-defined and easily adopted Process Effective, efficient, and fast execution Tools @amirootyet

Slide 38

Slide 38 text

Cyber Defense Matrix

Slide 39

Slide 39 text

Attack surface is wider than ever. Realign security efforts to incorporate the complexities Implement automated and manual security solutions. Tune, assess, evaluate, relearn, improve, repeat Do not fixate on tools: People > Process > Tools Identify gaps using a visual framework. Separate truth from marketing claims DevSecOps appears to artificially inject "security" into "devops." Security is an organic part of devops! Conclusion

Slide 40

Slide 40 text

Thank You @amirootyet