DevSecOps Mythbusters Detecting security vulnerabilities in and beyond CI/CD pipelines – “Open-Source Edition” OpenSouthCode Málaga – 9 June 2023 Jorge Hidalgo @deors314 in/deors 

ABOUT ME Associate Director – Software Engineering Group – Accenture Spain Global Java Community of Practice co-lead Spain Cloud First DevOps lead Spain Adv. Tech. Center Cloud First Architecture & DevOps & Cloud lead Communities matter: MálagaJUG / Málaga Scala Developers / BoquerónSec co-lead Co-organizer: OpenSouthCode Málaga Father of two, husband, whistle player, video gamer, sci-fi *.*, Lego, Raspberry Pi, Star Wars, Star Trek, LOTR, Halo, Borderlands, Watch Dogs, Diablo, StarCraft, Black Desert,… LLAP! @deors314 in/deors JORGE HIDALGO 

1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates AGENDA 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

DevSecOps == DevOps << DevOps is not possible unless security practices are adopted >> 

DEV wants CHANGE OPS wants STABILITY But…, what does SECURITY want? (let’s not promote stereotypes J) 

LUMINOUS BEINGS ARE WE 

No content

D E V ( S E C ) O P S “ D R A M A T I S P E R S O N A E ” Image adapted from Burr Sutter’s ”8 Steps to Becoming Awesome with Kubernetes” SEC BIZ QA SRE DEV INF DEV SRE INF DBA BIZ QA SEC DBA 

S P I D E R - M A N “ I N V E N T E D ” D E V ( S E C ) O P S J 

Ø Take decisions Ø Take ownership Ø Measure and improve continuously Ø Never asume that someone else with take care D E V ( S E C ) O P S T E A M S 

Ø Infrestructure & platform Ø Data protection Ø Identity and access management Ø Application security Ø Quality gates D E C I S I O N S A N D O W N E R S H I P , B U T , A B O U T W H A T ? 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

Ø Hardening Ø Periodic patch/upgrade cycles (special attention to EOLs) Ø Vulnerability detection powered by tools (vs. manually reviewing the CVE catalog) Ø Cloud-native vulnerabilities Ø Infrastructure as code vulnerabilities Ø Zero-day vulnerabilities I N F R A S T R U C T U R E & P L A T F O R M S E C U R I T Y 

Ø Of course, end-user computing/infrastructure Ø Access from unsupported/untrusted operating system Ø Access from old browsers (maybe 2-3 version tolerance) Ø Access from rooted/compromised devices I N F R A S T R U C T U R E & P L A T F O R M S E C U R I T Y 

https://www.cve.org/ 

No content

No content

No content

No content

No content

No content

No content

A G G R E G A T I N G E V E N T S F R O M M U L T I P L E S O U R C E S 

N O P R O C E S S O R T O O L I S P E R F E C T Identify and manage policy exceptions Identify and manage false positives 

Ø Falco Ø Clair Scanner Ø Trivy Ø TFLint Ø Checkov Ø Defect Dojo E X E M P L A R T O O L S 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

Ø Encryption in transit and at rest Ø Data classification and GDPR Ø Retention and deletion policies Ø Data access modeling, as per data classification Ø Access per functional role (e.g. user belongs to marketing dept.) Ø Information blocks in user interfaces (e.g. employee economic profile) Ø Report catalogue per role (e.g. report only available to insider list) D A T A S E C U R I T Y 

Ø Classification leads to compartment and isolation Ø Microservice-oriented architectures can be of help for that Ø More difficult in monolithic databases or data lakes, where a huge amount of company information is stored and made available Ø Mitigate with segmented access and RBAC, e.g. no direct access to data lake but through a data mart with a business intelligence tool D A T A S E C U R I T Y 

D A T A S E C U R I T Y Monolith Data Storage Data Repository Business Logic API Interface Presentation Layer Data Storage Data Repository Business Logic API Interface Presentation Layer Data Storage Data Repository Business Logic API Interface Data Storage Data Repository Business Logic API Interface Data Storage Data Repository Business Logic API Interface Order Service Shipping Service Payment Service Catalogue Service 

Ø Create test cases to validate all of the previous Ø Made them part of acceptance criteria and regression Ø Run them in CI/CD pipelines D A T A S E C U R I T Y given as a user from Human Resources group when I’m authenticated into the system then I have access to employee payroll data 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

Ø Authentication and authorization Ø Differentiate end users from internal users (admins, devs…) Ø Multi-factor authentication (MFA) Ø Privileged access management (PAM) Ø Machine-to-machine access (M2M) a.k.a. “application IDs” Ø Workload identities Ø Service principal / service account Ø Certificate management (HTTPS) Ø Role-based access control (RBAC) Ø Low-level granularity, key for multi-tenant (shared) platforms I D E N T I T Y A N D A C C E S S M A N A G E M E N T 

Ø Easy in small environments, e.g., a dozen of nodes Ø Easily solved using wildcard certificates and internal DNS to route access to individual tenants/applications Ø More complex in medium or large environments, e.g. a Kubernetes cluster with hundreds of nodes provisioned elastically – automation is key to escale the solution Ø cert-manager integrated with a certificate manager such as Let’s Encrypt Ø A service mesh such as Istio, capable to manage intra-cluster communication through mTLS and ingress/egress traffic C E R T I F I C A T E M A N A G E M E N T 

R O L E - B A S E D A C C E S S C O N T R O L Physical deployment in a container platform – not related workloads share computing resources from the common infrastructure (through hypervisor) Cluster MANAGER GUEST OS MANAGER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS PERSISTENT STORAGE GUEST OS PERSISTENT STORAGE GUEST OS IMAGE REGISTRY GUEST OS IMAGE REGISTRY GUEST OS Internet Gateway MANAGER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS PERSISTENT STORAGE GUEST OS PERSISTENT STORAGE GUEST OS 

R O L E - B A S E D A C C E S S C O N T R O L Logical deployment - - workloads are segregated in different namespaces which are isolated and protected through RBAC policies Internet Gateway ns A ns C ns B ns D pod pod pod pod ingress pod pod pod pod pod ingress pod pod pod pod pod pod pod pod pod ingress pod ingress pod pod 

R O L E - B A S E D A C C E S S C O N T R O L ns A ns C ns B ns D pod pod pod pod ingress pod pod pod pod pod ingress pod pod pod pod pod pod pod pod pod ingress pod ingress pod pod RO RW super-admin (platform admin) admin for namespace C user for namespace C 

R O L E - B A S E D A C C E S S C O N T R O L 

R O L E - B A S E D A C C E S S C O N T R O L Ø Extend the model to protect from supply-chain attacks Ø Container registry or binary artefact management Ø Who/what can push, who/what can pull Ø Extend the model to secret management Ø Isolate and segment access Ø Who/what can create or change a secret Ø Who/what can access or use a secret 

Ø cert-manager Ø Let’s Encrypt Ø Envoy Ø Linkerd Ø Istio Ø Open Service Mesh Ø Helm Secrets Ø Conjur E X E M P L A R T O O L S 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

Ø Static Application Security Testing (SAST) Ø Dynamic Application Security Testing (DAST) Ø Passive (web proxy) Ø Active (web spider/crawler) Ø Software Composition Analysis (SCA) Ø Secrets and sensitive configuration (leak protection) A P P L I C A T I O N S E C U R I T Y 

Ø SQL injection Ø File path injection Ø XPath injection Ø Regular expression injection Ø Command injection Ø Code injection Ø Object injection (serialization) Ø Log injection (e.g. log4j) Ø Data leaks Ø Cross-Site Scripting (XSS) Ø Server-Side Request Forgery (SSRF) Ø Open redirection Ø Wrong authentication Ø Wrong access control Ø Hard-coded credentials Ø Weak crypto ciphers Ø Buffer overflows D E T E C T O R T Y P E S 

No content

No content

A P P L I C A T I O N S E C U R I T Y T O O L S I N C I / C D P I P E L I N E S ENVIRONMENT PREPARATION COMPILER/ INTERPRETER UNIT TESTS MUTATION TESTS CODE INSPECTION SOFTWARE COMPOSITION ANALYSIS PACKAGE BUILD CONTAINER IMAGE RUN TEST CONTAINER INTEGRATION TESTS PERFORMANCE TESTS PROMOTE/ DEPLOY IMAGE This map is not exhaustive of every available option, but representative of common tools used by projects CI/CD ENGINE 

A P P L I C A T I O N S E C U R I T Y T O O L S I N C I / C D P I P E L I N E S 

Ø SonarQube Ø SpotBugs (FindSecBugs) Ø PMD Ø ESLint Ø Pylama Ø checkov Ø Nikto Ø OWASP ZAP Ø OWASP Dependency Tracker Ø Detectify E X E M P L A R T O O L S 

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates 

Ø For every control validating something it should exist the corresponding quality gate Ø Gates must be enforced in the CI/CD pipeline not allowing to continue when a threshold is not met Ø For infrastructure and platform, define clear policies and vulnerability resolution times Ø E.g., 30 days for non-critical vulnerabilities, 2 days for critical Ø Do not allow deployments after the resolution threshold is exceeded Ø For stuff already running, eliminate the deployment or preferably quarantine the stack (e.g., eliminate network routes but do not remove for forensics) Ø Make the process aware of exceptions and false positives Q U A L I T Y G A T E S 

Ø Should we have global gates at the organization level, or gates per domain or application? Ø DevSecOps promotes the second: Independence and responsibility of product- oriented teams Ø To make it viable at the organization level, govern them with transparency and traceability Ø Some tools already have their own project-level quality gate functionality, e.g., SonarQube. Should we use them? Ø It is possible, but may be of limited applicability (only partial measures, evaluation at inconvenient times along the pipeline) Ø It is generally better to set up gates per stage or even per tool Ø Should we hard-code thresholds in the pipelines? Ø A threshold in a pipeline should be a clear and traceable specification Ø It is better to externalize gate thresholds, for transparency and for clarity about who set what and why (e.g., do not “dilute” threshold values in long pipelines) Q U A L I T Y G A T E S F A Q 

Q U A L I T Y G A T E S E X T E R N A L I Z A T I O N GOOD BETTER! 

Q U A L I T Y G A T E S M A N A G E D B Y T O O L S 

A N Y Q U E S T I O N S ?