Slide 1

Slide 1 text

DevSecOps Mythbusters Detecting security vulnerabilities in and beyond CI/CD pipelines – “Open-Source Edition” OpenSouthCode Málaga – 9 June 2023 Jorge Hidalgo @deors314 in/deors

Slide 2

Slide 2 text

ABOUT ME Associate Director – Software Engineering Group – Accenture Spain Global Java Community of Practice co-lead Spain Cloud First DevOps lead Spain Adv. Tech. Center Cloud First Architecture & DevOps & Cloud lead Communities matter: MálagaJUG / Málaga Scala Developers / BoquerónSec co-lead Co-organizer: OpenSouthCode Málaga Father of two, husband, whistle player, video gamer, sci-fi *.*, Lego, Raspberry Pi, Star Wars, Star Trek, LOTR, Halo, Borderlands, Watch Dogs, Diablo, StarCraft, Black Desert,… LLAP! @deors314 in/deors JORGE HIDALGO

Slide 3

Slide 3 text

1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates AGENDA

Slide 4

Slide 4 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 5

Slide 5 text

DevSecOps == DevOps << DevOps is not possible unless security practices are adopted >>

Slide 6

Slide 6 text

DEV wants CHANGE OPS wants STABILITY But…, what does SECURITY want? (let’s not promote stereotypes J)

Slide 7

Slide 7 text

LUMINOUS BEINGS ARE WE

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

D E V ( S E C ) O P S “ D R A M A T I S P E R S O N A E ” Image adapted from Burr Sutter’s ”8 Steps to Becoming Awesome with Kubernetes” SEC BIZ QA SRE DEV INF DEV SRE INF DBA BIZ QA SEC DBA

Slide 10

Slide 10 text

S P I D E R - M A N “ I N V E N T E D ” D E V ( S E C ) O P S J

Slide 11

Slide 11 text

Ø Take decisions Ø Take ownership Ø Measure and improve continuously Ø Never asume that someone else with take care D E V ( S E C ) O P S T E A M S

Slide 12

Slide 12 text

Ø Infrestructure & platform Ø Data protection Ø Identity and access management Ø Application security Ø Quality gates D E C I S I O N S A N D O W N E R S H I P , B U T , A B O U T W H A T ?

Slide 13

Slide 13 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 14

Slide 14 text

Ø Hardening Ø Periodic patch/upgrade cycles (special attention to EOLs) Ø Vulnerability detection powered by tools (vs. manually reviewing the CVE catalog) Ø Cloud-native vulnerabilities Ø Infrastructure as code vulnerabilities Ø Zero-day vulnerabilities I N F R A S T R U C T U R E & P L A T F O R M S E C U R I T Y

Slide 15

Slide 15 text

Ø Of course, end-user computing/infrastructure Ø Access from unsupported/untrusted operating system Ø Access from old browsers (maybe 2-3 version tolerance) Ø Access from rooted/compromised devices I N F R A S T R U C T U R E & P L A T F O R M S E C U R I T Y

Slide 16

Slide 16 text

https://www.cve.org/

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

A G G R E G A T I N G E V E N T S F R O M M U L T I P L E S O U R C E S

Slide 25

Slide 25 text

N O P R O C E S S O R T O O L I S P E R F E C T Identify and manage policy exceptions Identify and manage false positives

Slide 26

Slide 26 text

Ø Falco Ø Clair Scanner Ø Trivy Ø TFLint Ø Checkov Ø Defect Dojo E X E M P L A R T O O L S

Slide 27

Slide 27 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 28

Slide 28 text

Ø Encryption in transit and at rest Ø Data classification and GDPR Ø Retention and deletion policies Ø Data access modeling, as per data classification Ø Access per functional role (e.g. user belongs to marketing dept.) Ø Information blocks in user interfaces (e.g. employee economic profile) Ø Report catalogue per role (e.g. report only available to insider list) D A T A S E C U R I T Y

Slide 29

Slide 29 text

Ø Classification leads to compartment and isolation Ø Microservice-oriented architectures can be of help for that Ø More difficult in monolithic databases or data lakes, where a huge amount of company information is stored and made available Ø Mitigate with segmented access and RBAC, e.g. no direct access to data lake but through a data mart with a business intelligence tool D A T A S E C U R I T Y

Slide 30

Slide 30 text

D A T A S E C U R I T Y Monolith Data Storage Data Repository Business Logic API Interface Presentation Layer Data Storage Data Repository Business Logic API Interface Presentation Layer Data Storage Data Repository Business Logic API Interface Data Storage Data Repository Business Logic API Interface Data Storage Data Repository Business Logic API Interface Order Service Shipping Service Payment Service Catalogue Service

Slide 31

Slide 31 text

Ø Create test cases to validate all of the previous Ø Made them part of acceptance criteria and regression Ø Run them in CI/CD pipelines D A T A S E C U R I T Y given as a user from Human Resources group when I’m authenticated into the system then I have access to employee payroll data

Slide 32

Slide 32 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 33

Slide 33 text

Ø Authentication and authorization Ø Differentiate end users from internal users (admins, devs…) Ø Multi-factor authentication (MFA) Ø Privileged access management (PAM) Ø Machine-to-machine access (M2M) a.k.a. “application IDs” Ø Workload identities Ø Service principal / service account Ø Certificate management (HTTPS) Ø Role-based access control (RBAC) Ø Low-level granularity, key for multi-tenant (shared) platforms I D E N T I T Y A N D A C C E S S M A N A G E M E N T

Slide 34

Slide 34 text

Ø Easy in small environments, e.g., a dozen of nodes Ø Easily solved using wildcard certificates and internal DNS to route access to individual tenants/applications Ø More complex in medium or large environments, e.g. a Kubernetes cluster with hundreds of nodes provisioned elastically – automation is key to escale the solution Ø cert-manager integrated with a certificate manager such as Let’s Encrypt Ø A service mesh such as Istio, capable to manage intra-cluster communication through mTLS and ingress/egress traffic C E R T I F I C A T E M A N A G E M E N T

Slide 35

Slide 35 text

R O L E - B A S E D A C C E S S C O N T R O L Physical deployment in a container platform – not related workloads share computing resources from the common infrastructure (through hypervisor) Cluster MANAGER GUEST OS MANAGER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS PERSISTENT STORAGE GUEST OS PERSISTENT STORAGE GUEST OS IMAGE REGISTRY GUEST OS IMAGE REGISTRY GUEST OS Internet Gateway MANAGER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS WORKER GUEST OS PERSISTENT STORAGE GUEST OS PERSISTENT STORAGE GUEST OS

Slide 36

Slide 36 text

R O L E - B A S E D A C C E S S C O N T R O L Logical deployment - - workloads are segregated in different namespaces which are isolated and protected through RBAC policies Internet Gateway ns A ns C ns B ns D pod pod pod pod ingress pod pod pod pod pod ingress pod pod pod pod pod pod pod pod pod ingress pod ingress pod pod

Slide 37

Slide 37 text

R O L E - B A S E D A C C E S S C O N T R O L ns A ns C ns B ns D pod pod pod pod ingress pod pod pod pod pod ingress pod pod pod pod pod pod pod pod pod ingress pod ingress pod pod RO RW super-admin (platform admin) admin for namespace C user for namespace C

Slide 38

Slide 38 text

R O L E - B A S E D A C C E S S C O N T R O L

Slide 39

Slide 39 text

R O L E - B A S E D A C C E S S C O N T R O L Ø Extend the model to protect from supply-chain attacks Ø Container registry or binary artefact management Ø Who/what can push, who/what can pull Ø Extend the model to secret management Ø Isolate and segment access Ø Who/what can create or change a secret Ø Who/what can access or use a secret

Slide 40

Slide 40 text

Ø cert-manager Ø Let’s Encrypt Ø Envoy Ø Linkerd Ø Istio Ø Open Service Mesh Ø Helm Secrets Ø Conjur E X E M P L A R T O O L S

Slide 41

Slide 41 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 42

Slide 42 text

Ø Static Application Security Testing (SAST) Ø Dynamic Application Security Testing (DAST) Ø Passive (web proxy) Ø Active (web spider/crawler) Ø Software Composition Analysis (SCA) Ø Secrets and sensitive configuration (leak protection) A P P L I C A T I O N S E C U R I T Y

Slide 43

Slide 43 text

Ø SQL injection Ø File path injection Ø XPath injection Ø Regular expression injection Ø Command injection Ø Code injection Ø Object injection (serialization) Ø Log injection (e.g. log4j) Ø Data leaks Ø Cross-Site Scripting (XSS) Ø Server-Side Request Forgery (SSRF) Ø Open redirection Ø Wrong authentication Ø Wrong access control Ø Hard-coded credentials Ø Weak crypto ciphers Ø Buffer overflows D E T E C T O R T Y P E S

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

A P P L I C A T I O N S E C U R I T Y T O O L S I N C I / C D P I P E L I N E S ENVIRONMENT PREPARATION COMPILER/ INTERPRETER UNIT TESTS MUTATION TESTS CODE INSPECTION SOFTWARE COMPOSITION ANALYSIS PACKAGE BUILD CONTAINER IMAGE RUN TEST CONTAINER INTEGRATION TESTS PERFORMANCE TESTS PROMOTE/ DEPLOY IMAGE This map is not exhaustive of every available option, but representative of common tools used by projects CI/CD ENGINE

Slide 47

Slide 47 text

A P P L I C A T I O N S E C U R I T Y T O O L S I N C I / C D P I P E L I N E S

Slide 48

Slide 48 text

Ø SonarQube Ø SpotBugs (FindSecBugs) Ø PMD Ø ESLint Ø Pylama Ø checkov Ø Nikto Ø OWASP ZAP Ø OWASP Dependency Tracker Ø Detectify E X E M P L A R T O O L S

Slide 49

Slide 49 text

AGENDA 1. Introduction 2. Infrastructure 3. Data 4. Identities 5. Applications 6. Gates

Slide 50

Slide 50 text

Ø For every control validating something it should exist the corresponding quality gate Ø Gates must be enforced in the CI/CD pipeline not allowing to continue when a threshold is not met Ø For infrastructure and platform, define clear policies and vulnerability resolution times Ø E.g., 30 days for non-critical vulnerabilities, 2 days for critical Ø Do not allow deployments after the resolution threshold is exceeded Ø For stuff already running, eliminate the deployment or preferably quarantine the stack (e.g., eliminate network routes but do not remove for forensics) Ø Make the process aware of exceptions and false positives Q U A L I T Y G A T E S

Slide 51

Slide 51 text

Ø Should we have global gates at the organization level, or gates per domain or application? Ø DevSecOps promotes the second: Independence and responsibility of product- oriented teams Ø To make it viable at the organization level, govern them with transparency and traceability Ø Some tools already have their own project-level quality gate functionality, e.g., SonarQube. Should we use them? Ø It is possible, but may be of limited applicability (only partial measures, evaluation at inconvenient times along the pipeline) Ø It is generally better to set up gates per stage or even per tool Ø Should we hard-code thresholds in the pipelines? Ø A threshold in a pipeline should be a clear and traceable specification Ø It is better to externalize gate thresholds, for transparency and for clarity about who set what and why (e.g., do not “dilute” threshold values in long pipelines) Q U A L I T Y G A T E S F A Q

Slide 52

Slide 52 text

Q U A L I T Y G A T E S E X T E R N A L I Z A T I O N GOOD BETTER!

Slide 53

Slide 53 text

Q U A L I T Y G A T E S M A N A G E D B Y T O O L S

Slide 54

Slide 54 text

A N Y Q U E S T I O N S ?