Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HACKING CI/CD PIPELINES by Sumir Broota

HACKING CI/CD PIPELINES by Sumir Broota

Title: HACKING CI/CD PIPELINES
Presenter: Sumir Broota
Event: BreachForce CyberSecurity Cohort
Talk Date: 3rd February, 2024

Key Takeaways: An interactive workshop exploring vulnerabilities in CI/CD pipelines and techniques for securing them.

BreachForce

April 21, 2024
Tweet

More Decks by BreachForce

Other Decks in Technology

Transcript

  1. From A Laptop to production the fun way ;P HACKERS

    SECURITY HACKING CI/CD HACKING CI/CD PIPELINES PIPELINES HIGHSCORE 2500
  2. 01 12 07 WHOAMI WHOAMI Hello, My name is SuMir

    Broota. I am a Research Developer @ Idfy. I was previously an Infosec Engineer, primarily focused on red team.
  3. 01 12 07 sumirbroota.com WHOAMI WHOAMI Hello, My name is

    SuMir Broota. I am a Research Developer @ Idfy. I was previously an Infosec Engineer, primarily focused on red team.
  4. 01 12 07 THEORY OWASP TOP 10 CI/CD RISKS Discussing

    5 of the owasp threats DEmo 1: TwiddleDUM Breif on Other 5 threats DEmo of one of the threats AGENDA AGENDA PRACTICAL CTF WALKTHROUGH (CI/CD GOAT)
  5. INSUFFICIENT INSUFFICIENT FLOW CONTROL FLOW CONTROL MECHANISMS MECHANISMS 1 1.

    . Description If you're allowed to directly push code to master/change pipeline working/alter artifact registry image (no branch protection/approval needed) Impact Abuse insufficient flow control mechanisms to deploy malicious artifacts potentially all the way to production - without any approval or review. Recommendation Configure branch protection rules on branches hosting code which is used in production and other sensitive systems. Where possible, avoid exclusion of user accounts or branches from branch protection rules. 1. Limit the usage of auto-merge rules and ensure that wherever they are in use - they are applicable to the minimal amount of contexts 2. Detect and prevent drifts and inconsistencies between code running in production and its CI/CD origin, and modify any resource that contains a drift. 3.
  6. 2. 2. INADEQUATE INADEQUATE IDENTITY AND IDENTITY AND ACCESS ACCESS

    MANAGEMENT MANAGEMENT Description Broken Authentication/Authorisation, Overly permissive, Shared Identities, Stale accounts, External accs, Non-IDP managed account Impact Difficulty in managing the number of accounts, their access and consistent security posture results in any leaked credentials being a major threat to the org Recommendation Conduct a continuous analysis and mapping of all identities across all systems & remove permissions not necessary for the ongoing work 1. Avoid creating local user accounts. Instead, create and manage identities using a centralized organization component (IdP). Preferably for external collaborators as well 2. Refrain from granting overly permissive base permissions in a system to all users, and refrain from allowing users to self-register to systems. Avoid using shared accounts. 3.
  7. 3. DEPENDENCY 3. DEPENDENCY CHAIN ABUSE CHAIN ABUSE DESCRIPTION Typosquatting,

    pkg’s with same name, gaining access to maintainers account and publishing mal artifact,phishing dependencies, AI hallucinations Impact Recommendation Enable checksum verification and signature verification for pulled packages. 1. Avoid configuring clients to pull the latest version of a package. Prefer configuring a pre-vetted version or version ranges. 2. Ensure all private packages are registered under the organization’s scope & ensure all packages are pulled through an internal proxy rather than directly from the internet. This allows deploying additional security controls at the proxy layer, as well as providing investigative capabilities around packages pulled - in case of a security incident. 3. IMpact Attacker can execute malicious code either on a developer’s workstation, or a build server pulling the package. Once the malicious code is running, it can be leveraged for credentials theft and lateral movement within the environment it is executed in.
  8. 4. POISONED PIPELINE 4. POISONED PIPELINE EXECUTION (PPE) EXECUTION (PPE)

    DESCRIPTION Broken Authentication/Authorisation, Overly permissive, Shared Identities, Stale accounts, External accs, Non-IDP managed account Impact Access to any secret/external assets available to the CI job are exposed. Ability to access additional hosts and assets in the network/environment of the job node. Ability to ship code and artifacts further down the pipeline Recommendation Ensure that pipelines running unreviewed code are executed on isolated nodes, not exposed to secrets, sensitive environments with minimum short lived credentials. 1. Evaluate the need for triggering pipelines on public repositories from external contributors, require manual approval for pipeline execution. 2. Each CI configuration file must be reviewed before the pipeline runs and should atleast be managed in a protected branch separate from code changes 3. Remove permissions granted on the SCM repository from users that do not need them. 4. DESCRIPTION When attacker can access source control systems without access to the build environment. The attacker may manipulate the build process by injecting malicious code/commands into the build pipeline configuration, essentially ‘poisoning’ the pipeline and running malicious code as part of the build process Types Direct - modifies the CI config file in a repository they have access to, either by pushing the change directly to an unprotected remote branch on the repo 1. Indirect - If CI configuration file is present in a separate, protected branch in the same repository; or in a seperate repo with no direct edit access; or in the CI system itself 2. Public - If the CI pipeline of a public repository runs unreviewed code suggested by anonymous users. 3.
  9. 5. INSUFFICIENT 5. INSUFFICIENT PBAC (PIPELINE- PBAC (PIPELINE- BASED ACCESS

    BASED ACCESS CONTROLS) CONTROLS) DESCRIPTION Pipelines often have excessively high privileges which can be abused by an attacker Impact Overly permissive shared pipeline systems can be abused for lateral movement, accessing secrets, and underlying host. The impact of the attack is multiplied by the sensitive info present in the environment Recommendation Do not use a shared node for pipelines with different levels of sensitivity / that require access to different resources 1. Ensure secrets that are used in CI/CD systems are scoped in a manner that allows each pipeline and step to have access to only the secrets it requires. 2. Ensure network segmentation in the environment the job is running on is configured to allow the execution node to access only the resources it requires within the network. Where possible, refrain from granting unlimited access towards the internet to build nodes. 3. Containerize runners to have an extra degree of isolation. Use tools such as SysBox 4. Use OIDC tokens with a 3rd party identity provider rather than sensitive secrets in the environment. 5.
  10. 6. INSUFFICIENT 6. INSUFFICIENT CREDENTIAL CREDENTIAL HYGIENE HYGIENE DESCRIPTION If

    credentials for the application in the pipeline are not managed well. If they are overly permissive/leaked in repo or console output/never rotating i.e insecure secret management Impact Potential for human error, paired with knowledge gaps around secure credentials management and the concern of breaking processes due to credential rotation, put the high-value resources of many organizations at the risk of compromise due the exposure of their credentials. Recommendation Avoid sharing the same set of credentials across multiple contexts and follows the principle of least privilege. 1. Prefer using temporary credentials over static credentials. 2. Scope credentials to a specific source IP or identity to ensure that even in case of compromise, exfiltrated credentials cannot be used outside your environment. 3. Use built-in vendor options or 3rd party tools to prevent secrets from being printed to console outputs of future builds. Ensure all existing outputs do not contain secrets. 4. Verify that secrets are removed from any type of artifact, such as from layers of container images, binaries, or Helm charts. 5.
  11. 7. INSECURE 7. INSECURE SYSTEM SYSTEM CONFIGURATION CONFIGURATION DescRiption Insecure

    system configuration risks stem from flaws in the security settings, configuration and hardening of the different systems across the pipeline (e.g. SCM, CI, Artifact repository) Impact A security flaw in one of the CI/CD systems may be leveraged by an adversary to compromise the system, manipulate legitimate CI/CD flows, obtain sensitive tokens and potentially access production environments. Recommendation Maintain an inventory of systems and versions in use, including mapping of a designated owner for each system. Continuously check for known vulnerabilities in these components. If a security patch is available, update the vulnerable component. 1. Establish a process to periodically review all system configurations for any setting that can have an effect on the security posture of the system. 2. Ensure permissions to the pipeline execution nodes are granted according to the principle of least privilege. 3.
  12. 8. UNGOVERNED 8. UNGOVERNED USAGE OF 3RD USAGE OF 3RD

    PARTY SERVICES PARTY SERVICES DESCRIPTION The CI/CD attack surface consists of an org’s 1st party assets, such as the SCM or CI, and the 3rd party services which are granted access to those org’s 1st party assets. Ungoverned usage of 3rd party services, effectively expand the attack surface of the organization. e.g OAuth application, webhook events/access tokens to be sent to the 3rd party Impact Insufficient implementation of RBAC and least privilege around 3rd parties, coupled with minimal governance and diligence around the process of 3rd party implementations create a significant increase of the organization’s attack surface. Recommendation Establish vetting procedures to ensure 3rd parties granted access to resources anywhere across the engineering ecosystem are approved prior to being granted access to the environment, and that the level of permission they are granted is aligned with the principle of least privilege. 1. Ensure each 3rd party is limited and scoped to the specific resources it requires access to and remove unused and/or redundant permissions. 3rd parties which are integrated as part of the Build process should run inside a scoped context with limited access to secrets and code, and with strict ingress and egress filters. 2. Periodically review all 3rd parties integrated and remove those no longer in use. 3.
  13. 9. IMPROPER 9. IMPROPER ARTIFACT ARTIFACT INTEGRITY INTEGRITY VALIDATION VALIDATION

    DESCRIPTION Allow an attacker with access to one of the systems in the CI/CD process to push malicious code or artifacts down the pipeline, due to insufficient mechanisms for ensuring the validation of code and artifacts. Impact It can be abused by an adversary to ship a malicious artifact through the pipeline, ultimately resulting in the execution of malicious code in production. Recommendation Validate the integrity of resources all the way from development to production. When a resource is generated, the process will include signing that resource using an external resource signing infrastructure 1. Sign commits using a unique key for each contributor. This can be leveraged to prevent unsigned commits from flowing down the pipeline. 2. Use tools for signing and verification of code and artifacts provide a way to prevent unverified software from being delivered down the pipeline. 3. Measure configuration drifts (e.g. resources in cloud environments which aren’t managed using a signed IaC template), potentially indicative of resources that were deployed by an untrusted source or process. 4.
  14. 10. 10. INSUFFICIENT INSUFFICIENT LOGGING AND LOGGING AND VISIBILITY VISIBILITY

    DESCRIPTION Allows an adversary to carry out malicious activities within the CI/CD environment without being detected including in any post-incident investigation. Impact Organizations which do not ensure the appropriate logging/visibility may fail to detect a breach, and face great difficulties in mitigation/remediation due to minimal investigative capabilities. Recommendation Identify and build an inventory of all the systems in use within the organization, containing every instance of these systems (specifically relevant for self-managed systems e.g. Jenkins). 1. Ensure that all relevant audit/application logs are enabled, as this is not the default state in the different systems. Visibility should be optimized around both human & programmatic access through all the various measures supported. 2. Shipping logs to a centralized location (e.g. SIEM), to support aggregation and correlation of logs between different systems for detection and investigation. 3. Create alerts to detect anomalies and potential malicious activity, in each system and in the code shipping process 4.
  15. Common issues Using user input in your scripts in an

    unsafe way (see refs) 1. Monitor for Application/Pipeline config issues 2. issue_* and pull_request* events for unverified input can be a potential threat 3. 01 12 07 Back to Agenda Page HACKING HACKING GITHUB GITHUB ACTIONS ACTIONS
  16. OWASP CI/CD TOP 10 REFERENCE REFERENCE LINKS LINKS HACKTRICKS GITHUB

    ACTIONS ABUSE TINDER GITHUB ACTIONS SEC TOOL RAVEN CI/CD SAST CI/CD GOAT SOLARWINDS ATTACK SYSBOX - CONTAINERIZE RUNNERS USING TRACEE FOR GITHUB ACTION SECURITY (DEV)