Tweet
Tweet

Slide 1

Slide 1 text

Rosemary Wang | June 8, 2022 Let’s Secure a CI/CD Pipeline 1

Slide 2

Slide 2 text

jenkins.io/security/advisory/2022-02-15/ about.codecov.io/security-update/ msrc-blog.microsoft.com/2020/12/31/microsoft-internal-solorigate-investigation-update/ 2

Slide 3

Slide 3 text

πŸ€” + 😬 = 😱 3

Slide 4

Slide 4 text

Rosemary Wang (she/her) Developer Advocate at HashiCorp @joatmon08 joatmon08.github.io 4

Slide 5

Slide 5 text

How to secure better… 5

Slide 6

Slide 6 text

How to secure better… and remediate faster. 6

Slide 7

Slide 7 text

Securing CI/CD Pipelines An Overview βœ“ Access Control βœ“ Secrets βœ“ Runners βœ“ Dependencies βœ“ Configuration 7

Slide 8

Slide 8 text

Access Control πŸš₯ 8

Slide 9

Slide 9 text

limit service’s access rights to minimum required least privilege 9

Slide 10

Slide 10 text

Pipelines need access. Code Repository accesses… Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Run integration tests. Store for User Data 10

Slide 11

Slide 11 text

Choose your access. Code Repository Checkout code. Read repositories. 11

Slide 12

Slide 12 text

Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Read repositories. Write speci fi c services. 12

Slide 13

Slide 13 text

Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. 13

Slide 14

Slide 14 text

Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. 14

Slide 15

Slide 15 text

Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. Read test results. 15

Slide 16

Slide 16 text

Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. Read test results. Run integration tests. Store for User Data Read table in testing environment. 16

Slide 17

Slide 17 text

The challenge of least privilege Solutions β˜… Limit access from pipeline early in development β˜… Limit write access to repositories early in development 17

Slide 18

Slide 18 text

The challenge of least privilege Solutions β˜… Limit access from pipeline early in development β˜… Limit write access to repositories early in development β˜… Offer self-service to refine access β˜… Offer templates of secure policies 18

Slide 19

Slide 19 text

The challenge of least privilege Solutions β˜… Limit access from pipeline early in development β˜… Limit write access to repositories early in development β˜… Offer self-service to refine access β˜… Offer templates of secure policies β˜… Review access on a regular cadence β˜… Audit pipeline runs 19

Slide 20

Slide 20 text

Secrets πŸ”’ 20

Slide 21

Slide 21 text

sensitive information linked to access to a system or service secrets 21

Slide 22

Slide 22 text

vault_database_secret_backend_connection.post gres will be created β€’ resource "vault_database_secret_backend_connection" "postgres" {
 β—¦ postgresql { β–ͺ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST] 22

Slide 23

Slide 23 text

vault_database_secret_backend_connection.post gres will be created β€’ resource "vault_database_secret_backend_connection" "postgres" {
 β—¦ postgresql { β–ͺ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST] 23 😱

Slide 24

Slide 24 text

Plan R Remediate the secret β€’ Regret β€’ Revoke β€’ Rotate β€’ Reference β€’ Replace β€’ Re-run 24

Slide 25

Slide 25 text

Plan R Remediate the secret β€’ Regret β€’ Revoke β€’ Rotate β€’ Reference β€’ Replace β€’ Re-run 25 100 pipelines later… πŸ˜“

Slide 26

Slide 26 text

Pipelines use secrets. 26 Certi fi cates Access Usernames & Passwords Testing User Data Tokens SSH Keys Encryption Keys

Slide 27

Slide 27 text

Pipelines create secrets. 27 Con fi guration Usernames & Passwords SSH Keys Tokens

Slide 28

Slide 28 text

The challenge of secrets Solutions β˜… Mask or omit in pipeline output β˜… Use a secrets manager 28

Slide 29

Slide 29 text

The challenge of secrets Solutions β˜… Mask or omit in pipeline output β˜… Use a secrets manager β˜… Issue new credentials per pipeline run β˜… Audit secrets usage 29

Slide 30

Slide 30 text

Runners πŸ‘Ÿ 30

Slide 31

Slide 31 text

resources that run pipeline stages or tasks runners 31

Slide 32

Slide 32 text

32 Virtual Machine πŸ€” 1. Someone accesses CI/CD runner (e.g., SSH). 2. Access other infrastructure. Database

Slide 33

Slide 33 text

33 Container πŸ€” 1. Someone accesses CI/CD runner (e.g., SSH). Virtual Machine $ mount /dev/ /mnt $ chroot /mnt 2. Container can access host fi lesystem. 3. Access code or fi les for other jobs. Other Jobs on Virtual Machine

Slide 34

Slide 34 text

34 Infrastructure Provider Runner Managed Service Engineer Must be authorized user. Must be authorized account. Secrets Manager

Slide 35

Slide 35 text

35 Infrastructure Provider Runner Managed Service Engineer Must be authorized user. Must be authorized account. Secrets Manager Allow IP address over VPN. Allow IP addresses for CI framework.

Slide 36

Slide 36 text

The challenge of securing runners Solutions β˜… Use trusted / verified images β˜… Scan for OS vulnerabilities 36

Slide 37

Slide 37 text

The challenge of securing runners Solutions β˜… Use trusted / verified images β˜… Scan for OS vulnerabilities β˜… Define network policy β˜… Run as a non-root 37

Slide 38

Slide 38 text

The challenge of securing runners Solutions β˜… Use trusted / verified images β˜… Scan for OS vulnerabilities β˜… Define network policy β˜… Run as a non-root β˜… Use ephemeral secrets β˜… Audit remote access to runner 38

Slide 39

Slide 39 text

Dependencies πŸ–‡ 39

Slide 40

Slide 40 text

third-party code used for pipeline stages or tasks dependencies 40

Slide 41

Slide 41 text

41 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist Downloaded from trusted source? Veri fi ed code? Correct plugin?

Slide 42

Slide 42 text

42 public class UnverifiedPlugin { protected static void getFiles(AbstractBuild b, FilePath workspace) { // code to replace project files or metadata // code to gather information // code to siphon credentials } }

Slide 43

Slide 43 text

The challenge of securing dependencies Solutions β˜… Scan for vulnerabilities β˜… Verify checksums and signatures 43

Slide 44

Slide 44 text

The challenge of securing dependencies Solutions β˜… Scan for vulnerabilities β˜… Verify checksums and signatures β˜… Use verified registry β˜… Pin versions 44

Slide 45

Slide 45 text

Configuration πŸ“„ 45

Slide 46

Slide 46 text

de fi ne delivery pipelines through source code pipeline as code 46

Slide 47

Slide 47 text

47 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist test_plugin_checkout_has_fetch_depth_of_1 test_plugin_go_uses_secure_version_1.14 test_plugin_release_includes_signature

Slide 48

Slide 48 text

The challenge of securing configuration Solutions β˜… Apply immutability to pipeline configuration β˜… Offer pipeline templates with secure defaults 48

Slide 49

Slide 49 text

The challenge of securing configuration Solutions β˜… Apply immutability to pipeline configuration β˜… Offer pipeline templates with secure defaults β˜… Test pipelines as code β˜… Secure dependencies that allow arbitrary code / command 49

Slide 50

Slide 50 text

The challenge of securing configuration Solutions β˜… Apply immutability to pipeline configuration β˜… Offer pipeline templates with secure defaults β˜… Test pipelines as code β˜… Secure dependencies that allow arbitrary code / command β˜… Audit changes to pipeline configuration 50

Slide 51

Slide 51 text

Securing CI/CD Pipelines In Summary βœ“ Access Control βœ“ Secrets βœ“ Runners βœ“ Dependencies βœ“ Configuration 51

Slide 52

Slide 52 text

Securing CI/CD Pipelines In Summary βœ“ Access Control βœ“ Secrets βœ“ Runners βœ“ Dependencies βœ“ Configuration 52 Favor immutability.

Slide 53

Slide 53 text

Securing CI/CD Pipelines In Summary βœ“ Access Control βœ“ Secrets βœ“ Runners βœ“ Dependencies βœ“ Configuration 53 Favor immutability. Limit blast radius.

Slide 54

Slide 54 text

Securing CI/CD Pipelines In Summary βœ“ Access Control βœ“ Secrets βœ“ Runners βœ“ Dependencies βœ“ Configuration 54 Favor immutability. Limit blast radius. Automate to reduce friction.

Slide 55

Slide 55 text

Rosemary Wang @joatmon08 joatmon08.github.io thank you! 55