$30 off During Our Annual Pro Sale. View Details »

Minimum Secure Pipeline

Minimum Secure Pipeline

Originally presented at HashiTalks: Secure 2023.

Whether you are a developer, operator, or other kind of engineer, you will need to secure your continuous integration frameworks and deployment pipelines. Rosemary Wang provides a list of practices and patterns to build a minimum secure pipeline. From access control to pipelines as code, you can apply these basics to improving your CI/CD pipelines and educating other engineers, no matter what you're deploying.

Rosemary Wang

May 03, 2023
Tweet

More Decks by Rosemary Wang

Other Decks in Technology

Transcript

  1. Rosemary Wang | May 3, 2023
    Minimum Secure Pipeline
    1

    View Slide

  2. jenkins.io/security/advisory/2022-02-15/
    about.codecov.io/security-update/
    msrc-blog.microsoft.com/2020/12/31/microsoft-internal-solorigate-investigation-update/
    2

    View Slide

  3. Rosemary Wang (she/her)


    Developer Advocate at HashiCorp


    @joatmon08


    joatmon08.github.io
    3

    View Slide

  4. How to secure better…


    and remediate faster.
    4

    View Slide

  5. Securing CI/CD Pipelines
    An Overview
    ✓ Access Control

    ✓ Secrets

    ✓ Runners

    ✓ Dependencies

    ✓ Configuration
    5

    View Slide

  6. Access Control 🚥
    6

    View Slide

  7. limit service’s access rights to minimum required
    least privilege
    7

    View Slide

  8. Choose your access.
    Code Repository
    Build infrastructure. Infrastructure Provider
    Checkout code.
    Deploy application.
    Application Platform /
    Release Repository
    Test for security. Security Tool
    Check code quality. Quality Assurance Tool
    Read repositories.
    Write speci
    fi
    c services.
    Write speci
    fi
    c nodes or
    namespaces.
    Read test results.
    Read test results.
    Run integration tests. Store for User Data Read table in testing
    environment.
    8

    View Slide

  9. The challenge of least privilege
    Solutions
    ★ Limit access from pipeline early in development

    ★ Limit write access to repositories early in development
    9

    View Slide

  10. The challenge of least privilege
    Solutions
    ★ Limit access from pipeline early in development

    ★ Limit write access to repositories early in development

    ★ Offer self-service to refine access

    ★ Offer templates of secure policies
    10

    View Slide

  11. The challenge of least privilege
    Solutions
    ★ Limit access from pipeline early in development

    ★ Limit write access to repositories early in development

    ★ Offer self-service to refine access

    ★ Offer templates of secure policies

    ★ Review access on a regular cadence

    ★ Audit pipeline runs
    11

    View Slide

  12. Secrets 🔒
    12

    View Slide

  13. sensitive information linked to access to a system or service
    secrets
    13

    View Slide

  14. vault_database_secret_backend_connection.post
    gres will be created
    • resource
    "vault_database_secret_backend_connection"
    "postgres" {

    ◦ postgresql {
    ▪ connection_url = "postgres://
    hcpvault:ZWtW62okZyJh@terraform-2020113
    0215226595400000001.cho1mmdxhp1z.us-
    west-2.rds.amazonaws.com:5432/prod"
    PIPELINE LOGS
    [UNIT TEST]
    TERRAFORM
    FMT
    [BUILD]
    TERRAFORM
    INIT
    [DEPLOY]
    TERRAFORM
    PLAN
    [RELEASE]
    TERRAFORM
    APPLY
    [TEST]
    14
    😱

    View Slide

  15. Plan R
    Remediate the secret
    • Regret

    • Revoke

    • Rotate

    • Reference

    • Replace

    • Re-run
    15
    100 pipelines later… 😓

    View Slide

  16. Pipelines use secrets.
    16
    Certi
    fi
    cates
    Access Usernames & Passwords
    Testing User Data
    Tokens
    SSH Keys
    Encryption Keys

    View Slide

  17. Pipelines create secrets.
    17
    Con
    fi
    guration Usernames & Passwords
    SSH Keys
    Tokens

    View Slide

  18. The challenge of secrets
    Solutions
    ★ Mask or omit in pipeline output

    ★ Use a secrets manager
    18

    View Slide

  19. The challenge of secrets
    Solutions
    ★ Mask or omit in pipeline output

    ★ Use a secrets manager

    ★ Issue new credentials per pipeline run

    ★ Audit secrets usage
    19

    View Slide

  20. Runners 👟
    20

    View Slide

  21. resources that run pipeline stages or tasks
    runners
    21

    View Slide

  22. 22
    Virtual
    Machine
    🤔
    1. Someone accesses CI/CD runner (e.g., SSH).
    2. Access other infrastructure.
    Database

    View Slide

  23. 23
    Container
    🤔
    1. Someone accesses CI/CD runner (e.g., SSH).
    Virtual
    Machine
    $ mount /dev/ /mnt


    $ chroot /mnt
    2. Container can access host
    fi
    lesystem.
    3. Access code or
    fi
    les for other jobs.
    Other Jobs
    on Virtual
    Machine

    View Slide

  24. 24
    Infrastructure Provider
    Runner Managed Service
    Engineer
    Must be
    authorized
    user.
    Must be
    authorized
    account.
    Secrets Manager
    Allow IP
    address over
    VPN.
    Allow IP
    addresses for CI
    framework.

    View Slide

  25. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images

    ★ Scan for OS vulnerabilities
    25

    View Slide

  26. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images

    ★ Scan for OS vulnerabilities

    ★ Define network policy

    ★ Run as a non-root
    26

    View Slide

  27. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images

    ★ Scan for OS vulnerabilities

    ★ Define network policy

    ★ Run as a non-root

    ★ Use ephemeral secrets

    ★ Audit remote access to runner
    27

    View Slide

  28. Dependencies 🖇
    28

    View Slide

  29. third-party code used for pipeline stages or tasks
    dependencies
    29

    View Slide

  30. 30
    name: release


    jobs:


    goreleaser:


    runs-on: ubuntu-latest


    steps:


    - name: Checkout


    uses: actions/checkout@v2


    with:


    fetch-depth: 0


    - name: Set up Go


    uses: actions/setup-go@v2


    with:


    go-version: 1.14


    - name: Run GoReleaser


    uses: goreleaser/goreleaser-action@v2


    with:


    version: latest


    args: release --rm-dist


    Downloaded from trusted source?
    Veri
    fi
    ed code?
    Correct plugin?

    View Slide

  31. 31
    public class UnverifiedPlugin {


    protected static void getFiles(AbstractBuild b,


    FilePath workspace) {


    // code to replace project files or metadata


    // code to gather information


    // code to siphon credentials


    }


    }

    View Slide

  32. The challenge of securing dependencies
    Solutions
    ★ Scan for vulnerabilities

    ★ Verify checksums and signatures
    32

    View Slide

  33. The challenge of securing dependencies
    Solutions
    ★ Scan for vulnerabilities

    ★ Verify checksums and signatures

    ★ Use verified registry

    ★ Pin versions
    33

    View Slide

  34. Configuration 📄
    34

    View Slide

  35. de
    fi
    ne delivery pipelines through source code
    pipeline as
    code
    35

    View Slide

  36. 36
    name: release


    jobs:


    goreleaser:


    runs-on: ubuntu-latest


    steps:


    - name: Checkout


    uses: actions/checkout@v2


    with:


    fetch-depth: 0


    - name: Set up Go


    uses: actions/setup-go@v2


    with:


    go-version: 1.14


    - name: Run GoReleaser


    uses: goreleaser/goreleaser-action@v2


    with:


    version: latest


    args: release --rm-dist


    test_plugin_checkout_has_fetch_depth_of_1
    test_plugin_go_uses_secure_version_1.14
    test_plugin_release_includes_signature

    View Slide

  37. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration

    ★ Offer pipeline templates with secure defaults
    37

    View Slide

  38. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration

    ★ Offer pipeline templates with secure defaults

    ★ Test pipelines as code

    ★ Secure dependencies that allow arbitrary code / command
    38

    View Slide

  39. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration

    ★ Offer pipeline templates with secure defaults

    ★ Test pipelines as code

    ★ Secure dependencies that allow arbitrary code / command

    ★ Audit changes to pipeline configuration
    39

    View Slide

  40. Securing CI/CD Pipelines
    In Summary
    ✓ Access Control

    ✓ Secrets

    ✓ Runners

    ✓ Dependencies

    ✓ Configuration
    40

    View Slide

  41. Securing CI/CD Pipelines
    In Summary
    ✓ Access Control

    ✓ Secrets

    ✓ Runners

    ✓ Dependencies

    ✓ Configuration
    41
    Favor immutability.
    Limit blast radius.
    Automate to reduce friction.

    View Slide

  42. Rosemary Wang
    @joatmon08
    joatmon08.github.io
    thank you!
    42

    View Slide