Minimum Secure Pipeline

Originally presented at HashiTalks: Secure 2023.

Whether you are a developer, operator, or other kind of engineer, you will need to secure your continuous integration frameworks and deployment pipelines. Rosemary Wang provides a list of practices and patterns to build a minimum secure pipeline. From access control to pipelines as code, you can apply these basics to improving your CI/CD pipelines and educating other engineers, no matter what you're deploying.

Transcript

1. Rosemary Wang | May 3, 2023
   Minimum Secure Pipeline
   1
   

   View Slide

2. jenkins.io/security/advisory/2022-02-15/
   about.codecov.io/security-update/
   msrc-blog.microsoft.com/2020/12/31/microsoft-internal-solorigate-investigation-update/
   2
   

   View Slide

3. Rosemary Wang (she/her)
   
   
   Developer Advocate at HashiCorp
   
   
   @joatmon08
   
   
   joatmon08.github.io
   3
   

   View Slide

4. How to secure better…
   
   
   and remediate faster.
   4
   

   View Slide

5. Securing CI/CD Pipelines
   An Overview
   ✓ Access Control
   
   ✓ Secrets
   
   ✓ Runners
   
   ✓ Dependencies
   
   ✓ Configuration
   5
   

   View Slide

6. Access Control 🚥
   6
   

   View Slide

7. limit service’s access rights to minimum required
   least privilege
   7
   

   View Slide

8. Choose your access.
   Code Repository
   Build infrastructure. Infrastructure Provider
   Checkout code.
   Deploy application.
   Application Platform /
   Release Repository
   Test for security. Security Tool
   Check code quality. Quality Assurance Tool
   Read repositories.
   Write speci
   fi
   c services.
   Write speci
   fi
   c nodes or
   namespaces.
   Read test results.
   Read test results.
   Run integration tests. Store for User Data Read table in testing
   environment.
   8
   

   View Slide

9. The challenge of least privilege
   Solutions
   ★ Limit access from pipeline early in development
   
   ★ Limit write access to repositories early in development
   9
   

   View Slide

10. The challenge of least privilege
    Solutions
    ★ Limit access from pipeline early in development
    
    ★ Limit write access to repositories early in development
    
    ★ Offer self-service to refine access
    
    ★ Offer templates of secure policies
    10
    

    View Slide

11. The challenge of least privilege
    Solutions
    ★ Limit access from pipeline early in development
    
    ★ Limit write access to repositories early in development
    
    ★ Offer self-service to refine access
    
    ★ Offer templates of secure policies
    
    ★ Review access on a regular cadence
    
    ★ Audit pipeline runs
    11
    

    View Slide

12. Secrets 🔒
    12
    

    View Slide

13. sensitive information linked to access to a system or service
    secrets
    13
    

    View Slide

14. vault_database_secret_backend_connection.post
    gres will be created
    • resource
    "vault_database_secret_backend_connection"
    "postgres" {

    ◦ postgresql {
    ▪ connection_url = "postgres://
    hcpvault:ZWtW62okZyJh@terraform-2020113
    0215226595400000001.cho1mmdxhp1z.us-
    west-2.rds.amazonaws.com:5432/prod"
    PIPELINE LOGS
    [UNIT TEST]
    TERRAFORM
    FMT
    [BUILD]
    TERRAFORM
    INIT
    [DEPLOY]
    TERRAFORM
    PLAN
    [RELEASE]
    TERRAFORM
    APPLY
    [TEST]
    14
    😱
    

    View Slide

15. Plan R
    Remediate the secret
    • Regret
    
    • Revoke
    
    • Rotate
    
    • Reference
    
    • Replace
    
    • Re-run
    15
    100 pipelines later… 😓
    

    View Slide

16. Pipelines use secrets.
    16
    Certi
    fi
    cates
    Access Usernames & Passwords
    Testing User Data
    Tokens
    SSH Keys
    Encryption Keys
    

    View Slide

17. Pipelines create secrets.
    17
    Con
    fi
    guration Usernames & Passwords
    SSH Keys
    Tokens
    

    View Slide

18. The challenge of secrets
    Solutions
    ★ Mask or omit in pipeline output
    
    ★ Use a secrets manager
    18
    

    View Slide

19. The challenge of secrets
    Solutions
    ★ Mask or omit in pipeline output
    
    ★ Use a secrets manager
    
    ★ Issue new credentials per pipeline run
    
    ★ Audit secrets usage
    19
    

    View Slide

20. Runners 👟
    20
    

    View Slide

21. resources that run pipeline stages or tasks
    runners
    21
    

    View Slide

22. 22
    Virtual
    Machine
    🤔
    1. Someone accesses CI/CD runner (e.g., SSH).
    2. Access other infrastructure.
    Database
    

    View Slide

23. 23
    Container
    🤔
    1. Someone accesses CI/CD runner (e.g., SSH).
    Virtual
    Machine
    $ mount /dev/ /mnt
    
    
    $ chroot /mnt
    2. Container can access host
    fi
    lesystem.
    3. Access code or
    fi
    les for other jobs.
    Other Jobs
    on Virtual
    Machine
    

    View Slide

24. 24
    Infrastructure Provider
    Runner Managed Service
    Engineer
    Must be
    authorized
    user.
    Must be
    authorized
    account.
    Secrets Manager
    Allow IP
    address over
    VPN.
    Allow IP
    addresses for CI
    framework.
    

    View Slide

25. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images
    
    ★ Scan for OS vulnerabilities
    25
    

    View Slide

26. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images
    
    ★ Scan for OS vulnerabilities
    
    ★ Define network policy
    
    ★ Run as a non-root
    26
    

    View Slide

27. The challenge of securing runners
    Solutions
    ★ Use trusted / verified images
    
    ★ Scan for OS vulnerabilities
    
    ★ Define network policy
    
    ★ Run as a non-root
    
    ★ Use ephemeral secrets
    
    ★ Audit remote access to runner
    27
    

    View Slide

28. Dependencies 🖇
    28
    

    View Slide

29. third-party code used for pipeline stages or tasks
    dependencies
    29
    

    View Slide

30. 30
    name: release
    
    
    jobs:
    
    
    goreleaser:
    
    
    runs-on: ubuntu-latest
    
    
    steps:
    
    
    - name: Checkout
    
    
    uses: actions/checkout@v2
    
    
    with:
    
    
    fetch-depth: 0
    
    
    - name: Set up Go
    
    
    uses: actions/setup-go@v2
    
    
    with:
    
    
    go-version: 1.14
    
    
    - name: Run GoReleaser
    
    
    uses: goreleaser/goreleaser-action@v2
    
    
    with:
    
    
    version: latest
    
    
    args: release --rm-dist
    
    
    Downloaded from trusted source?
    Veri
    fi
    ed code?
    Correct plugin?
    

    View Slide

31. 31
    public class UnverifiedPlugin {
    
    
    protected static void getFiles(AbstractBuild b,
    
    
    FilePath workspace) {
    
    
    // code to replace project files or metadata
    
    
    // code to gather information
    
    
    // code to siphon credentials
    
    
    }
    
    
    }
    

    View Slide

32. The challenge of securing dependencies
    Solutions
    ★ Scan for vulnerabilities
    
    ★ Verify checksums and signatures
    32
    

    View Slide

33. The challenge of securing dependencies
    Solutions
    ★ Scan for vulnerabilities
    
    ★ Verify checksums and signatures
    
    ★ Use verified registry
    
    ★ Pin versions
    33
    

    View Slide

34. Configuration 📄
    34
    

    View Slide

35. de
    fi
    ne delivery pipelines through source code
    pipeline as
    code
    35
    

    View Slide

36. 36
    name: release
    
    
    jobs:
    
    
    goreleaser:
    
    
    runs-on: ubuntu-latest
    
    
    steps:
    
    
    - name: Checkout
    
    
    uses: actions/checkout@v2
    
    
    with:
    
    
    fetch-depth: 0
    
    
    - name: Set up Go
    
    
    uses: actions/setup-go@v2
    
    
    with:
    
    
    go-version: 1.14
    
    
    - name: Run GoReleaser
    
    
    uses: goreleaser/goreleaser-action@v2
    
    
    with:
    
    
    version: latest
    
    
    args: release --rm-dist
    
    
    test_plugin_checkout_has_fetch_depth_of_1
    test_plugin_go_uses_secure_version_1.14
    test_plugin_release_includes_signature
    

    View Slide

37. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration
    
    ★ Offer pipeline templates with secure defaults
    37
    

    View Slide

38. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration
    
    ★ Offer pipeline templates with secure defaults
    
    ★ Test pipelines as code
    
    ★ Secure dependencies that allow arbitrary code / command
    38
    

    View Slide

39. The challenge of securing configuration
    Solutions
    ★ Apply immutability to pipeline configuration
    
    ★ Offer pipeline templates with secure defaults
    
    ★ Test pipelines as code
    
    ★ Secure dependencies that allow arbitrary code / command
    
    ★ Audit changes to pipeline configuration
    39
    

    View Slide

40. Securing CI/CD Pipelines
    In Summary
    ✓ Access Control
    
    ✓ Secrets
    
    ✓ Runners
    
    ✓ Dependencies
    
    ✓ Configuration
    40
    

    View Slide

41. Securing CI/CD Pipelines
    In Summary
    ✓ Access Control
    
    ✓ Secrets
    
    ✓ Runners
    
    ✓ Dependencies
    
    ✓ Configuration
    41
    Favor immutability.
    Limit blast radius.
    Automate to reduce friction.
    

    View Slide

42. Rosemary Wang
    @joatmon08
    joatmon08.github.io
    thank you!
    42
    

    View Slide