Securing Jenkins X

The Jenkins X platform consists of a number of microservices running on Kubernetes, which work together to provide users a cloud-native CI/CD experience. In this talk, we will walk you through a number of security features and good practices that allow us to achieve a more secure Jenkins X setup.

The topics covered are:
- Learning about some security features available in Jenkins X.
- Ensuring your cluster is secure before the installation with the scan cluster command.
- Diving into the recommended configuration for a secure Jenkins X installation.
- Adopting best practices for handling secrets in Jenkins X pipeline.
- Managing secrets using the HashiCorp Vault integration.
- Configuring the single sign-on using the SSO add-on and operator easily.

Transcript

1. Securing Jenkins X Cosmin Cojocar
2. © 2019 All Rights Reserved. 2
3. © 2019 All Rights Reserved. 3 Agenda • Security Principles • Cloud Security • Kubernetes Security • Securing the Jenkins X Installation • Overview • Enable Security Features • Securing Git Providers • Handling Secrets in the Jenkins X Pipeline • Configure Single Sign-On using the sso app • Q&A
4. © 2019 All Rights Reserved. 4 Security Principles 1/2 • Minimize attack surface • Apply defence in depth • Least privilege • Establish secure defaults • Fail-safe defaults (fail securely) Source: OWASP
5. © 2019 All Rights Reserved. 5 Security Principles 2/2 • Keep security simple • Avoid security by obscurity (open design) • Assume breach (fast secrets rotation, quick recovery) • Detect intrusions • Assume zero trust Source: OWASP
6. Cloud and Kubernetes Security
7. © 2019 All Rights Reserved. 7 Cloud Security • jx runs in the cloud (Kubernetes) and consumes various cloud services • First line of defence • Isolate the jx cluster and the cloud services • Set up restrictive access policies (not always easy) • Monitor and audit the access • Use configuration as code to manage the cloud resources (e.g. terraform)
8. © 2019 All Rights Reserved. 8 Kubernetes Security • jx runs on Kubernetes • Second line of defence • Managed Kubernetes services quite secure by default • Scan and adjust the default config (e.g. using kube-hunter) jx scan cluster Source: Kubernetes Security Book
9. © 2019 All Rights Reserved. 9 Scan Results (e.g. GKE) vulnerabilities: - category: Access Risk vulnerability: CAP_NET_RAW Enabled - category: Access Risk vulnerability: Read access to pod's service account token - category: Access Risk vulnerability: Access to pod's secrets - category: Information Disclosure vulnerability: K8s Version Disclosure - category: Access Risk vulnerability: Privileged Container - category: Information Disclosure vulnerability: Cluster Health Disclosure - category: Information Disclosure vulnerability: Exposed Pods - category: Unauthenticated Access vulnerability: Unauthenticated access to API
10. © 2019 All Rights Reserved. 10 Kubernetes Security Building Blocks • Service Accounts and other Authentication methods (authentication) • Role-Based Access Control - RBAC (authorization) • Secrets • Pod Security Policies (run containers securely) • Network Policies (define network boundaries) • Admission Controllers (run trusted containers images) • Audit Logging • A good start is the Kubernetes Security book
11. Jenkins X Security
12. © 2019 All Rights Reserved. 12 Jenkins X Topologies Static Serverless Classic Jenkins Jenkinsfile Runner Tekton
13. © 2019 All Rights Reserved. 13 Architecture
14. © 2019 All Rights Reserved. 14 Security Perspective
15. © 2019 All Rights Reserved. 15 Security Challenges 1/2 • Collection of microservices (Kubernetes controllers) interacting with each other • Large amount of integrations • Cloud Services (require different cloud permissions) • External Services (e.g. Git Providers: GitHub etc.) • Pipeline is an execution engine (many commands in the builder containers) • Serverless architecture - builds trigger on events (Denial of Service)
16. © 2019 All Rights Reserved. 16 Security Challenges 2/2 • jx CLI runs locally as well as in the cluster (consistent permissions across Kubernetes and Cloud Providers) • jx CLI has many commands (large surface, some of them endanger security) • auth and authz for integrated tools (e.g. git, vault cli) • jx installation consists of many container images (patch, scan, minimize images) • Exposed services need to be secured (chartmuseum, nexus, docker registry etc.) • Secrets management (Jenkins X installation, pipeline)
17. © 2019 All Rights Reserved. 17 Security Features 1/2 • Domain registration via external-dns and Cloud DNS • Automatic TLS certificates provisioned for all exposed endpoints (issued by Let’s Encrypt CA) • jx CLI uses the same authentication/authorization like kubectl (kubeconfig) • HashiCorp Vault for secrets management • Interpolation of secrets from Vault into Helm chart values during deployment
18. © 2019 All Rights Reserved. 18 Security Features 2/2 • Segregation of Cloud access permissions (transition to managed identity) • Private git repositories (on GitHub) • Access control for build triggering • Branch protection for GitHub • Secrets masking in logs (only partially) • Rootless container image builds (via kaniko)
19. © 2019 All Rights Reserved. 19 Enable Security Features jx boot --requirements=jx-requirements.yaml
20. © 2019 All Rights Reserved. 20 Enable TLS and Domain Registration ingress: cloud_dns_secret_name: external-dns-gcp-sa domain: cluster-domain.jenkins-x.example externalDNS: true namespaceSubDomain: -jx. tls: email: my-email@jenkins-x.example enabled: true production: true See: Documentation
21. © 2019 All Rights Reserved. 21 Enable Vault 1/2 • Entirely automated provisioning • Kubernetes authentication • Cloud Backend (storage and encryption/decryption via KMS) • Cloud auto-unsealing • Exposed over TLS
22. © 2019 All Rights Reserved. 22 Enable Vault 2/2 secretStorage: vault Reuse existing vault resources: vault: name: my-vault bucket: my-vault-bucket keyring: my-vault-keyring serviceAccount: my-vault-cloud-service-account recreateBucket: false disableURLDiscovery: false See: Documentation
23. © 2019 All Rights Reserved. 23 Secrets Interpolation into Helm Charts (boot config) env/parameters.yaml adminUser: password: vault:mycluster/adminUser:password username: admin pipelineUser: token: vault:mycluster/pipelineUser:token username: pipeline-user prow: hmacToken: vault:mycluser/prow:hmacToken
24. © 2019 All Rights Reserved. 24 Secrets Interpolation into Helm Charts (values) jx step helm apply chart/values.yaml service: password: vault:mychart/service:password username: admin
25. Git Providers Security
26. © 2019 All Rights Reserved. 26 Git Providers Integration (GitHub)
27. © 2019 All Rights Reserved. 27 Securing Git Providers (GitHub) 1/2 • Configure private repositories by default • Enable TLS on webhook endpoint to protect the shared secret • Use an organisation instead of an user account • Create a dedicated bot token and assign it as a collaborator to the organisation
28. © 2019 All Rights Reserved. 28 Securing Git Providers (GitHub) 2/2 • Use minimum set of scopes for bot token: repo,read:user,read:org,user:email,write:repo_hook,delete_repo • Add all developer accounts as collaborators of the organisation • Maintain a list of reviewers and approvers for each repository • Define a secrets baseline on each repository as a pre-commit hook to avoid leaking secrets into git repositories (see Yelp/detect-secrets and jx secrets baseline)
29. © 2019 All Rights Reserved. 29 Configure Git Providers (Organisation) jx-requirements.yaml cluster: environmentGitOwner: myconfig-org environmentGitPublic: false gitKind: github gitName: github gitServer: https://github.com
30. © 2019 All Rights Reserved. 30 Configure Git Providers (Bot User and Webhook) env/parameters.yaml pipelineUser: email: mybot-user@jenkins-x.example token: vault:mycluster/pipelineUser:token username: mybot-user prow: hmacToken: vault:mycluster/prow:hmacToken
31. © 2019 All Rights Reserved. 31 Access Control for Build Triggering repository/OWNERS approvers: - developer1 - developer2 reviewers: - developer1 - developer2
32. © 2019 All Rights Reserved. 32 Access Control for Build Triggering • OWNERS file is stored in the root of each repository • Pull request pipeline doesn’t start automatically for contributors who are not in the list, requires /ok-to-test from someone in the list • DoS protection for open repositories • reviewers can comment on a pull request with /lgtm • approvers can approve a pull request with /approve • A pull request gets automatically merged when both labels are available
33. Jenkins X Pipeline Secrets
34. © 2019 All Rights Reserved. 34 Secrets in Jenkins X Pipeline (Kubernetes Secrets) buildPack: none pipelineConfig: pipelines: pullRequest: stages: - name: ci options: volumes: - name: ci-secret-volume secret: secretName: ci-secret items: - key: ci-secret.json path: ci/secret.json containerOptions: volumeMounts: - mountPath: /secrets name: ci-secret-volume steps: - command: ci.sh name: runci /secrets/ci/secret.json
35. © 2019 All Rights Reserved. 35 Secrets in Jenkins X Pipeline (Kubernetes Secrets) buildPack: none pipelineConfig: pipelines: pullRequest: pipeline: environment: - name: CI_TOKEN valueFrom: secretKeyRef: name: ci-token key: password stages: - name: ci steps: - command: ci.sh name: runci
36. © 2019 All Rights Reserved. 36 Secrets in Jenkins X Pipeline (Vault Secrets) • Kubernetes CSI: Vault Provider for Secret Store CSI Driver • Need to be installed and configured in the cluster (see documentation) • Allow to consume secrets from vault in a similar way as Kubernetes secrets • Require Kubernetes Version 1.15.x+ Source: github.com/deislabs/secrets-store-csi-driver
37. © 2019 All Rights Reserved. 37 Secrets in Jenkins X Pipeline (Vault Secrets) buildPack: none pipelineConfig: pipelines: pullRequest: stages: - name: ci options: volumes: - name: vault-secrets-volume csi: driver: secrets-store.csi.k8s.com volumeAttributes: secretProviderClass: vault-secrets containerOptions: volumeMounts: - name: vault-secrets-volume mountPath: /secrets steps: - command: ci.sh name: runci apiVersion: secrets-store.csi.k8s.com/v1alpha1 kind: SecretProviderClass metadata: name: vault-secrets spec: provider: vault parameters: roleName: "vault-app-role" vaultAddress: "http://jx-vault-mycluster:8200" vaultSkipTLSVerify: "false" objects: | array: - | objectPath: "/mycluster/adminUser" objectName: "password" objectVersion: ""
38. © 2019 All Rights Reserved. 38 Recommendations • Avoid using custom commands to consume secrets in the pipeline • jx step credential • jx step git credentials • You risk to accidentally leak the credentials into the git repository • Tekton pipeline fixes up the git credentials before every build (see Tekton Auth) • Avoid logging secrets (current masking capabilities are not so smart) • Rotate the credentials regularly
39. Single Sign-On Operator
40. © 2019 All Rights Reserved. 40 Configure Single Sign-On • Generic way to automatically enable SSO for any exposed HTTPS service • Check github.com/jenkins-x/sso-operator • Can be installed as an app in Jenkins X • Pre-configured identity providers ◦ Google ◦ GitHub
41. © 2019 All Rights Reserved. 41 Single Sign-On App jx add app jx-app-sso
42. © 2019 All Rights Reserved. 42 SSO Definition apiVersion: "jenkins.io/v1" kind: "SSO" metadata: name: "sso-golang-http" namespace: jx-staging spec: oidcIssuerUrl: "https://dex.jx-staging.example.com" upstreamService: "golang-http" forwardToken: false domain: "example.com" certIssuerName: "letsencrypt-prod" skipExposeService: false urlTemplate: "{{.Service}}.{{.Namespace}}.{{.Domain}}" cookieSpec: name: "sso-golang-http" expire: "168h" refresh: "60m" secure: true httpOnly: true proxyImage: "quay.io/pusher/oauth2_proxy" proxyImageTag: 4.0.0"
43. © 2019 All Rights Reserved. 43 Q&A
44. © 2019 All Rights Reserved. 44 Thank You! Cosmin Cojocar ccojocar @CosminCojocar ccojocar@cloudbees.com