Blasting Through the Clouds: Automating Cloud Foundry with Concourse CI

Cloud Foundry has an extremly high release velocity with new versions being available multiple times every week for a usual deployment. It is important for operators to deploy these releases in a timely manner in order to keep up with security patches and feature improvements. Commonly, there is not only one Cloud Foundry deployment to be kept up to date, but rather a couple of different stages that need to be upgraded in a specific order, for example from a sandbox to a development to a production environment.

Automation is key to keep up to date with Cloud Foundry's release velocity and Concourse CI is the continuous thing-doer of choice to do this honorable task. In this talk we'll first get to know Concourse CI basics and then see how we can leverage Concourse to automate staged platform updates for Pivotal Cloud Foundry. With pcf-automation being sunsetted in favor of PCF Automation we'll have a look at how we can tailor upgrade pipelines to suit different needs all while keeping the thrust at high pace to blast through the clouds!

Transcript

1. Blasting Through the Clouds Automating Cloud Foundry with Concourse CI Photo by Emiel Molenaar on Unsplash
2. About me Fabian Keller Software Engineer Woodworker @_fabiankeller www.fabian-keller.de
3. PCF Releases Sep‘18 Oct‘18 Nov‘18 Dec‘18 Jan‘19 Feb‘19 Mar‘19 Apr‘19 May‘19 Maintenance Release Security Release PCF AS Stemcell 2.3 PCF AS Stemcell 2.4 PCF AS Stemcell 2.5 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 3
4. Then add environments… Bleeding Edge Developers Upgrade Test Operators Sandbox Operators Development Developers Testing Developers Production Developers 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 4
5. Help! We need to automate this... Photo by Alex Knight on Unsplash
6. Concourse An open-source continuous thing-doer
7. Why Concourse? And not [Jenkins | Bamboo | …] • Automate everything! • Controlled and reproducible builds • Easy mechanics à Containers • Automation as Code • Shiny UI (and CLI) 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 7
8. Tasks The basic mechanics of Concourse --- platform: linux image_resource: type: docker-image source: {repository: busybox} run: path: echo args: ["Hello CF Meetup!"] 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 8
9. Tasks == Functions 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 9
10. Task Inputs / Outputs Parameters and return values 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 10 --- platform: linux image_resource: type: docker-image source: {repository: ubuntu} inputs: - name: name outputs: - name: greeting run: path: bash args: - -c - | echo "Hello $(cat ./name/*)!" > ./greeting/greeting.txt
11. Resources Concourse can process any resource • Check resources for new versions • Pull resources at specific versions • Push resources to create new versions • Default resource types available: git, s3, time, semver, docker-image, … resources: - name: project type: git source: uri: ssh://git@gitlab.com/my/repo.git branch: master - name: cache type: s3 source: access_key_id: ((s3_access_key_id)) secret_access_key: ((s3_secret_key)) endpoint: ((s3_endpoint)) region_name: ((s3_region)) bucket: ((s3_bucket)) 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 11
12. Connecting Tasks Pipelines connect resources with tasks… 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 12
13. Connecting Tasks … in a simple YAML file # install-pcf-pipeline.yml resoures: - name: pcf-pipelines type: git source: # ... jobs: - name: create-infrastructure plan: - aggregate: - get: pcf-pipelines - get: terraform-state - task: find-ami # ... - task: create-infrastructure file: pcf-pipelines/tasks/create-infra/task.yml # ... 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 13
14. Handling Secrets • Secrets are passed to tasks as parameters • Concourse fetches secrets from external credentials manager (Vault / Credhub) • No secrets in pipeline/task definitions 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 14 # deploy-to-aws-pipeline.yml jobs: - name: create-infrastructure plan: - task: deploy-on-aws config: params: AWS_ACCESS_KEY: ((aws-access-key)) AWS_SECRET_KEY: ((aws-secret-key)) AWS_REGION: ((aws-region))
15. PCF Automation Status Quo
16. pcf-pipelines A collection of pipelines for installing and upgrading PCF 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 16
17. pcf-pipelines Tile upgrade pipeline Well, that was easy! Maybe too easy? 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 17
18. pcf-pipelines Ready-made tasks PCF Installation • config-ert • config-opsman • configure-ert • configure-product • create-initial-terraform-state • delete-vm Tile Upgrades • disable-errands • toggle-errands • download-boshio-stemcells • download-custom-stemcells-from-s3 • download-pivnet-stemcells • delete-unused-products • stage-and-apply-updates • stage-product • upload-product-and-stemcell • wait-opsman-clear • apply-changes OpsManager Upgrade • deploy-opsman-vm • export-opsmgr-settings • import-opsmgr-settings • replace-vm 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 18
19. pcf-pipelines Is deprecated unfortunately 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 19
20. PCF Automation pcf-pipelines • Currently in beta: http://docs.pivotal.io/platform-automation/ • No reference pipeline for tile upgrades so far • Improved tasks: upload-product-and-stemcell upload-and-stage-product upload-product upload-stemcell assign-stemcell 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 20
21. CONCOURSE GOOD PRACTICE #1 Build reusable tasks … and combine them to tailored pipelines 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 21
22. What do we really need? A tool operators want to work with every day! Photo by Todd Quackenbush on Unsplash
23. How to upgrade CF Upload product, stage product, apply changes. IaaS Operations Manager Concourse Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 23
24. We need environment support Bleeding Edge Upgrade Test Sandbox Development Testing Production Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Isolation Segment Three RabbitMQ for PCF Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two New Relic RabbitMQ for PCF 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 24
25. We need offline support Well, we‘re in Germany after all Concourse Pivotal Network PCF DMZ Download Concourse Pivotal Network S3 Upgrade Concourse PCF 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 25
26. What do we really need? • Multi-environment • Offline Cloud Foundry à Offline Pipelines • Stemcells?! • Really bleeding edge? • Organizational Constraints 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 26
27. Let‘s build it And the journey begins... 6 Photo by Will Suddreth on Unsplash
28. S3 Layout The shared data of our automation setup • Upload: Upload .pivotal tiles once • Info: Extract required metadata and store it for each version • Installed: Copy the .pivotal tiles to each environment folder when deploying ├── upload │ ├── cf │ │ ├── cf-2.4.1.pivotal │ │ └── cf-2.4.2.pivotal │ └── p-isolation-segment │ └── p-iso-2.4.1.pivotal ├── info │ ├── cf │ │ ├── 2.4.1.yml │ │ └── 2.4.2.yml │ └── p-isolation-segment │ └── 2.4.1.yml └── installed ├── dev │ ├── cf │ │ ├── cf-2.4.1.pivotal │ └── p-isolation-segment │ └── p-iso-2.4.1.pivotal └── prod 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 28
29. Triggering Deployments Making it simple to deploy the correct version Operations Manager CF Application Service 2.4.1 Isolation Segment One 2.4.1 MySQL for PCF 1.9.6 Isolation Segment Two 2.4.1 Operations Manager CF Application Service 2.4.3 Isolation Segment One 2.4.3 MySQL for PCF 1.9.8 Isolation Segment Two 2.4.3 cf: 2.4.1 p-isoseg-one: 2.4.1 p-isoseg-two: 2.4.1 p-mysql: 1.9.6 cf: 2.4.3 p-isoseg-one: 2.4.3 p-isoseg-two: 2.4.3 p-mysql: 1.9.8 dev.yml dev.yml 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 29
30. CONCOURSE GOOD PRACTICE #2 Operator Experience (OX) … making it fun and safe to use all the tools 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 30
31. Multi-product Upgrade Pipeline One pipeline to upgrade a whole foundation 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 31
32. We have built some tasks Tailoring the tasks that pcf-pipelines provides Preparation Pipeline • extract-product-info • create-isolation-segment Update Pipeline • validate-versions • product-setup • upload-stemcell • upload-product • stage-product • apply-changes 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 32
33. pcfup We haven‘t open-sourced the tasks, but they are heavily using pcfup 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 33
34. Demo Time! Photo by SpaceX on Unsplash
35. Buildpack Pipelines Building from source and distributing accross environments 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 35
36. Challenges There is still a lot to do • Concourse resource versions • Run pipeline with Version X not possible • Triggers work only good with latest version • Pipeline Complexity • Complex tasks are harder to maintain • At times difficult to oversee all moving parts 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 36
37. Thank you! Questions? 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 37 @_fabiankeller Also check out: https://blog.mimacom.com/tag/cloud-foundry/