Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Blasting Through the Clouds: Automating Cloud F...

Blasting Through the Clouds: Automating Cloud Foundry with Concourse CI

Cloud Foundry has an extremly high release velocity with new versions being available multiple times every week for a usual deployment. It is important for operators to deploy these releases in a timely manner in order to keep up with security patches and feature improvements. Commonly, there is not only one Cloud Foundry deployment to be kept up to date, but rather a couple of different stages that need to be upgraded in a specific order, for example from a sandbox to a development to a production environment.

Automation is key to keep up to date with Cloud Foundry's release velocity and Concourse CI is the continuous thing-doer of choice to do this honorable task. In this talk we'll first get to know Concourse CI basics and then see how we can leverage Concourse to automate staged platform updates for Pivotal Cloud Foundry. With pcf-automation being sunsetted in favor of PCF Automation we'll have a look at how we can tailor upgrade pipelines to suit different needs all while keeping the thrust at high pace to blast through the clouds!

Fabian Kleiser

May 22, 2019
Tweet

More Decks by Fabian Kleiser

Other Decks in Programming

Transcript

  1. PCF Releases Sep‘18 Oct‘18 Nov‘18 Dec‘18 Jan‘19 Feb‘19 Mar‘19 Apr‘19

    May‘19 Maintenance Release Security Release PCF AS Stemcell 2.3 PCF AS Stemcell 2.4 PCF AS Stemcell 2.5 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 3
  2. Then add environments… Bleeding Edge Developers Upgrade Test Operators Sandbox

    Operators Development Developers Testing Developers Production Developers 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 4
  3. Why Concourse? And not [Jenkins | Bamboo | …] •

    Automate everything! • Controlled and reproducible builds • Easy mechanics à Containers • Automation as Code • Shiny UI (and CLI) 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 7
  4. Tasks The basic mechanics of Concourse --- platform: linux image_resource:

    type: docker-image source: {repository: busybox} run: path: echo args: ["Hello CF Meetup!"] 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 8
  5. Task Inputs / Outputs Parameters and return values 2019-05-22 /

    CF Meetup Stuttgart / Fabian Keller 10 --- platform: linux image_resource: type: docker-image source: {repository: ubuntu} inputs: - name: name outputs: - name: greeting run: path: bash args: - -c - | echo "Hello $(cat ./name/*)!" > ./greeting/greeting.txt
  6. Resources Concourse can process any resource • Check resources for

    new versions • Pull resources at specific versions • Push resources to create new versions • Default resource types available: git, s3, time, semver, docker-image, … resources: - name: project type: git source: uri: ssh://[email protected]/my/repo.git branch: master - name: cache type: s3 source: access_key_id: ((s3_access_key_id)) secret_access_key: ((s3_secret_key)) endpoint: ((s3_endpoint)) region_name: ((s3_region)) bucket: ((s3_bucket)) 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 11
  7. Connecting Tasks … in a simple YAML file # install-pcf-pipeline.yml

    resoures: - name: pcf-pipelines type: git source: # ... jobs: - name: create-infrastructure plan: - aggregate: - get: pcf-pipelines - get: terraform-state - task: find-ami # ... - task: create-infrastructure file: pcf-pipelines/tasks/create-infra/task.yml # ... 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 13
  8. Handling Secrets • Secrets are passed to tasks as parameters

    • Concourse fetches secrets from external credentials manager (Vault / Credhub) • No secrets in pipeline/task definitions 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 14 # deploy-to-aws-pipeline.yml jobs: - name: create-infrastructure plan: - task: deploy-on-aws config: params: AWS_ACCESS_KEY: ((aws-access-key)) AWS_SECRET_KEY: ((aws-secret-key)) AWS_REGION: ((aws-region))
  9. pcf-pipelines A collection of pipelines for installing and upgrading PCF

    2019-05-22 / CF Meetup Stuttgart / Fabian Keller 16
  10. pcf-pipelines Tile upgrade pipeline Well, that was easy! Maybe too

    easy? 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 17
  11. pcf-pipelines Ready-made tasks PCF Installation • config-ert • config-opsman •

    configure-ert • configure-product • create-initial-terraform-state • delete-vm Tile Upgrades • disable-errands • toggle-errands • download-boshio-stemcells • download-custom-stemcells-from-s3 • download-pivnet-stemcells • delete-unused-products • stage-and-apply-updates • stage-product • upload-product-and-stemcell • wait-opsman-clear • apply-changes OpsManager Upgrade • deploy-opsman-vm • export-opsmgr-settings • import-opsmgr-settings • replace-vm 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 18
  12. PCF Automation pcf-pipelines • Currently in beta: http://docs.pivotal.io/platform-automation/ • No

    reference pipeline for tile upgrades so far • Improved tasks: upload-product-and-stemcell upload-and-stage-product upload-product upload-stemcell assign-stemcell 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 20
  13. CONCOURSE GOOD PRACTICE #1 Build reusable tasks … and combine

    them to tailored pipelines 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 21
  14. What do we really need? A tool operators want to

    work with every day! Photo by Todd Quackenbush on Unsplash
  15. How to upgrade CF Upload product, stage product, apply changes.

    IaaS Operations Manager Concourse Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 23
  16. We need environment support Bleeding Edge Upgrade Test Sandbox Development

    Testing Production Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Isolation Segment Three RabbitMQ for PCF Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two Operations Manager Cloud Foundry Application Service Isolation Segment One MySQL for PCF Isolation Segment Two New Relic RabbitMQ for PCF 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 24
  17. We need offline support Well, we‘re in Germany after all

    Concourse Pivotal Network PCF DMZ Download Concourse Pivotal Network S3 Upgrade Concourse PCF 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 25
  18. What do we really need? • Multi-environment • Offline Cloud

    Foundry à Offline Pipelines • Stemcells?! • Really bleeding edge? • Organizational Constraints 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 26
  19. S3 Layout The shared data of our automation setup •

    Upload: Upload .pivotal tiles once • Info: Extract required metadata and store it for each version • Installed: Copy the .pivotal tiles to each environment folder when deploying ├── upload │ ├── cf │ │ ├── cf-2.4.1.pivotal │ │ └── cf-2.4.2.pivotal │ └── p-isolation-segment │ └── p-iso-2.4.1.pivotal ├── info │ ├── cf │ │ ├── 2.4.1.yml │ │ └── 2.4.2.yml │ └── p-isolation-segment │ └── 2.4.1.yml └── installed ├── dev │ ├── cf │ │ ├── cf-2.4.1.pivotal │ └── p-isolation-segment │ └── p-iso-2.4.1.pivotal └── prod 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 28
  20. Triggering Deployments Making it simple to deploy the correct version

    Operations Manager CF Application Service 2.4.1 Isolation Segment One 2.4.1 MySQL for PCF 1.9.6 Isolation Segment Two 2.4.1 Operations Manager CF Application Service 2.4.3 Isolation Segment One 2.4.3 MySQL for PCF 1.9.8 Isolation Segment Two 2.4.3 cf: 2.4.1 p-isoseg-one: 2.4.1 p-isoseg-two: 2.4.1 p-mysql: 1.9.6 cf: 2.4.3 p-isoseg-one: 2.4.3 p-isoseg-two: 2.4.3 p-mysql: 1.9.8 dev.yml dev.yml 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 29
  21. CONCOURSE GOOD PRACTICE #2 Operator Experience (OX) … making it

    fun and safe to use all the tools 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 30
  22. Multi-product Upgrade Pipeline One pipeline to upgrade a whole foundation

    2019-05-22 / CF Meetup Stuttgart / Fabian Keller 31
  23. We have built some tasks Tailoring the tasks that pcf-pipelines

    provides Preparation Pipeline • extract-product-info • create-isolation-segment Update Pipeline • validate-versions • product-setup • upload-stemcell • upload-product • stage-product • apply-changes 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 32
  24. pcfup We haven‘t open-sourced the tasks, but they are heavily

    using pcfup 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 33
  25. Challenges There is still a lot to do • Concourse

    resource versions • Run pipeline with Version X not possible • Triggers work only good with latest version • Pipeline Complexity • Complex tasks are harder to maintain • At times difficult to oversee all moving parts 2019-05-22 / CF Meetup Stuttgart / Fabian Keller 36
  26. Thank you! Questions? 2019-05-22 / CF Meetup Stuttgart / Fabian

    Keller 37 @_fabiankeller Also check out: https://blog.mimacom.com/tag/cloud-foundry/