The Challenges of Taking Open Source Cloud Foundry to Production

When committing to Cloud Foundry some companies make the strategic decision to take full ownership of platform deployment and operations and deploy open source Cloud Foundry to fully understand all moving parts. While open-source Cloud Foundry has gotten rather easy to delpoy with the cf-deployment project, there is still a lot of more to do in order to make an open-source deployment production ready.

This talk will show the challenges we've gone through in taking open-source Cloud Foundry to production. We'll see what it takes to run and keep the platform up to date including various operational aspects such as monitoring, logging and backing up the platform. After the talk you'll have a better understanding of the effort it takes to run the open source distribution.

Transcript

1. The Challenges of Taking Open Source Cloud Foundry to Production // Fabian Keller
2. Fabian Keller Software Engineer @mimacom Woodworker @127.0.0.1 # CLOUD PLATFORMS # ARCHITECTURE # DESIGN SYSTEMS # DEVOPS @_fabiankeller www.fabian-keller.de 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 2
3. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 3
4. Why self-hosting at all? There are some unique advantages 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 4 PRIVACY CONTROL ACCESS
5. The positions There is more than yes or no 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 5 OSCF is easy. It‘s the same technology. There is no license cost attached. OSCF is scary. We don‘t have SLAs. We‘re not an open source company. This talk Let‘s do this Better not do it
6. What are the challenges unique to Open Source Cloud Foundry? 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 6
7. The Case How we migrated the workload of our retailer 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 7 6 months to go live 200 software engineers 900 app instances in total 6 platform engineers
8. The 4 Phases we went through to take OSCF to production 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 8 Planning Plan and create required infrastructure on IaaS Thriving In Concourse we trust Deploying We can really do this! Operating Mh, we need to get this updated and changed Go Live Today
9. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 9 Technology Organisational Processes Culture The Challenges
10. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 10 Technology Organisational Processes Culture The Challenges
11. For developers, there is no difference in using OSCF!* Same DX. Same Operations. Same Runtime. *at least, there shouldn‘t be. 11 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller
12. So what‘s the difference? It‘s just a different package for the same components 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 12 OSCF Vendor YAML GUI Packaging Hiding Complexity
13. BOSH BOSH BOSH Learning BOSH and all it‘s YAML 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 13
14. Tailoring the Deployment Get inspired by the official operations files… • Backup and Restore cf-deployment/operations/backup-and-restore/*.yml • Log Management cf-deployment/operations/addons/enable-component-syslog.yml • Monitoring bosh-deployment/hm/*.yml • Resurrector by default • Instance Sizing cf-deployment/operations/scale-to-one-az.yml • High Availability by default • Security several ops files for enabling TLS, rate limits, etc. • Authentication cf-deployment/operations/test/add-oidc-provider.yml 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 14
15. Tailoring the Deployment … and then write custom operations files 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 15 https://bosh.io/jobs/uaa?source=github.com/cloudfoundry/uaa-release&version=74.0.0#p%3duaa.ldap
16. Tailoring the Deployment Dive deep to understand what happens 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 16
17. Tailoring the Deployment Dive deep to understand what happens 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 17
18. Don’t be afraid to read the source code! 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 18 Photo by Jeremy Bishop on Unsplash
19. Migrating to Another Foundation Leveraging route services to proxy traffic 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 19 CF-A App App App App Load Balancer CF-B App App App App Before Migration CF-A App App Migration Proxy Load Balancer CF-B App App App App During Migration CF-A App App Migration Proxy Load Balancer CF-B App App App App After Migration
20. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 20 Technology Organisational Processes Culture The Challenges
21. Hiring We‘re looking for a platform engineer Required Skills • With 10 years experience operating a PaaS • Who contributed to Cloud Foundry • Who used BOSH to run a deployment • Who knows Kubernetes, as it‘s our next challenge Required Skills • Who has used a PaaS before • Who knows Cloud Foundry • Who has heard of BOSH before • Who is interested in learning Kubernetes 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 21
22. Upskilling Invest in a culture of learning and sharing • Everyone must be able to do everything • Cloud Tooling JSON / YAML / jq / Git / S3 / Concourse / BOSH • Pair & Mob Programming • Cloud Foundry Architecture 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 22 Photo by Aditya Romansa on Unsplash
23. Risk Management Does OSCF have different risks compared to a vendor distribution? • Same risks. Different Probability. • Platform Breach: Equal Chances • Platform Outage: Higher Chances • Big difference: no risk outsourcing to a vendor! • No guaranteed repair time • No guaranteed security patches • No help with disaster recovery • What you can do: upskill & prepare! 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 23
24. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 24 Technology Organisational Processes Culture The Challenges
25. Buildpacks Compiling from source… 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 25
26. Buildpacks …and distributing accross environments 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 26
27. Upgrade Automation Start small and iterate • We‘re using GitOps and Concourse CI • Pipeline pattern: upload releases à deploy à smoke test 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 27
28. Upgrade Automation Roadmap So little time, so much to automate • Automated release integration testing • Download and bundle all releases • Deploy to multiple environments with the exact same bits • Optimize GitOps workflow 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 28
29. Support Getting involved with the community • CF Community: https://www.cloudfoundry.org/community/ • Slack: https://slack.cloudfoundry.org/ • GitHub: https://github.com/cloudfoundry • Twitter: https://twitter.com/cloudfoundry • Mailing Lists: https://lists.cloudfoundry.org/g/announce • No SLAs 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 29
30. Bug Fixing & Patching Learn Go • Prepare to fix and patch things yourself • Make a small patch in Go • Build the BOSH release • Verify the patch works • Roll it out • Give back to the community • Companies are not set up for open source • Open sourcing is not part of usual processes 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 30
31. 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 31 Technology Organisational Processes Culture The Challenges
32. Agile Adopting an agile mindset • Delivering value, fast • Maintaining and prioritizing a backlog • Daily Standup • Pair / Mob programming 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 32 Photo by İrfan Simsar on Unsplash
33. Failure Happens All the difference makes how you deal with it • Failures by the platform team quickly affect hundreds of apps • To err is human. No finger-pointing! • Appreciate if someone did a good job! • Be kind 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 33
34. Give back • Share what you do • Share how you do it • Raise issues • Contribute patches • Get involved 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 34 Photo by Tim Marshall on Unsplash
35. Cloud Foundry Operators Guide See how we do things 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 35 https://cfops-guide.mimacom.com/ https://github.com/mimacom/cfops-guide
36. Retrospective • Would we do it again? Hell yes! • Enjoy the learning curve • When to run OSCF • If the PaaS is part of the business strategy • Hybrid mode: critical vs. non-critical workloads 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 36
37. Thanks! bosh -d audience deploy questions.yml @_fabiankeller 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller Photo by Bill Jelen on Unsplash 37
38. Credits • Technology photo: Photo by Tim Mossholder on Unsplash • Organisational photo: Photo by Tomoe Steineck on Unsplash • Processes photo: Photo by Crystal Kwok on Unsplash • Culture photo: Photo by Helena Lopes on Unsplash • Icons: https://fontawesome.com 2019-09-11 / Cloud Foundry Summit Europe / Fabian Keller 38