Reviving the platform every day

Natural disaster hit your data centre? Don't worry. Cloud Foundry is prepared for disaster from day one thanks to built-in support for BOSH backup and restore (BBR). Nice. But how do we know the platform is ready for catastrophe? What if our worst fears come true?

The BBR framework laid the technical foundation for implementing disaster recovery. However, Cloud Foundry is a complex distributed system and contributors are spread across multiple teams and timezones. The challenge was to find a way to drive out this cross-cutting feature across all Cloud Foundry components.

Thus the Disaster Recovery Acceptance Tests (DRATs) were born. Josh and Emmanouil will show you the test framework that continuously tests a critical feature of the platform. DRATs ensures that all the Cloud Foundry components continue to work together, so we can all be confident that the platform can recover when mishaps occur.

After this talk attendees will:
* take away patterns for developing features with cross-cutting concerns.
* understand DRATs and how Cloud Foundry teams test their components.
* understand how to deliver CI tasks that can be shared by many teams in their own CI pipelines.

Transcript

2. Reviving the platform every day Emmanouil Kiagias & Josh Hill Platform Recovery, Pivotal #bbr on Cloud Foundry Slack
3. Agenda • Disaster Recovery • First steps • Ownership and Integration • DRATs • Shared CI Tasks • Ongoing development • What’s next?
4. Disaster Recovery How do we test it? Framing the problem
5. Disaster Recovery A plan describing how to recover my system in case of a catastrophe! (Assuming I have taken backups!)
6. Disaster Recovery A plan describing how to recover my system in case of a catastrophe! (Assuming I have taken backups!) Can Cloud Foundry recover from a disaster?
7. Disaster Recovery A plan describing how to recover my system in case of a catastrophe! (Assuming I have taken backups!) Can Cloud Foundry recover from a disaster? - Sure! BBR (Bosh Backup and Restore) can backup CF and help you restore it when a calamity hits your system
8. Disaster Recovery Good, BBR has laid the technical foundation for implementing disaster recovery Can I feel safe then? How do I know that my platform can actually recover? How do I that my disaster recovery solution works?
9. Framing the problem DR strategy: ➔ Take a backup of my system (record state) ➔ DISASTER ➔ Restore from backup ➔ Things should be back normal
10. Framing the problem DR strategy: ➔ Take a backup of my system (record state) ➔ DISASTER ➔ Restore from backup ➔ Things should be back normal Now how do I test that this works?
11. Framing the problem Test DR strategy of a single component: ➔ Setup a fresh component ➔ Create from state A ➔ Take a backup ➔ DESTROY THINGS! ➔ Restore from backup ➔ Assert state A is back, safe and sound
12. Framing the problem Test DR strategy of a single component: ➔ Setup a fresh component ➔ Create from state A ➔ Take a backup ➔ DESTROY THINGS! ➔ Restore from backup ➔ Assert state A is back, safe and sound Simple test case, isn’t it?
13. Framing the problem
14. Framing the problem Component
15. Framing the problem Component State
16. Framing the problem Component State
17. Framing the problem How would this look like in a distributed system like CF? Component State
18. Framing the problem CF UAA CF Networking Credhub CAPI ... ...
19. Framing the problem CF UAA CF Networking Credhub CAPI ... ...
20. First steps Challenge Cross cutting work
21. Challenge Platform Recovery was given the challenge to deliver this feature across all the teams. - How to test DR on each system component? - How to test all the components as a single system? How do can we coordinate this across the teams?
22. Challenge • Many components and teams ◦ CAPI ◦ UAA ◦ CF-Networking ◦ Credhub ◦ Platform Recovery ◦ ... • Different time zones: ◦ San Francisco (GMT-7), New York (GMT-4), London (GMT+1)...
23. First steps
24. First steps Big problem
25. First steps Smaller problems Big problem
26. First steps Smaller problems Big problem CF Individual components
27. First steps Break down the problem into smaller pieces - Each component will have its own testcase(s) Write first sample test cases
28. First steps Test DR strategy of a single component: ➔ Setup a fresh component ➔ Create from state A ➔ Take a backup ➔ DESTROY THINGS! ➔ Restore from backup ➔ Assert state A is back, safe and sound
29. First steps What do we do with these test cases now? Implement a test suite which can run the test cases - A framework which orchestrates the lifecycle of DR tests (This is how DRATs came in life)
30. Ownership & Integration
31. Ownership We now have: - Test suite - Sample test cases It is time to: Divide and Conquer - Distributed system? Distribute ownership Each CF team: - Owns and develops the test cases for their component - Works independently
32. Integration Things are now in motion DR test pieces are being developed CF teams are test driving their component’s DR strategy
33. Integration Things are now in motion DR test pieces are being developed CF teams are test driving their component’s DR strategy But DR is not useful unless it works for the whole system!
34. Integration Integrate all the things together! - Gather all test cases from the teams in a common place - Run them all against a CF deployment using our test suite
35. Integration Now we have a bunch of tests running as one Who should run these? - Platform Recovery team runs them all - We own and coordinate DR of the platform - Each CF team should run them to test their components’ DR - Release Integration team runs them before cutting a new cf-deployment
36. DRATS Disaster Recovery Acceptance Tests
37. DRATS • Automates a full backup and restore of Cloud Foundry • Provides hooks for test cases • Allows teams to choose which test cases to run
38. Full backup and restore $ bbr drats
39. Full backup and restore bbr deployment backup bbr deployment restore drats
40. Hooks ➔ Create state bbr deployment backup ➔ Change state bbr deployment restore ➔ Check state ➔ Cleanup state drats
41. Test cases Every test case must implement the hooks: type TestCase interface { Name() string BeforeBackup(Config) AfterBackup(Config) AfterRestore(Config) Cleanup(Config) } Source code
42. Test case for UAA Before backup • create a user • verify can log in After backup • delete the user • verify cannot log in After restore • verify can log in Cleanup • delete the user
43. Test cases • App • App uptime • UAA • CredHub • Networking • Router groups • NFS broker
44. Test case for app uptime Before backup • push an app • start polling the app After backup • stop polling the app • verify every request was successful After restore Cleanup • delete app
45. Full recovery bbr deployment backup ➔ Delete CF and redeploy bbr deployment restore drats
46. Focus • Every team can choose which test cases to run • Allows teams to iterate independently • Every team can run DRATS in their CI pipelines { "include_cf-uaa": true, "include_cf-credhub": false, ... }
47. Shared CI Tasks Running DRATS in many pipelines
48. Shared CI Tasks • All CF teams use Concourse CI (thankfully!) • Pipelines change often • Tasks are more stable
49. drats-with-integration-config inputs: - name: disaster-recovery-acceptance-tests - name: bbr-binary-release - name: drats-integration-config
50. update-integration-config inputs: - name: integration-config - name: vars-store - name: bbl-state-store outputs: - name: updated-integration-config
51. CF team pipelines • CAPI • UAA • CredHub • CF Networking • Persistence • Release Integration • Platform Recovery
52. CF team pipelines capi uaa credhub cf-networking persistence drats Platform Recovery cf-deployment Release Integration
53. CredHub
54. CAPI
55. Release Integration
56. Platform Recovery
57. Ongoing development
58. Core components • DRATS maintained by Platform Recovery team • Core CF teams submit pull requests • Support multiple versions e.g. Credhub v1 & v2
59. Datastore configurations • Extended support to external databases • MySQL • MariaDB & PXC clusters • Postgres • Extending support for external blobstores • AWS S3-compatible • Azure • Google Cloud
60. P-DRATS • We extended DRATS to test Pivotal Application Service • Uses the same open source test cases • Additional test cases for Pivotal components • Tests every version of PAS since 1.11
61. P-DRATS
62. What’s next?
63. Recap
64. Agenda • Disaster Recovery • First steps • Ownership and Integration • DRATs • Shared CI Tasks • Ongoing development • What’s next?
65. Thank you for listening! Emmanouil Kiagias & Josh Hill Platform Recovery, Pivotal #bbr on Cloud Foundry Slack