Deployments

One teams war on flaky and slow deployments on OpenStack

Transcript

1. Deploy
2. About Me • Member of Configuration Automation Team in DEA • I’m usually work on building and managing stateful systems (ELK, KeyValueApi/ETCD) on OpenStack • I have held a number of roles (Sysadmin, Developer, Reliability Engineer, System Analyst, Senior Developer) mainly in mining and transport systems • ‘Full(er) Stack’ - I’ve worked in embedded systems, CI/CD, frontend web, backend, and cloud. • Sporadic Open Source Developer
3. Goals • Introduce the latest version of the deployment pipeline on OpenStack to a wider audience. • Understand the decisions that influenced the design and why they were made. • Frame this design in the wider context of cloud applications
4. Deploy (verb) 1. Move (troops or equipment) into position for military action. 2. Bring into effective action.
5. What makes a good deploy? • Simple • Light on details; do not expose infrastructure! • Little human intervention required • Automatic Testing • Fast • Assumes that failure is an option
6. Report Card • Simple • Light on details; do not expose infrastructure! • Little human intervention required • Automatic Testing • Fast • Assumes that failure is an option Fail Fail Pass Pass Fail Fail
7. A note on PaaS systems
8. Constraints
9. Immutable Infrastructure Build Live S1 S1 S1 + Δ = S2 S2 Build Live S1 S1 Δ S2 Shared vs Bundled, Dynamic vs Static
10. E2E Deployment Lifecycle App Deploy
11. A solution Preconfigured Base Image = SoE + Patches + OpenStack Configuration Platform Images = Preconfigured Base Image + Framework (.NET or JRE) Platform Image App v1 App v2
12. Application Deploy Octopus Deploy PreDeploy.ps1 Deploy.ps1 PostDeploy.ps1 VS In-built IIS/Tomcat Integration
13. OS Deploy Create Stack Register Tentacle App Deploy + Block Delete Stack
14. OS Deploy Get-Clouds Init-Deployment New-Deployment Move-DeploymentLive Remove-Deployment Output "clouds": [ { "Name": "cloud1", "Code": "A", "KeystoneUrl": ... }, { "Name": "cloud2", ... }, ] "deployments": [ { "Cloud": "A", "ApplicationId": "KeyValueApi", "Environment": "test", "Release": "1.0.57", "RandomId": "a76eop", "StackName": …” "Keystone": { ... } }, … "deployments": [ { "Status": "CREATE_COMPLETE", "Cloud": "A", "ApplicationId": "KeyValueApi", "Environment": "test", "Release": "1.0.57", "RandomId": "a76eop", "StackName": ... "Keystone": { ... } }, ... Wrap Update-DeploymentPools { "A": { "prelive": { "admin_state_up": false }, "live": { "admin_state_up": true } }, "B": { ... } } Heat Calls Get-Keystone Create-Stack Get-Stack Get-StackResources Get-StackResources Remove-Stack Neutron Calls Update-Member
15. Results • Applications deployed ~ 7x Faster • Application deploys are less likely to fail • Operating System concerns can potentially be abstracted away from teams managing and operating their applications • Moved much closer to a PaaS model
16. On OpenStack • Our version of OpenStack is ancient ◦ Many new API’s have been published ◦ Many additions to existing APIs have been made ◦ There are many bugs that have been fixed that have not been backported • Internal processes never adapted to the realities of running a cloud ◦ Who pays? Project-based model does not work for shared services ◦ Learning to operate a cloud platform whilst building cloud applications on it was always going to be a tall order - particularly at an IaaS level Poor Utilization (VM) + No (real) Elasticity
17. Report Card • Simple • Light on details; do not expose infrastructure! • Little human intervention required • Automatic Testing • Fast • Assumes that failure is an option Pass Pass Pass Pass Pass Pass
18. Questions?