CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud

Our pipeline that builds on individual configuration management modules to layer up to a full multi-node dev environment and on to production deploys

Transcript

1. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Simon McCartney, Site Reliability Engineering OpenStack Summit 2014 - Paris CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
2. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Background •  This project predates HP Helion OpenStack •  We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack
3. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why?
4. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? •  Repeatable, Reliable & Incrementally Improving •  Constant flow of changes to benefit users
5. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? •  Infrastructure as Code > Infrastructure as Art •  Frequent Small Batches > Infrequent Big Batches
6. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? •  Time Vampires •  Manual Builds •  Manual Testing •  Improved consistency & quality •  Faster Cycle Time
7. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery •  Unit Tests - salt formulae •  Integration Tests - do these salt formulae work together? •  End-to-End Tests - does the OpenStack Cloud we built work? •  Performance Test - did we break/de-tune something? •  Deployment Tests - do we have everything? •  Configuration Tests - can we test different environments before hitting the real environment?
8. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline
9. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Tech Stack •  Vendor packages for OpenStack (Ubuntu) •  Configuration management (SaltStack) •  Software Engineering •  git, gerrit, gitshelf & jenkins •  test-kitchen (kitchen-salt, serverspec) •  Infrastructure Engineering •  Vagrant/VirtualBox & Public Cloud
10. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline 1.  Individual Salt Formula development 2.  Personal multi-node, package based OpenStack environment on your workstation for dev & validation 3.  Go public – push to gerrit for review & automated testing 4.  Pull in to the deploy-kit 5.  deploy-kit tooling kicks in & builds deploy artefacts 6.  Auto deploy to ephemeral public cloud test environment 7.  Deploy to physical staging environment 8.  Ready for production
11. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
12. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Validating & Approving a change •  gerrit - defacto web based open source code review system •  submit changes (a patch set) for review by your peers.. •  ..and for testing by Jenkins •  Jenkins & Gerrit Trigger •  Gerrit plugins allow certain jobs to be triggered on patch-set events (create, merge etc) •  results of these jobs are then recorded in gerrit •  test-kitchen •  test harness to execute your configuration management code in isolation (i.e. fresh VMs) •  providers (vagrant, LXC, openstack, gce, aws etc) •  provisioners (Chef, Salt, Puppet) •  testing frameworks (Bats, RSpec, serverspec etc) •  can be used interactively in your dev environment as well as in validation jobs
13. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
14. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. In-cloud validation 1.  contractor creates instances & networks 2.  bootstrap.sh used prepare vanilla image for use with salt 3.  Jenkins uploads release artifacts and deploys salt states 4.  Performa validation tests "roles": {! ...! "compute": {! "image": "261844b3-479c-...",! "flavor": "101",! "keypair": "jenkins",! ! "instances": {! "ae1": [{! "az": "az3",! "number": 1,! "nics": [{! "network": "public-net",! "fixed_ip": "172.16.0.101",! "floating_ip": "15.126.241.109"! },! {! "network": "private-net",! "fixed_ip": "10.0.0.101"! }]! !
15. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Deployment Automation •  Production Deploys •  Move slowly when required •  Confirm host is still “good” using existing monitoring framework •  Caution over nova-* service restarts •  Next •  rundeck •  chatops
16. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Links •  http://devops.com/blogs/continuous-delivery-pipeline/ •  http://martinfowler.com/bliki/DeploymentPipeline.html •  https://github.com/secure-pipeline/jenkins-example •  https://speakerdeck.com/garethr/continuous-integration-for- infrastructure •  https://github.com/gitshelf/gitshelf •  https://github.com/moniker-dns/contractor •  https://github.com/macgreagoir/gobstack
17. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Questions?
18. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Thank you! Simon McCartney @simonmcc & Mick Gregg mickgregg.com