Share
Embed
CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
Our pipeline that builds on individual configuration management modules to layer up to a full multi-node dev environment and on to production deploys
Simon McCartney
November 04, 2014
Transcript
18 slides
-
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Simon McCartney, Site Reliability Engineering OpenStack Summit 2014 - Paris CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Background • This project predates HP Helion OpenStack • We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Why? -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Why Continuous Integration & Delivery? • Repeatable, Reliable & Incrementally Improving • Constant flow of changes to benefit users -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Why Continuous Integration & Delivery? • Infrastructure as Code > Infrastructure as Art • Frequent Small Batches > Infrequent Big Batches -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Why Continuous Integration & Delivery? • Time Vampires • Manual Builds • Manual Testing • Improved consistency & quality • Faster Cycle Time -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Why Continuous Integration & Delivery • Unit Tests - salt formulae • Integration Tests - do these salt formulae work together? • End-to-End Tests - does the OpenStack Cloud we built work? • Performance Test - did we break/de-tune something? • Deployment Tests - do we have everything? • Configuration Tests - can we test different environments before hitting the real environment? -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Our Pipeline -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Our Tech Stack • Vendor packages for OpenStack (Ubuntu) • Configuration management (SaltStack) • Software Engineering • git, gerrit, gitshelf & jenkins • test-kitchen (kitchen-salt, serverspec) • Infrastructure Engineering • Vagrant/VirtualBox & Public Cloud -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Our Pipeline 1. Individual Salt Formula development 2. Personal multi-node, package based OpenStack environment on your workstation for dev & validation 3. Go public – push to gerrit for review & automated testing 4. Pull in to the deploy-kit 5. deploy-kit tooling kicks in & builds deploy artefacts 6. Auto deploy to ephemeral public cloud test environment 7. Deploy to physical staging environment 8. Ready for production -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Validating & Approving a change • gerrit - defacto web based open source code review system • submit changes (a patch set) for review by your peers.. • ..and for testing by Jenkins • Jenkins & Gerrit Trigger • Gerrit plugins allow certain jobs to be triggered on patch-set events (create, merge etc) • results of these jobs are then recorded in gerrit • test-kitchen • test harness to execute your configuration management code in isolation (i.e. fresh VMs) • providers (vagrant, LXC, openstack, gce, aws etc) • provisioners (Chef, Salt, Puppet) • testing frameworks (Bats, RSpec, serverspec etc) • can be used interactively in your dev environment as well as in validation jobs -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. In-cloud validation 1. contractor creates instances & networks 2. bootstrap.sh used prepare vanilla image for use with salt 3. Jenkins uploads release artifacts and deploys salt states 4. Performa validation tests "roles": {! ...! "compute": {! "image": "261844b3-479c-...",! "flavor": "101",! "keypair": "jenkins",! ! "instances": {! "ae1": [{! "az": "az3",! "number": 1,! "nics": [{! "network": "public-net",! "fixed_ip": "172.16.0.101",! "floating_ip": "15.126.241.109"! },! {! "network": "private-net",! "fixed_ip": "10.0.0.101"! }]! ! -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Deployment Automation • Production Deploys • Move slowly when required • Confirm host is still “good” using existing monitoring framework • Caution over nova-* service restarts • Next • rundeck • chatops -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Links • http://devops.com/blogs/continuous-delivery-pipeline/ • http://martinfowler.com/bliki/DeploymentPipeline.html • https://github.com/secure-pipeline/jenkins-example • https://speakerdeck.com/garethr/continuous-integration-for- infrastructure • https://github.com/gitshelf/gitshelf • https://github.com/moniker-dns/contractor • https://github.com/macgreagoir/gobstack -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Questions? -
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained
herein is subject to change without notice. Thank you! Simon McCartney @simonmcc & Mick Gregg mickgregg.com