© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Simon McCartney, Site Reliability Engineering OpenStack Summit 2014 - Paris CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Background ●  This project predates HP Helion OpenStack ●  We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why? 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? •  Repeatable, Reliable & Incrementally Improving •  Constant flow of changes to benefit users 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? ●  Infrastructure as Code > Infrastructure as Art ●  Frequent Small Batches > Infrequent Big Batches 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? ●  Time Vampires ●  Manual Builds ●  Manual Testing ●  Improved consistency & quality ●  Faster Cycle Time 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery ●  Unit Tests - salt formulae ●  Integration Tests - do these salt formulae work together? ●  End-to-End Tests - does the OpenStack Cloud we built work? ●  Performance Test - did we break/de-tune something? ●  Deployment Tests - do we have everything? ●  Configuration Tests - can we test different environments before hitting the real environment? 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Tech Stack ●  Vendor packages for OpenStack (Ubuntu) ●  Configuration management (SaltStack) ●  Software Engineering ●  git, gerrit, gitshelf & jenkins ●  test-kitchen (kitchen-salt, serverspec) ●  Infrastructure Engineering ●  Vagrant/VirtualBox & Public Cloud 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline 1.  Individual Salt Formula development 2.  Personal multi-node, package based OpenStack environment on your workstation for dev & validation 3.  Go public – push to gerrit for review & automated testing 4.  Pull in to the deploy-kit 5.  deploy-kit tooling kicks in & builds deploy artefacts 6.  Auto deploy to ephemeral public cloud test environment 7.  Deploy to physical staging environment 8.  Ready for production 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Validating & Approving a change •  gerrit - defacto web based open source code review system •  submit changes (a patch set) for review by your peers.. •  ..and for testing by Jenkins •  Jenkins & Gerrit Trigger •  Gerrit plugins allow certain jobs to be triggered on patch-set events (create, merge etc) •  results of these jobs are then recorded in gerrit •  test-kitchen •  test harness to execute your configuration management code in isolation (i.e. fresh VMs) •  providers (vagrant, LXC, openstack, gce, aws etc) •  provisioners (Chef, Salt, Puppet) •  testing frameworks (Bats, RSpec, serverspec etc) •  can be used interactively in your dev environment as well as in validation jobs 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. In-cloud validation 1.  contractor creates instances & networks 2.  bootstrap.sh used prepare vanilla image for use with salt 3.  Jenkins uploads release artifacts and deploys salt states 4.  Performa validation tests "roles": {! ...! "compute": {! "image": "261844b3-479c-...",! "flavor": "101",! "keypair": "jenkins",! ! "instances": {! "ae1": [{! "az": "az3",! "number": 1,! "nics": [{! "network": "public-net",! "fixed_ip": "172.16.0.101",! "floating_ip": "15.126.241.109"! },! {! "network": "private-net",! "fixed_ip": "10.0.0.101"! }]! ! 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Deployment Automation •  Production Deploys •  Move slowly when required •  Confirm host is still “good” using existing monitoring framework •  Caution over nova-* service restarts •  Next •  rundeck •  chatops 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Links ●  http://devops.com/blogs/continuous-delivery-pipeline/ ●  http://martinfowler.com/bliki/DeploymentPipeline.html ●  https://github.com/secure-pipeline/jenkins-example ●  https://speakerdeck.com/garethr/continuous-integration-for- infrastructure ●  https://github.com/gitshelf/gitshelf ●  https://github.com/moniker-dns/contractor ●  https://github.com/macgreagoir/gobstack 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Questions? 

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Thank you! Simon McCartney @simonmcc & Mick Gregg mickgregg.com