A CI-CD Alternative to Push and Pray for OpenStack

Transcript

A CI/CD Alternative to Push and Pray for OpenStack

Introduction Clayton O’Neill –  [email protected] –  IRC: clayton (Twitter: @clayton_oneill)

Matt Fischer –  [email protected] –  IRC: mfisch (Twitter: @openmfisch) Eric Peterson –  [email protected] –  IRC: ducttape_

Background •  New team started in Nov 2013 •  OpenStack

in production July 2014 •  CI/CD infrastructure started after production

Our CI/CD Stats •  Over 4300 Reviews in our gerrit

instance •  Over 300k nodepool slaves spun-up •  Hundreds of deploys •  Thousands of lines of Puppet, Ansible, and Python •  16.5k lines of upstream changes

Development to Production Dev OpenStack on Prod Team Member DEV01

DEV02 Team Member Dev02 DEV02 Dev01 DEV01 Team Member DEV01 Shared Dev BFD02 DEV01 DEV02 Staging STG02 STG01 DEV01 DEV02 Production PRD02 PRD01 Infra Gerrit Jenkins

Region 2 Region 1 Control Control MySQL Cluster Compute Node

Node Node RabbitMQ Cluster Keystone/Horizon Node Node Node Node Node Nodes Keystone/Horizon Node Node Node HAProxy Node Node MySQL Cluster Node Node Node RabbitMQ Cluster MySQL Cluster Ceph Node Node Nodes Compute Node Node Nodes Ceph Node Node Nodes HAProxy Node Node Build Server Puppet Cobbler Build Server Puppet Cobbler

Dev OpenStack on Prod Dev OpenStack on Prod Team Member

DEV01 DEV02 Team Member Dev02 DEV02 Dev01 DEV01 Team Member DEV01 Shared Dev BFD02 DEV01 DEV02 Staging STG02 STG01 DEV01 DEV02 Production PRD02 PRD01 Infra Gerrit Jenkins

Virtualized Dev Environments •  Vagrant based –  ggiamarchi/vagrant-openstack-provider •  All

environments built with the same tools •  Team members can have multiple environments •  Can be any or all node types •  Sharable for troubleshooting

Code Review Dev OpenStack on Prod Team Member DEV01 DEV02

Team Member Dev02 DEV02 Dev01 DEV01 Team Member DEV01 Shared Dev BFD02 DEV01 DEV02 Staging STG02 STG01 DEV01 DEV02 Production PRD02 PRD01 Infra Gerrit Jenkins

•  Code Quality •  Mentoring •  Shared Ownership •  Pre-merge

Testing Code Review

Jenkins Dev OpenStack on Prod Team Member DEV01 DEV02 Team

Member Dev02 DEV02 Dev01 DEV01 Team Member DEV01 Shared Dev BFD02 DEV01 DEV02 Staging STG02 STG01 DEV01 DEV02 Production PRD02 PRD01 Infra Gerrit Jenkins

Single Use Jenkins Slaves •  Clean state on every job

run •  Nodepool –  Builds Jenkins slaves as needed –  Builds Glance images once a day –  Supports multi-node slaves •  Single Use Slave Jenkins Plugin –  Ensures slaves are are offlined after job

Jenkins Job Builder •  Jenkins job management doesn’t scale • 

Using JJB for all jobs –  Jobs defined as YAML –  Code/config reuse via macros and templates –  Deploy and version jobs like everything else –  JJB is authoritative.

Automated Testing Dev OpenStack on Prod Team Member DEV01 DEV02

Team Member Dev02 DEV02 Dev01 DEV01 Team Member DEV01 Shared Dev BFD02 DEV01 DEV02 Staging STG02 STG01 DEV01 DEV02 Production PRD02 PRD01 Infra Gerrit Jenkins

Pre-Merge Testing •  Puppet lint •  Syntax tests •  Unit

tests •  Tox •  Puppet catalog compiles & diffs

Puppet Catalog Compiles? •  Puppet builds a catalog for each

node •  Inputs are code + config + facts •  Jenkins builds before and after view for one of each node type in every environment •  Jenkins posts comment on review with changed nodes + link to diffs •  Diffs are built with ripienaar/puppet-catalog-diff module

No changes!

Updating puppet-nova

None

Catalog Diff Pro/Con •  Validate config in all environments • 

More detail than unit tests •  Faster than integration testing (<5 min) •  Doesn’t report on –  Ruby code changes –  Service restarts, package upgrades, etc

Automated Integration Testing •  Hourly testing of core node types

•  Using Nodepool multi-node slaves •  Node builds are parallelized •  Test runs in ~45 minutes •  Catches breaking changes post-commit •  Future focus area

Region 2 Region 1 Control Control MySQL Cluster Compute Node

Puppet Configuration •  Single Puppet master per environment •  Not

using dynamic environments •  Minimal PuppetDB usage •  Hiera for environment specific changes •  Use hiera-eyaml for secret data •  Use r10k for module versioning & deployment

puppet-config/master Puppetfile Hiera Data Puppetfile.yaml keystone neutron nova cirrus cirrus_icinga

... mysql galera haproxy gerrit stdlib ...

Normal Puppetfile mod "mysql", :git => 'https://github.com/puppetlabs/puppetlabs-mysql.git', :tag => '3.3.0'

# Forked until this PR is accepted # https://github.com/michaeltchapman/puppet-galera/pull/27 mod 'galera', :git => 'https://github.com/twc-openstack/puppet-galera', :commit => 'c3e37c8e61b367e64263f80f89642ce12dfecdb7' mod 'haproxy', :git => 'https://github.com/puppetlabs/puppetlabs-haproxy.git', :tag => '0.5.0'

Top of Puppetfile # This loads the YAML file that

Jenkins maintains of the latest commits # approved through Gerrit. PUPPET_DIR = ENV['PUPPET_DIR'] || '/etc/puppet' PUPPETFILE_YAML = File.join(PUPPET_DIR, 'Puppetfile.yaml') if File.readable?(PUPPETFILE_YAML) require 'yaml' data = YAML.load_file(PUPPETFILE_YAML) data['modules'].each_pair do |modulename, moduledata| mod modulename, :git => moduledata['git'], :commit => moduledata['ref'] end end

Puppetfile.yaml entry modules: cirrus: author_date: "2015-05-04 15:20:04 +0000" author_email: "[email protected]"

author_name: "Clayton O'Neill" changeid: I686e85f61cd7526d4faa56727351206ea9789dfa gerrit_url: "https://bfd-gerrit.os.cloud.twc.net/#q,I686e85f61cd7526d4faa56727..." git: ssh://deploy-host/puppet-cirrus issue: CIRRUS-2103 ref: 2b7d32df9f2c741425e8396414ecf2927f8b510d subject: "Add designate sink support

A CI-CD Alternative to Push and Pray for OpenStack

A CI-CD Alternative to Push and Pray for OpenStack

More Decks by Matt Fischer

Other Decks in Programming

Featured

Transcript