Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending CI/CD in Operations Using Chef & LXC

Extending CI/CD in Operations Using Chef & LXC

Chef is a popular configuration management system used for system automation, that promotes the idea of infrastructure as code. In this presentation I'll walk through how we are using LXC to perform integration testing of our entire infrastructure automation code base, and then re-using the build/test artifacts to compose developer workstation. I'll be covering a) current state of chef-lxc integration and how we are using it, b) how our build-ci infrastructure was extended to build and test our chef/automation code base using LXC, and c) how the test artifacts are served via traditional debain repos to build developer workstations (i.e. the LXC rootfs packaging gears)

Ranjib Dey

August 20, 2015

More Decks by Ranjib Dey

Other Decks in Programming


  1. CI and CD @RanjibDey 1.Layered test suites invoked on every

    commit 2.Every commit that passes test gets deployed 3.Deployment stage can be semi-automatic
  2. CI and CD @RanjibDey 1. Running tests reduces failure 2.

    Automated deployment reduces human error. 3. Reduce build test deploy cycle time → →
  3. But CI/CD In Operations Is Hard @RanjibDey 1.Lack of domain

    knowledge 2.Maturity of tooling, interoperability issues 3.Diversity of system components 4.Longer build/test times
  4. GoCD – Fan In & Fan Out @RanjibDey 1. An

    OpenSource CI system focused on pipelines 2.Run tests on dependency changes 3. Run tests of dependent projects 4. Allows extensive parallelization http://www.go.cd/
  5. LXC – System containers for the win @RanjibDey 1. Full

    system containers (cron, init, syslog) 2. Usernamespace, seccomp, apparmor 3. Liblxc allows first class python/ruby binding 4. No filesystem layers. Isolated container rootfs 5. Minimal containment system.
  6. Ruby LXC @RanjibDey Ruby bindings for liblxc, allows container operations

    from arbitrary ruby code. require 'lxc' c = LXC::Container.new('foo') c.create('ubuntu') c.start c.stop c.destroy https://github.com/lxc/ruby-lxc
  7. Chef – A system automation framework @RanjibDey 1. Represent infrastructure

    with vanilla Ruby. 2. A configuration management system 3. Baked in metadata service 4. Baked in host discovery 5. Subsystem for building CLI tools (knife)
  8. Chef-LXC @RanjibDey Chef integration for LXC, allows creating containers From

    chef cookbooks. lxc "web" do template "ubuntu" action [:create, :start] end https://github.com/ranjib/chef-lxc
  9. Chef-LXC @RanjibDey Allows executing chef resources inside containers without installing

    chef inside them. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end
  10. Standard CI pipeline @RanjibDey 1.Unit tests for fast feedback 2.Functional

    tests for feature validation 3.Integration tests for end to end testing
  11. Unit tests @RanjibDey directory "/var/go/.ssh" do owner 'go' group 'go'

    mode 0700 end it 'creates the go user ssh directory' do expect(runner).to create_directory('/var/go/.ssh').with( owner: 'go', group: 'go', mode: 0700 ) end https://github.com/sethvargo/chefspec
  12. Functional tests @RanjibDey it 'creates a standalone zk node' do

    ct = helper.container('pd-zk') out = ct.execute do h= 'localhost' p = 2181 telnet = Net::Telnet::new('Host' => h, 'Port' => p) state = telnet.cmd('String' => 'ruok') telnet.close state end expect(out).to eq('imok') end
  13. Courtesy artifacts @RanjibDey Chef cookbooks used for building production infrastructure,

    are distributed as debian packages that can be consumed by ancillary projects, like building developer infrastructure
  14. Blender @RanjibDey 1. A modular remote command execution framework 2.

    Pluggable host discovery (Chef, Serf etc) 3. Pluggable command dispatch mechanism (ssh, serf)
  15. Blender :: Integration Test Suite @RanjibDey Blender-Core Blender-Serf Blender-Chef Blender-Zk

    Blender Integration Tests https://github.com/pagerduty/blender-integration
  16. GoatOS @RanjibDey 1. An integration testing framework for Chef cookbooks

    2. Build chef omnibus installers for every commit 3. Tests community cookbooks against new installers
  17. GoatOS :: Stages @RanjibDey Chef • Unit tests • Functional

    test • Build Gem Chef-LXC • Functional Tests GoatOS-Spec • Test cookbooks (mysql, sudo, nginx etc) Build Omnibus • Debian package https://github.com/GoatOS
  18. GoatOS :: Chef build on Raspberry Pi @RanjibDey Chef omnibus

    installers for Raspberry Pi are created via GoatOS
  19. Learnings @RanjibDey 1. Co-develop CI gears with automation suite 2.

    Embrace standard software engineering principles 3. Avoid ad hoc scripts. 4. Upstream everything