Extending CI/CD in Operations Using Chef & LXC

Extending CI/CD in Operations Using Chef & LXC

Chef is a popular configuration management system used for system automation, that promotes the idea of infrastructure as code. In this presentation I'll walk through how we are using LXC to perform integration testing of our entire infrastructure automation code base, and then re-using the build/test artifacts to compose developer workstation. I'll be covering a) current state of chef-lxc integration and how we are using it, b) how our build-ci infrastructure was extended to build and test our chef/automation code base using LXC, and c) how the test artifacts are served via traditional debain repos to build developer workstations (i.e. the LXC rootfs packaging gears)


Ranjib Dey

August 20, 2015


  1. Extending CI/CD in Operations Using Chef & LXC @RanjibDey

  2. About Me @RanjibDey

  3. @RanjibDey Part 1 : The Case for CI/CD in Operations

  4. Challenges in Operations @RanjibDey 1.Integrating disparate systems 2.Maintaining legacy components

    3.The burden of tactical works
  5. CI and CD @RanjibDey 1.Layered test suites invoked on every

    commit 2.Every commit that passes test gets deployed 3.Deployment stage can be semi-automatic
  6. CI and CD @RanjibDey 1. Running tests reduces failure 2.

    Automated deployment reduces human error. 3. Reduce build test deploy cycle time → →
  7. But CI/CD In Operations Is Hard @RanjibDey 1.Lack of domain

    knowledge 2.Maturity of tooling, interoperability issues 3.Diversity of system components 4.Longer build/test times
  8. @RanjibDey Part 2 : Tools of the Trade

  9. GoCD – Fan In & Fan Out @RanjibDey 1. An

    OpenSource CI system focused on pipelines 2.Run tests on dependency changes 3. Run tests of dependent projects 4. Allows extensive parallelization http://www.go.cd/
  10. LXC – System containers for the win @RanjibDey 1. Full

    system containers (cron, init, syslog) 2. Usernamespace, seccomp, apparmor 3. Liblxc allows first class python/ruby binding 4. No filesystem layers. Isolated container rootfs 5. Minimal containment system.
  11. Ruby LXC @RanjibDey Ruby bindings for liblxc, allows container operations

    from arbitrary ruby code. require 'lxc' c = LXC::Container.new('foo') c.create('ubuntu') c.start c.stop c.destroy https://github.com/lxc/ruby-lxc
  12. Chef – A system automation framework @RanjibDey 1. Represent infrastructure

    with vanilla Ruby. 2. A configuration management system 3. Baked in metadata service 4. Baked in host discovery 5. Subsystem for building CLI tools (knife)
  13. Chef-LXC @RanjibDey Chef integration for LXC, allows creating containers From

    chef cookbooks. lxc "web" do template "ubuntu" action [:create, :start] end https://github.com/ranjib/chef-lxc
  14. Chef-LXC @RanjibDey Allows executing chef resources inside containers without installing

    chef inside them. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end
  15. @RanjibDey Part 3 : A CI/CD Pipeline for Operations

  16. Standard CI pipeline @RanjibDey 1.Unit tests for fast feedback 2.Functional

    tests for feature validation 3.Integration tests for end to end testing
  17. Unit tests @RanjibDey directory "/var/go/.ssh" do owner 'go' group 'go'

    mode 0700 end it 'creates the go user ssh directory' do expect(runner).to create_directory('/var/go/.ssh').with( owner: 'go', group: 'go', mode: 0700 ) end https://github.com/sethvargo/chefspec
  18. Unit tests @RanjibDey pagerduty/infra[master]$rake spec Finished in 1 minute 15.16

    seconds 5629 examples, 0 failures, 4 pending
  19. Functional tests @RanjibDey it 'creates a standalone zk node' do

    ct = helper.container('pd-zk') out = ct.execute do h= 'localhost' p = 2181 telnet = Net::Telnet::new('Host' => h, 'Port' => p) state = telnet.cmd('String' => 'ruok') telnet.close state end expect(out).to eq('imok') end
  20. Integration Tests @RanjibDey

  21. CI Pipeline Stages for Operations @RanjibDey

  22. Courtesy artifacts @RanjibDey Chef cookbooks used for building production infrastructure,

    are distributed as debian packages that can be consumed by ancillary projects, like building developer infrastructure
  23. @RanjibDey Part 3 : Case studies

  24. Blender @RanjibDey 1. A modular remote command execution framework 2.

    Pluggable host discovery (Chef, Serf etc) 3. Pluggable command dispatch mechanism (ssh, serf)
  25. Blender :: Integration Test Suite @RanjibDey Blender-Core Blender-Serf Blender-Chef Blender-Zk

    Blender Integration Tests https://github.com/pagerduty/blender-integration
  26. Blender :: Integration Test Suite @RanjibDey

  27. GoatOS @RanjibDey 1. An integration testing framework for Chef cookbooks

    2. Build chef omnibus installers for every commit 3. Tests community cookbooks against new installers
  28. GoatOS :: Stages @RanjibDey Chef • Unit tests • Functional

    test • Build Gem Chef-LXC • Functional Tests GoatOS-Spec • Test cookbooks (mysql, sudo, nginx etc) Build Omnibus • Debian package https://github.com/GoatOS
  29. GoatOS :: GoCD Pipeline View @RanjibDey

  30. GoatOS :: Chef build on Raspberry Pi @RanjibDey Chef omnibus

    installers for Raspberry Pi are created via GoatOS
  31. Learnings @RanjibDey 1. Co-develop CI gears with automation suite 2.

    Embrace standard software engineering principles 3. Avoid ad hoc scripts. 4. Upstream everything
  32. @RanjibDey Thank You