Extending CI/CD in Operations Using Chef & LXC

Chef is a popular configuration management system used for system automation, that promotes the idea of infrastructure as code. In this presentation I'll walk through how we are using LXC to perform integration testing of our entire infrastructure automation code base, and then re-using the build/test artifacts to compose developer workstation. I'll be covering a) current state of chef-lxc integration and how we are using it, b) how our build-ci infrastructure was extended to build and test our chef/automation code base using LXC, and c) how the test artifacts are served via traditional debain repos to build developer workstations (i.e. the LXC rootfs packaging gears)

Transcript

1. Extending CI/CD in Operations Using Chef & LXC @RanjibDey
2. About Me @RanjibDey
3. @RanjibDey Part 1 : The Case for CI/CD in Operations
4. Challenges in Operations @RanjibDey 1.Integrating disparate systems 2.Maintaining legacy components 3.The burden of tactical works
5. CI and CD @RanjibDey 1.Layered test suites invoked on every commit 2.Every commit that passes test gets deployed 3.Deployment stage can be semi-automatic
6. CI and CD @RanjibDey 1. Running tests reduces failure 2. Automated deployment reduces human error. 3. Reduce build test deploy cycle time → →
7. But CI/CD In Operations Is Hard @RanjibDey 1.Lack of domain knowledge 2.Maturity of tooling, interoperability issues 3.Diversity of system components 4.Longer build/test times
8. @RanjibDey Part 2 : Tools of the Trade
9. GoCD – Fan In & Fan Out @RanjibDey 1. An OpenSource CI system focused on pipelines 2.Run tests on dependency changes 3. Run tests of dependent projects 4. Allows extensive parallelization http://www.go.cd/
10. LXC – System containers for the win @RanjibDey 1. Full system containers (cron, init, syslog) 2. Usernamespace, seccomp, apparmor 3. Liblxc allows first class python/ruby binding 4. No filesystem layers. Isolated container rootfs 5. Minimal containment system.
11. Ruby LXC @RanjibDey Ruby bindings for liblxc, allows container operations from arbitrary ruby code. require 'lxc' c = LXC::Container.new('foo') c.create('ubuntu') c.start c.stop c.destroy https://github.com/lxc/ruby-lxc
12. Chef – A system automation framework @RanjibDey 1. Represent infrastructure with vanilla Ruby. 2. A configuration management system 3. Baked in metadata service 4. Baked in host discovery 5. Subsystem for building CLI tools (knife)
13. Chef-LXC @RanjibDey Chef integration for LXC, allows creating containers From chef cookbooks. lxc "web" do template "ubuntu" action [:create, :start] end https://github.com/ranjib/chef-lxc
14. Chef-LXC @RanjibDey Allows executing chef resources inside containers without installing chef inside them. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end
15. @RanjibDey Part 3 : A CI/CD Pipeline for Operations
16. Standard CI pipeline @RanjibDey 1.Unit tests for fast feedback 2.Functional tests for feature validation 3.Integration tests for end to end testing
17. Unit tests @RanjibDey directory "/var/go/.ssh" do owner 'go' group 'go' mode 0700 end it 'creates the go user ssh directory' do expect(runner).to create_directory('/var/go/.ssh').with( owner: 'go', group: 'go', mode: 0700 ) end https://github.com/sethvargo/chefspec
18. Unit tests @RanjibDey pagerduty/infra[master]$rake spec Finished in 1 minute 15.16 seconds 5629 examples, 0 failures, 4 pending
19. Functional tests @RanjibDey it 'creates a standalone zk node' do ct = helper.container('pd-zk') out = ct.execute do h= 'localhost' p = 2181 telnet = Net::Telnet::new('Host' => h, 'Port' => p) state = telnet.cmd('String' => 'ruok') telnet.close state end expect(out).to eq('imok') end
20. Integration Tests @RanjibDey
21. CI Pipeline Stages for Operations @RanjibDey
22. Courtesy artifacts @RanjibDey Chef cookbooks used for building production infrastructure, are distributed as debian packages that can be consumed by ancillary projects, like building developer infrastructure
23. @RanjibDey Part 3 : Case studies
24. Blender @RanjibDey 1. A modular remote command execution framework 2. Pluggable host discovery (Chef, Serf etc) 3. Pluggable command dispatch mechanism (ssh, serf)
25. Blender :: Integration Test Suite @RanjibDey Blender-Core Blender-Serf Blender-Chef Blender-Zk Blender Integration Tests https://github.com/pagerduty/blender-integration
26. Blender :: Integration Test Suite @RanjibDey
27. GoatOS @RanjibDey 1. An integration testing framework for Chef cookbooks 2. Build chef omnibus installers for every commit 3. Tests community cookbooks against new installers
28. GoatOS :: Stages @RanjibDey Chef • Unit tests • Functional test • Build Gem Chef-LXC • Functional Tests GoatOS-Spec • Test cookbooks (mysql, sudo, nginx etc) Build Omnibus • Debian package https://github.com/GoatOS
29. GoatOS :: GoCD Pipeline View @RanjibDey
30. GoatOS :: Chef build on Raspberry Pi @RanjibDey Chef omnibus installers for Raspberry Pi are created via GoatOS
31. Learnings @RanjibDey 1. Co-develop CI gears with automation suite 2. Embrace standard software engineering principles 3. Avoid ad hoc scripts. 4. Upstream everything
32. @RanjibDey Thank You